Chapter 10. Advanced managed cluster configuration with PolicyGenTemplate resources

Procedure

1. Update the PolicyGenTemplate entry for PerformanceProfile in the group-du-sno-ranGen.yaml reference file in out/argocd/example/policygentemplates as follows to set performance mode.

   - fileName: PerformanceProfile.yaml
     policyName: "config-policy"
     metadata:
       [...]
     spec:
       [...]
       workloadHints:
            realTime: true
            highPowerConsumption: false
            perPodPowerManagement: false

2. Commit the PolicyGenTemplate change in Git, and then push to the Git repository being monitored by the GitOps ZTP Argo CD application.

Procedure

1. Update the PolicyGenTemplate entry for PerformanceProfile in the group-du-sno-ranGen.yaml reference file in out/argocd/example/policygentemplates as follows to set high-performance mode.

   - fileName: PerformanceProfile.yaml
     policyName: "config-policy"
     metadata:
       [...]
     spec:
       [...]
       workloadHints:
            realTime: true
            highPowerConsumption: true
            perPodPowerManagement: false

2. Commit the PolicyGenTemplate change in Git, and then push to the Git repository being monitored by the GitOps ZTP Argo CD application.

Procedure

1. Update the PolicyGenTemplate entry for PerformanceProfile in the group-du-sno-ranGen.yaml reference file in out/argocd/example/policygentemplates as follows to configure power saving mode. It is recommended to configure the CPU governor for the power saving mode through the additional kernel arguments object.

   - fileName: PerformanceProfile.yaml
     policyName: "config-policy"
     metadata:
       [...]
     spec:
       [...]
       workloadHints:
            realTime: true
            highPowerConsumption: false
            perPodPowerManagement: true
       [...]
       additionalKernelArgs:
          - [...]
          - "cpufreq.default_governor=schedutil" 1

   1

   The schedutil governor is recommended, however, other governors that can be used include ondemand and powersave.

2. Commit the PolicyGenTemplate change in Git, and then push to the Git repository being monitored by the GitOps ZTP Argo CD application.

Verification

1. Select a worker node in your deployed cluster from the list of nodes identified by using the following command:

   $ oc get nodes

2. Log in to the node by using the following command:

   $ oc debug node/<node-name>

   Replace <node-name> with the name of the node you want to verify the power state on.

3. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths as shown in the following example:

   # chroot /host

4. Run the following command to verify the applied power state:

   # cat /proc/cmdline

Procedure

1. Update the PolicyGenTemplate entry for TunedPerformancePatch in the group-du-sno-ranGen.yaml reference file in out/argocd/example/policygentemplates. To maximize power savings, add max_perf_pct as shown in the following example:

   - fileName: TunedPerformancePatch.yaml
         policyName: "config-policy"
         spec:
           profile:
             - name: performance-patch
               data: |
                 [...]
                 [sysfs]
                 /sys/devices/system/cpu/intel_pstate/max_perf_pct=<x> 1

   1

   The max_perf_pct controls the maximum frequency the cpufreq driver is allowed to set as a percentage of the maximum supported CPU frequency. This value applies to all CPUs. You can check the maximum supported frequency in /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq. As a starting point, you can use a percentage that caps all CPUs at the All Cores Turbo frequency. The All Cores Turbo frequency is the frequency that all cores will run at when the cores are all fully occupied.

   Note

   To maximize power savings, set a lower value. Setting a lower value for max_perf_pct limits the maximum CPU frequency, thereby reducing power consumption, but also potentially impacting performance. Experiment with different values and monitor the system’s performance and power consumption to find the optimal setting for your use-case.

2. Commit the PolicyGenTemplate change in Git, and then push to the Git repository being monitored by the GitOps ZTP Argo CD application.

Procedure

1. Apply the following PolicyGenTemplate changes to group-du-3node-ranGen.yaml, group-du-sno-ranGen.yaml, or group-du-standard-ranGen.yaml files according to your requirements:

   1. In .sourceFiles, add the PtpOperatorConfig CR file that configures the transport host:

      - fileName: PtpOperatorConfigForEvent.yaml
        policyName: "config-policy"
        spec:
          daemonNodeSelector: {}
          ptpEventConfig:
            enableEventPublisher: true
            transportHost: http://ptp-event-publisher-service-NODE_NAME.openshift-ptp.svc.cluster.local:9043

      Note

      In OpenShift Container Platform 4.13 or later, you do not need to set the transportHost field in the PtpOperatorConfig resource when you use HTTP transport with PTP events.

   2. Configure the linuxptp and phc2sys for the PTP clock type and interface. For example, add the following stanza into .sourceFiles:

      - fileName: PtpConfigSlave.yaml 1
        policyName: "config-policy"
        metadata:
          name: "du-ptp-slave"
        spec:
          profile:
          - name: "slave"
            interface: "ens5f1" 2
            ptp4lOpts: "-2 -s --summary_interval -4" 3
            phc2sysOpts: "-a -r -m -n 24 -N 8 -R 16" 4
          ptpClockThreshold: 5
            holdOverTimeout: 30 #secs
            maxOffsetThreshold: 100  #nano secs
            minOffsetThreshold: -100 #nano secs

      1

      Can be PtpConfigMaster.yaml or PtpConfigSlave.yaml depending on your requirements. For configurations based on group-du-sno-ranGen.yaml or group-du-3node-ranGen.yaml, use PtpConfigSlave.yaml.

      2

      Device specific interface name.

      3

      You must append the --summary_interval -4 value to ptp4lOpts in .spec.sourceFiles.spec.profile to enable PTP fast events.

      4

      Required phc2sysOpts values. -m prints messages to stdout. The linuxptp-daemon DaemonSet parses the logs and generates Prometheus metrics.

      5

      Optional. If the ptpClockThreshold stanza is not present, default values are used for the ptpClockThreshold fields. The stanza shows default ptpClockThreshold values. The ptpClockThreshold values configure how long after the PTP master clock is disconnected before PTP events are triggered. holdOverTimeout is the time value in seconds before the PTP clock event state changes to FREERUN when the PTP master clock is disconnected. The maxOffsetThreshold and minOffsetThreshold settings configure offset values in nanoseconds that compare against the values for CLOCK_REALTIME (phc2sys) or master offset (ptp4l). When the ptp4l or phc2sys offset value is outside this range, the PTP clock state is set to FREERUN. When the offset value is within this range, the PTP clock state is set to LOCKED.

2. Merge any other required changes and files with your custom site repository.
3. Push the changes to your site configuration repository to deploy PTP fast events to new sites using GitOps ZTP.

Procedure

1. Add the following YAML into .spec.sourceFiles in the common-ranGen.yaml file to configure the AMQP Operator:

   #AMQ interconnect operator for fast events
   - fileName: AmqSubscriptionNS.yaml
     policyName: "subscriptions-policy"
   - fileName: AmqSubscriptionOperGroup.yaml
     policyName: "subscriptions-policy"
   - fileName: AmqSubscription.yaml
     policyName: "subscriptions-policy"

2. Apply the following PolicyGenTemplate changes to group-du-3node-ranGen.yaml, group-du-sno-ranGen.yaml, or group-du-standard-ranGen.yaml files according to your requirements:

   1. In .sourceFiles, add the PtpOperatorConfig CR file that configures the AMQ transport host to the config-policy:

      - fileName: PtpOperatorConfigForEvent.yaml
        policyName: "config-policy"
        spec:
          daemonNodeSelector: {}
          ptpEventConfig:
            enableEventPublisher: true
            transportHost: "amqp://amq-router.amq-router.svc.cluster.local"

   2. Configure the linuxptp and phc2sys for the PTP clock type and interface. For example, add the following stanza into .sourceFiles:

      - fileName: PtpConfigSlave.yaml 1
        policyName: "config-policy"
        metadata:
          name: "du-ptp-slave"
        spec:
          profile:
          - name: "slave"
            interface: "ens5f1" 2
            ptp4lOpts: "-2 -s --summary_interval -4" 3
            phc2sysOpts: "-a -r -m -n 24 -N 8 -R 16" 4
          ptpClockThreshold: 5
            holdOverTimeout: 30 #secs
            maxOffsetThreshold: 100  #nano secs
            minOffsetThreshold: -100 #nano secs

      1

      Can be PtpConfigMaster.yaml or PtpConfigSlave.yaml depending on your requirements. For configurations based on group-du-sno-ranGen.yaml or group-du-3node-ranGen.yaml, use PtpConfigSlave.yaml.

      2

      Device specific interface name.

      3

      You must append the --summary_interval -4 value to ptp4lOpts in .spec.sourceFiles.spec.profile to enable PTP fast events.

      4

      Required phc2sysOpts values. -m prints messages to stdout. The linuxptp-daemon DaemonSet parses the logs and generates Prometheus metrics.

      5

      Optional. If the ptpClockThreshold stanza is not present, default values are used for the ptpClockThreshold fields. The stanza shows default ptpClockThreshold values. The ptpClockThreshold values configure how long after the PTP master clock is disconnected before PTP events are triggered. holdOverTimeout is the time value in seconds before the PTP clock event state changes to FREERUN when the PTP master clock is disconnected. The maxOffsetThreshold and minOffsetThreshold settings configure offset values in nanoseconds that compare against the values for CLOCK_REALTIME (phc2sys) or master offset (ptp4l). When the ptp4l or phc2sys offset value is outside this range, the PTP clock state is set to FREERUN. When the offset value is within this range, the PTP clock state is set to LOCKED.

3. Apply the following PolicyGenTemplate changes to your specific site YAML files, for example, example-sno-site.yaml:

   1. In .sourceFiles, add the Interconnect CR file that configures the AMQ router to the config-policy:

      - fileName: AmqInstance.yaml
        policyName: "config-policy"

4. Merge any other required changes and files with your custom site repository.
5. Push the changes to your site configuration repository to deploy PTP fast events to new sites using GitOps ZTP.

Procedure

1. Configure the Bare Metal Event Relay Operator by adding the following YAML to spec.sourceFiles in the common-ranGen.yaml file:

   # Bare Metal Event Relay operator
   - fileName: BareMetalEventRelaySubscriptionNS.yaml
     policyName: "subscriptions-policy"
   - fileName: BareMetalEventRelaySubscriptionOperGroup.yaml
     policyName: "subscriptions-policy"
   - fileName: BareMetalEventRelaySubscription.yaml
     policyName: "subscriptions-policy"

2. Add the HardwareEvent CR to spec.sourceFiles in your specific group configuration file, for example, in the group-du-sno-ranGen.yaml file:

   - fileName: HardwareEvent.yaml 1
     policyName: "config-policy"
     spec:
       nodeSelector: {}
       transportHost: "http://hw-event-publisher-service.openshift-bare-metal-events.svc.cluster.local:9043"
       logLevel: "info"

   1

   Each baseboard management controller (BMC) requires a single HardwareEvent CR only.

   Note

   In OpenShift Container Platform 4.13 or later, you do not need to set the transportHost field in the HardwareEvent custom resource (CR) when you use HTTP transport with bare-metal events.

3. Merge any other required changes and files with your custom site repository.
4. Push the changes to your site configuration repository to deploy bare-metal events to new sites with GitOps ZTP.

5. Create the Redfish Secret by running the following command:

   $ oc -n openshift-bare-metal-events create secret generic redfish-basic-auth \
   --from-literal=username=<bmc_username> --from-literal=password=<bmc_password> \
   --from-literal=hostaddr="<bmc_host_ip_addr>"

Procedure

1. To configure the AMQ Interconnect Operator and the Bare Metal Event Relay Operator, add the following YAML to spec.sourceFiles in the common-ranGen.yaml file:

   # AMQ interconnect operator for fast events
   - fileName: AmqSubscriptionNS.yaml
     policyName: "subscriptions-policy"
   - fileName: AmqSubscriptionOperGroup.yaml
     policyName: "subscriptions-policy"
   - fileName: AmqSubscription.yaml
     policyName: "subscriptions-policy"
   # Bare Metal Event Rely operator
   - fileName: BareMetalEventRelaySubscriptionNS.yaml
     policyName: "subscriptions-policy"
   - fileName: BareMetalEventRelaySubscriptionOperGroup.yaml
     policyName: "subscriptions-policy"
   - fileName: BareMetalEventRelaySubscription.yaml
     policyName: "subscriptions-policy"

2. Add the Interconnect CR to .spec.sourceFiles in the site configuration file, for example, the example-sno-site.yaml file:

   - fileName: AmqInstance.yaml
     policyName: "config-policy"

3. Add the HardwareEvent CR to spec.sourceFiles in your specific group configuration file, for example, in the group-du-sno-ranGen.yaml file:

   - fileName: HardwareEvent.yaml
     policyName: "config-policy"
     spec:
       nodeSelector: {}
       transportHost: "amqp://<amq_interconnect_name>.<amq_interconnect_namespace>.svc.cluster.local" 1
       logLevel: "info"

   1

   The transportHost URL is composed of the existing AMQ Interconnect CR name and namespace. For example, in transportHost: "amqp://amq-router.amq-router.svc.cluster.local", the AMQ Interconnect name and namespace are both set to amq-router.

   Note

   Each baseboard management controller (BMC) requires a single HardwareEvent resource only.

4. Commit the PolicyGenTemplate change in Git, and then push the changes to your site configuration repository to deploy bare-metal events monitoring to new sites using GitOps ZTP.

5. Create the Redfish Secret by running the following command:

   $ oc -n openshift-bare-metal-events create secret generic redfish-basic-auth \
   --from-literal=username=<bmc_username> --from-literal=password=<bmc_password> \
   --from-literal=hostaddr="<bmc_host_ip_addr>"

Procedure

1. Create the storage.bu file by using the following example YAML file:

   variant: fcos
   version: 1.3.0
   storage:
     disks:
     - device: /dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0 1
       wipe_table: false
       partitions:
       - label: var-lib-containers
         start_mib: <start_of_partition> 2
         size_mib: <partition_size> 3
     filesystems:
       - path: /var/lib/containers
         device: /dev/disk/by-partlabel/var-lib-containers
         format: xfs
         wipe_filesystem: true
         with_mount_unit: true
         mount_options:
           - defaults
           - prjquota

   1

   Specify the root disk.

   2

   Specify the start of the partition in MiB. If the value is too small, the installation fails.

   3

   Specify the size of the partition. If the value is too small, the deployments fails.

2. Convert the storage.bu to an Ignition file by running the following command:

   $ butane storage.bu

   Example output

   {"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# # Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}

3. Use a tool such as JSON Pretty Print to convert the output into JSON format.

4. Copy the output into the .spec.clusters.nodes.ignitionConfigOverride field in the SiteConfig CR.

   Example

   [...]
   spec:
     clusters:
       - nodes:
           - ignitionConfigOverride: |
             {
               "ignition": {
                 "version": "3.2.0"
               },
               "storage": {
                 "disks": [
                   {
                     "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0",
                     "partitions": [
                       {
                         "label": "var-lib-containers",
                         "sizeMiB": 0,
                         "startMiB": 250000
                       }
                     ],
                     "wipeTable": false
                   }
                 ],
                 "filesystems": [
                   {
                     "device": "/dev/disk/by-partlabel/var-lib-containers",
                     "format": "xfs",
                     "mountOptions": [
                       "defaults",
                       "prjquota"
                     ],
                     "path": "/var/lib/containers",
                     "wipeFilesystem": true
                   }
                 ]
               },
               "systemd": {
                 "units": [
                   {
                     "contents": "# # Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                     "enabled": true,
                     "name": "var-lib-containers.mount"
                   }
                 ]
               }
             }
   [...]

   Note

   If the .spec.clusters.nodes.ignitionConfigOverride field does not exist, create it.

Verification

1. During or after installation, verify on the hub cluster that the BareMetalHost object shows the annotation by running the following command:

   $ oc get bmh -n my-sno-ns my-sno -ojson | jq '.metadata.annotations["bmac.agent-install.openshift.io/ignition-config-overrides"]

   Example output

   "{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62\",\"partitions\":[{\"label\":\"var-lib-containers\",\"sizeMiB\":0,\"startMiB\":250000}],\"wipeTable\":false}],\"filesystems\":[{\"device\":\"/dev/disk/by-partlabel/var-lib-containers\",\"format\":\"xfs\",\"mountOptions\":[\"defaults\",\"prjquota\"],\"path\":\"/var/lib/containers\",\"wipeFilesystem\":true}]},\"systemd\":{\"units\":[{\"contents\":\"# Generated by Butane\\n[Unit]\\nRequires=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\nAfter=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\n\\n[Mount]\\nWhere=/var/lib/containers\\nWhat=/dev/disk/by-partlabel/var-lib-containers\\nType=xfs\\nOptions=defaults,prjquota\\n\\n[Install]\\nRequiredBy=local-fs.target\",\"enabled\":true,\"name\":\"var-lib-containers.mount\"}]}}"

2. After installation, check the single-node OpenShift disk status.

   1. Enter into a debug session on the single-node OpenShift node by running the following command.

      This step instantiates a debug pod called <node_name>-debug:

      $ oc debug node/my-sno-node

      1. Set /host as the root directory within the debug shell by running the following command.

         The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

         # chroot /host

      2. List information about all available block devices by running the following command:

         # lsblk

         Example output

         NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
         sda      8:0    0 446.6G  0 disk
         ├─sda1   8:1    0     1M  0 part
         ├─sda2   8:2    0   127M  0 part
         ├─sda3   8:3    0   384M  0 part /boot
         ├─sda4   8:4    0 243.6G  0 part /var
         │                                /sysroot/ostree/deploy/rhcos/var
         │                                /usr
         │                                /etc
         │                                /
         │                                /sysroot
         └─sda5   8:5    0 202.5G  0 part /var/lib/containers

      3. Display information about the file system disk space usage by running the following command:

         # df -h

         Example output

         Filesystem      Size  Used Avail Use% Mounted on
         devtmpfs        4.0M     0  4.0M   0% /dev
         tmpfs           126G   84K  126G   1% /dev/shm
         tmpfs            51G   93M   51G   1% /run
         /dev/sda4       244G  5.2G  239G   3% /sysroot
         tmpfs           126G  4.0K  126G   1% /tmp
         /dev/sda5       203G  119G   85G  59% /var/lib/containers
         /dev/sda3       350M  110M  218M  34% /boot
         tmpfs            26G     0   26G   0% /run/user/1000

Procedure

1. Configure the storage class, persistent volume claim, persistent volume, and image registry configuration in the appropriate PolicyGenTemplate CR. For example, to configure an individual site, add the following YAML to the file example-sno-site.yaml:

   sourceFiles:
     # storage class
     - fileName: StorageClass.yaml
       policyName: "sc-for-image-registry"
       metadata:
         name: image-registry-sc
         annotations:
           ran.openshift.io/ztp-deploy-wave: "100" 1
     # persistent volume claim
     - fileName: StoragePVC.yaml
       policyName: "pvc-for-image-registry"
       metadata:
         name: image-registry-pvc
         namespace: openshift-image-registry
         annotations:
           ran.openshift.io/ztp-deploy-wave: "100"
       spec:
         accessModes:
           - ReadWriteMany
         resources:
           requests:
             storage: 100Gi
         storageClassName: image-registry-sc
         volumeMode: Filesystem
     # persistent volume
     - fileName: ImageRegistryPV.yaml 2
       policyName: "pv-for-image-registry"
       metadata:
         annotations:
           ran.openshift.io/ztp-deploy-wave: "100"
     - fileName: ImageRegistryConfig.yaml
       policyName: "config-for-image-registry"
       complianceType: musthave
       metadata:
         annotations:
           ran.openshift.io/ztp-deploy-wave: "100"
       spec:
         storage:
           pvc:
             claim: "image-registry-pvc"

   1

   Set the appropriate value for ztp-deploy-wave depending on whether you are configuring image registries at the site, common, or group level. ztp-deploy-wave: "100" is suitable for development or testing because it allows you to group the referenced source files together.

   2

   In ImageRegistryPV.yaml, ensure that the spec.local.path field is set to /var/imageregistry to match the value set for the mount_point field in the SiteConfig CR.

   Important

   Do not set complianceType: mustonlyhave for the - fileName: ImageRegistryConfig.yaml configuration. This can cause the registry pod deployment to fail.

2. Commit the PolicyGenTemplate change in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application.

Procedure

1. Create three ConfigMap CRs that contain the group and site configuration:

   1. Create a ConfigMap CR named group-hardware-types-configmap to hold the hardware-specific configuration. For example:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: group-hardware-types-configmap
        namespace: ztp-group
        annotations:
          argocd.argoproj.io/sync-options: Replace=true 1
      data:
        # SriovNetworkNodePolicy.yaml
        hardware-type-1-sriov-node-policy-pfNames-1: "[\"ens5f0\"]"
        hardware-type-1-sriov-node-policy-pfNames-2: "[\"ens7f0\"]"
        # PerformanceProfile.yaml
        hardware-type-1-cpu-isolated: "2-31,34-63"
        hardware-type-1-cpu-reserved: "0-1,32-33"
        hardware-type-1-hugepages-default: "1G"
        hardware-type-1-hugepages-size: "1G"
        hardware-type-1-hugepages-count: "32"

      1

      The argocd.argoproj.io/sync-options annotation is required only if the ConfigMap is larger than 1 MiB in size.

   2. Create a ConfigMap CR named group-zones-configmap to hold the regional configuration. For example:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: group-zones-configmap
        namespace: ztp-group
      data:
        # ClusterLogForwarder.yaml
        zone-1-cluster-log-fwd-outputs: "[{\"type\":\"kafka\", \"name\":\"kafka-open\", \"url\":\"tcp://10.46.55.190:9092/test\"}]"
        zone-1-cluster-log-fwd-pipelines: "[{\"inputRefs\":[\"audit\", \"infrastructure\"], \"labels\": {\"label1\": \"test1\", \"label2\": \"test2\", \"label3\": \"test3\", \"label4\": \"test4\"}, \"name\": \"all-to-default\", \"outputRefs\": [\"kafka-open\"]}]"

   3. Create a ConfigMap CR named site-data-configmap to hold the site-specific configuration. For example:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: site-data-configmap
        namespace: ztp-group
      data:
        # SriovNetwork.yaml
        du-sno-1-zone-1-sriov-network-vlan-1: "140"
        du-sno-1-zone-1-sriov-network-vlan-2: "150"

   Note

   Each ConfigMap CR must be in the same namespace as the policy to be generated from the group PolicyGenTemplate CR.

2. Commit the ConfigMap CRs in Git, and then push to the Git repository being monitored by the Argo CD application.

3. Apply the hardware type and region labels to the clusters. The following command applies to a single cluster named du-sno-1-zone-1 and the labels chosen are "hardware-type": "hardware-type-1" and "group-du-sno-zone": "zone-1":

   $ oc patch managedclusters.cluster.open-cluster-management.io/du-sno-1-zone-1 --type merge -p '{"metadata":{"labels":{"hardware-type": "hardware-type-1", "group-du-sno-zone": "zone-1"}}}'

4. Create a group PolicyGenTemplate CR that uses hub templates to obtain the required data from the ConfigMap objects. This example PolicyGenTemplate CR configures logging, VLAN IDs, NICs and Performance Profile for the clusters that match the labels listed under spec.bindingRules:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: group-du-sno-pgt
     namespace: ztp-group
   spec:
     bindingRules:
       # These policies will correspond to all clusters with these labels
       group-du-sno-zone: "zone-1"
       hardware-type: "hardware-type-1"
     mcp: "master"
     sourceFiles:
       - fileName: ClusterLogForwarder.yaml # wave 10
         policyName: "group-du-sno-cfg-policy"
         spec:
           outputs: '{{hub fromConfigMap "" "group-zones-configmap" (printf "%s-cluster-log-fwd-outputs" (index .ManagedClusterLabels "group-du-sno-zone")) | toLiteral hub}}'
           pipelines: '{{hub fromConfigMap "" "group-zones-configmap" (printf "%s-cluster-log-fwd-pipelines" (index .ManagedClusterLabels "group-du-sno-zone")) | toLiteral hub}}'
   
       - fileName: PerformanceProfile.yaml # wave 10
         policyName: "group-du-sno-cfg-policy"
         metadata:
           name: openshift-node-performance-profile
         spec:
           additionalKernelArgs:
           - rcupdate.rcu_normal_after_boot=0
           - vfio_pci.enable_sriov=1
           - vfio_pci.disable_idle_d3=1
           - efi=runtime
           cpu:
             isolated: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-cpu-isolated" (index .ManagedClusterLabels "hardware-type")) hub}}'
             reserved: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-cpu-reserved" (index .ManagedClusterLabels "hardware-type")) hub}}'
           hugepages:
             defaultHugepagesSize: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-hugepages-default" (index .ManagedClusterLabels "hardware-type")) hub}}'
             pages:
               - size: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-hugepages-size" (index .ManagedClusterLabels "hardware-type")) hub}}'
                 count: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-hugepages-count" (index .ManagedClusterLabels "hardware-type")) | toInt hub}}'
           realTimeKernel:
             enabled: true
   
       - fileName: SriovNetwork.yaml # wave 100
         policyName: "group-du-sno-sriov-policy"
         metadata:
           name: sriov-nw-du-fh
         spec:
           resourceName: du_fh
           vlan: '{{hub fromConfigMap "" "site-data-configmap" (printf "%s-sriov-network-vlan-1" .ManagedClusterName) | toInt hub}}'
   
       - fileName: SriovNetworkNodePolicy.yaml # wave 100
         policyName: "group-du-sno-sriov-policy"
         metadata:
           name: sriov-nnp-du-fh
         spec:
           deviceType: netdevice
           isRdma: false
           nicSelector:
             pfNames: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-sriov-node-policy-pfNames-1" (index .ManagedClusterLabels "hardware-type")) | toLiteral hub}}'
           numVfs: 8
           priority: 10
           resourceName: du_fh
   
       - fileName: SriovNetwork.yaml # wave 100
         policyName: "group-du-sno-sriov-policy"
         metadata:
           name: sriov-nw-du-mh
         spec:
           resourceName: du_mh
           vlan: '{{hub fromConfigMap "" "site-data-configmap" (printf "%s-sriov-network-vlan-2" .ManagedClusterName) | toInt hub}}'
   
       - fileName: SriovNetworkNodePolicy.yaml # wave 100
         policyName: "group-du-sno-sriov-policy"
         metadata:
           name: sriov-nw-du-fh
         spec:
           deviceType: netdevice
           isRdma: false
           nicSelector:
             pfNames: '{{hub fromConfigMap "" "group-hardware-types-configmap" (printf "%s-sriov-node-policy-pfNames-2" (index .ManagedClusterLabels "hardware-type")) | toLiteral hub}}'
           numVfs: 8
           priority: 10
           resourceName: du_fh

   Note

   To retrieve site-specific configuration values, use the .ManagedClusterName field. This is a template context value set to the name of the target managed cluster.

   To retrieve group-specific configuration, use the .ManagedClusterLabels field. This is a template context value set to the value of the managed cluster’s labels.

5. Commit the site PolicyGenTemplate CR in Git and push to the Git repository that is monitored by the ArgoCD application.

   Note

   Subsequent changes to the referenced ConfigMap CR are not automatically synced to the applied policies. You need to manually sync the new ConfigMap changes to update existing PolicyGenTemplate CRs. See "Syncing new ConfigMap changes to existing PolicyGenTemplate CRs".

   You can use the same PolicyGenTemplate CR for multiple clusters. If there is a configuration change, then the only modifications you need to make are to the ConfigMap objects that hold the configuration for each cluster and the labels of the managed clusters.

Procedure

1. Update the contents of your ConfigMap CR, and apply the changes in the hub cluster.

2. To sync the contents of the updated ConfigMap CR to the deployed policy, do either of the following:

   1. Option 1: Delete the existing policy. ArgoCD uses the PolicyGenTemplate CR to immediately recreate the deleted policy. For example, run the following command:

      $ oc delete policy <policy_name> -n <policy_namespace>

   2. Option 2: Apply a special annotation policy.open-cluster-management.io/trigger-update to the policy with a different value every time when you update the ConfigMap. For example:

      $ oc annotate policy <policy_name> -n <policy_namespace> policy.open-cluster-management.io/trigger-update="1"

      Note

      You must apply the updated policy for the changes to take effect. For more information, see Special annotation for reprocessing.

3. Optional: If it exists, delete the ClusterGroupUpdate CR that contains the policy. For example:

   $ oc delete clustergroupupgrade <cgu_name> -n <cgu_namespace>

   1. Create a new ClusterGroupUpdate CR that includes the policy to apply with the updated ConfigMap changes. For example, add the following YAML to the file cgr-example.yaml:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: <cgr_name>
        namespace: <policy_namespace>
      spec:
        managedPolicies:
          - <managed_policy>
        enable: true
        clusters:
        - <managed_cluster_1>
        - <managed_cluster_2>
        remediationStrategy:
          maxConcurrency: 2
          timeout: 240

   2. Apply the updated policy:

      $ oc apply -f cgr-example.yaml