Chapter 9. Managing cluster polices with PolicyGenerator resources

Procedure

1. Create a PolicyGenerator CR for site-specific configuration CRs.

   1. Choose the appropriate example for your CR from the out/argocd/example/acmpolicygenerator/ folder, for example, acm-example-sno-site.yaml or acm-example-multinode-site.yaml.

   2. Change the policyDefaults.placement.labelSelector field in the example file to match the site-specific label included in the SiteConfig CR. In the example SiteConfig file, the site-specific label is sites: example-sno.

      Note

      Ensure that the labels defined in your PolicyGenerator policyDefaults.placement.labelSelector field correspond to the labels that are defined in the related managed clusters SiteConfig CR.

   3. Change the content in the example file to match the desired configuration.

2. Optional: Create a PolicyGenerator CR for any common configuration CRs that apply to the entire fleet of clusters.

   1. Select the appropriate example for your CR from the out/argocd/example/acmpolicygenerator/ folder, for example, acm-common-ranGen.yaml.
   2. Change the content in the example file to match the required configuration.

3. Optional: Create a PolicyGenerator CR for any group configuration CRs that apply to the certain groups of clusters in the fleet.

   Ensure that the content of the overlaid spec files matches your required end state. As a reference, the out/source-crs directory contains the full list of source-crs available to be included and overlaid by your PolicyGenerator templates.

   Note

   Depending on the specific requirements of your clusters, you might need more than a single group policy per cluster type, especially considering that the example group policies each have a single PerformancePolicy.yaml file that can only be shared across a set of clusters if those clusters consist of identical hardware configurations.

   1. Select the appropriate example for your CR from the out/argocd/example/acmpolicygenerator/ folder, for example, acm-group-du-sno-ranGen.yaml.
   2. Change the content in the example file to match the required configuration.
4. Optional. Create a validator inform policy PolicyGenerator CR to signal when the GitOps ZTP installation and configuration of the deployed cluster is complete. For more information, see "Creating a validator inform policy".

5. Define all the policy namespaces in a YAML file similar to the example out/argocd/example/acmpolicygenerator//ns.yaml file.

   Important

   Do not include the Namespace CR in the same file with the PolicyGenerator CR.

6. Add the PolicyGenerator CRs and Namespace CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/acmpolicygenerator/kustomization.yaml.

7. Commit the PolicyGenerator CRs, Namespace CR, and associated kustomization.yaml file in your Git repository and push the changes.

   The ArgoCD pipeline detects the changes and begins the managed cluster deployment. You can push the changes to the SiteConfig CR and the PolicyGenerator CR simultaneously.

Procedure

1. The Topology Aware Lifecycle Manager (TALM) applies the configuration policies that are bound to the cluster.

   After the cluster installation is complete and the cluster becomes Ready, a ClusterGroupUpgrade CR corresponding to this cluster, with a list of ordered policies defined by the ran.openshift.io/ztp-deploy-wave annotations, is automatically created by the TALM. The cluster’s policies are applied in the order listed in ClusterGroupUpgrade CR.

   You can monitor the high-level progress of configuration policy reconciliation by using the following commands:

   $ export CLUSTER=<clusterName>

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER -o jsonpath='{.status.conditions[-1:]}' | jq

   Example output

   {
     "lastTransitionTime": "2022-11-09T07:28:09Z",
     "message": "Remediating non-compliant policies",
     "reason": "InProgress",
     "status": "True",
     "type": "Progressing"
   }

2. You can monitor the detailed cluster policy compliance status by using the RHACM dashboard or the command line.

   1. To check policy compliance by using oc, run the following command:

      $ oc get policies -n $CLUSTER

      Example output

      NAME                                                     REMEDIATION ACTION   COMPLIANCE STATE   AGE
      ztp-common.common-config-policy                          inform               Compliant          3h42m
      ztp-common.common-subscriptions-policy                   inform               NonCompliant       3h42m
      ztp-group.group-du-sno-config-policy                     inform               NonCompliant       3h42m
      ztp-group.group-du-sno-validator-du-policy               inform               NonCompliant       3h42m
      ztp-install.example1-common-config-policy-pjz9s          enforce              Compliant          167m
      ztp-install.example1-common-subscriptions-policy-zzd9k   enforce              NonCompliant       164m
      ztp-site.example1-config-policy                          inform               NonCompliant       3h42m
      ztp-site.example1-perf-policy                            inform               NonCompliant       3h42m

   2. To check policy status from the RHACM web console, perform the following actions:

      1. Click Governance Find policies.
      2. Click on a cluster policy to check its status.

Procedure

1. To display detailed information about the policies, run the following command:

   $ oc describe -n openshift-gitops application policies

2. Check for Status: Conditions: to show the error logs. For example, setting an invalid sourceFile entry to fileName: generates the error shown below:

   Status:
     Conditions:
       Last Transition Time:  2021-11-26T17:21:39Z
       Message:               rpc error: code = Unknown desc = `kustomize build /tmp/https___git.com/ran-sites/policies/ --enable-alpha-plugins` failed exit status 1: 2021/11/26 17:21:40 Error could not find test.yaml under source-crs/: no such file or directory Error: failure in plugin configured via /tmp/kust-plugin-config-52463179; exit status 1: exit status 1
       Type:  ComparisonError

3. Check for Status: Sync:. If there are log errors at Status: Conditions:, the Status: Sync: shows Unknown or Error:

   Status:
     Sync:
       Compared To:
         Destination:
           Namespace:  policies-sub
           Server:     https://kubernetes.default.svc
         Source:
           Path:             policies
           Repo URL:         https://git.com/ran-sites/policies/.git
           Target Revision:  master
       Status:               Error

4. When Red Hat Advanced Cluster Management (RHACM) recognizes that policies apply to a ManagedCluster object, the policy CR objects are applied to the cluster namespace. Check to see if the policies were copied to the cluster namespace:

   $ oc get policy -n $CLUSTER

   Example output:

   NAME                                         REMEDIATION ACTION   COMPLIANCE STATE   AGE
   ztp-common.common-config-policy              inform               Compliant          13d
   ztp-common.common-subscriptions-policy       inform               Compliant          13d
   ztp-group.group-du-sno-config-policy         inform               Compliant          13d
   ztp-group.group-du-sno-validator-du-policy   inform               Compliant          13d
   ztp-site.example-sno-config-policy           inform               Compliant          13d

   RHACM copies all applicable policies into the cluster namespace. The copied policy names have the format: <PolicyGenerator.Namespace>.<PolicyGenerator.Name>-<policyName>.

5. Check the placement rule for any policies not copied to the cluster namespace. The matchSelector in the Placement for those policies should match labels on the ManagedCluster object:

   $ oc get Placement -n $NS

6. Note the Placement name appropriate for the missing policy, common, group, or site, using the following command:

   $ oc get Placement -n $NS <placement_rule_name> -o yaml

   • The status-decisions should include your cluster name.
   • The key-value pair of the matchSelector in the spec must match the labels on your managed cluster.

7. Check the labels on the ManagedCluster object by using the following command:

   $ oc get ManagedCluster $CLUSTER -o jsonpath='{.metadata.labels}' | jq

8. Check to see what policies are compliant by using the following command:

   $ oc get policy -n $CLUSTER

   If the Namespace, OperatorGroup, and Subscription policies are compliant but the Operator configuration policies are not, it is likely that the Operators did not install on the managed cluster. This causes the Operator configuration policies to fail to apply because the CRD is not yet applied to the spoke.

Procedure

1. A ClusterGroupUpgrade CR is generated in the namespace ztp-install by the Topology Aware Lifecycle Manager after the managed cluster becomes Ready:

   $ export CLUSTER=<clusterName>

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER

2. If there are unexpected issues and the policies fail to become complaint within the configured timeout (the default is 4 hours), the status of the ClusterGroupUpgrade CR shows UpgradeTimedOut:

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

3. A ClusterGroupUpgrade CR in the UpgradeTimedOut state automatically restarts its policy reconciliation every hour. If you have changed your policies, you can start a retry immediately by deleting the existing ClusterGroupUpgrade CR. This triggers the automatic creation of a new ClusterGroupUpgrade CR that begins reconciling the policies immediately:

   $ oc delete clustergroupupgrades -n ztp-install $CLUSTER

Procedure

1. Remove the content that you no longer need from the affected CRs. In this example, the disableDrain: false line was removed from the SriovOperatorConfig CR.

   Example CR

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovOperatorConfig
   metadata:
     name: default
     namespace: openshift-sriov-network-operator
   spec:
     configDaemonNodeSelector:
       "node-role.kubernetes.io/$mcp": ""
     disableDrain: true
     enableInjector: true
     enableOperatorWebhook: true

2. Change the complianceType of the affected policies to mustonlyhave in the acm-group-du-sno-ranGen.yaml file.

   Example YAML

   # ...
   policyDefaults:
     complianceType: "mustonlyhave"
   # ...
   policies:
     - name: config-policy
       policyAnnotations:
         ran.openshift.io/ztp-deploy-wave: ""
       manifests:
         - path: source-crs/SriovOperatorConfig.yaml

3. Create a ClusterGroupUpdates CR and specify the clusters that must receive the CR changes::

   Example ClusterGroupUpdates CR

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-remove
     namespace: default
   spec:
     managedPolicies:
       - ztp-group.group-du-sno-config-policy
     enable: false
     clusters:
     - spoke1
     - spoke2
     remediationStrategy:
       maxConcurrency: 2
       timeout: 240
     batchTimeoutAction:

4. Create the ClusterGroupUpgrade CR by running the following command:

   $ oc create -f cgu-remove.yaml

5. When you are ready to apply the changes, for example, during an appropriate maintenance window, change the value of the spec.enable field to true by running the following command:

   $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-remove \
   --patch '{"spec":{"enable":true}}' --type=merge

Verification

1. Check the status of the policies by running the following command:

   $ oc get <kind> <changed_cr_name>

   Example output

   NAMESPACE   NAME                                                   REMEDIATION ACTION   COMPLIANCE STATE   AGE
   default     cgu-ztp-group.group-du-sno-config-policy               enforce                                 17m
   default     ztp-group.group-du-sno-config-policy                   inform               NonCompliant       15h

   When the COMPLIANCE STATE of the policy is Compliant, it means that the CR is updated and the unwanted content is removed.

2. Check that the policies are removed from the targeted clusters by running the following command on the managed clusters:

   $ oc get <kind> <changed_cr_name>

   If there are no results, the CR is removed from the managed cluster.

Procedure

1. Review the baseline source CR for existing content. You can review the source CRs listed in the reference PolicyGenerator CRs by extracting them from the GitOps Zero Touch Provisioning (ZTP) container.

   1. Create an /out folder:

      $ mkdir -p ./out

   2. Extract the source CRs:

      $ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.16.1 extract /home/ztp --tar | tar x -C ./out

2. Review the baseline PerformanceProfile CR in ./out/source-crs/PerformanceProfile.yaml:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: $name
     annotations:
       ran.openshift.io/ztp-deploy-wave: "10"
   spec:
     additionalKernelArgs:
     - "idle=poll"
     - "rcupdate.rcu_normal_after_boot=0"
     cpu:
       isolated: $isolated
       reserved: $reserved
     hugepages:
       defaultHugepagesSize: $defaultHugepagesSize
       pages:
         - size: $size
           count: $count
           node: $node
     machineConfigPoolSelector:
       pools.operator.machineconfiguration.openshift.io/$mcp: ""
     net:
       userLevelNetworking: true
     nodeSelector:
       node-role.kubernetes.io/$mcp: ''
     numa:
       topologyPolicy: "restricted"
     realTimeKernel:
       enabled: true

   Note

   Any fields in the source CR which contain $…​ are removed from the generated CR if they are not provided in the PolicyGenerator CR.

3. Update the PolicyGenerator entry for PerformanceProfile in the acm-group-du-sno-ranGen.yaml reference file. The following example PolicyGenerator CR stanza supplies appropriate CPU specifications, sets the hugepages configuration, and adds a new field that sets globallyDisableIrqLoadBalancing to false.

   - path: source-crs/PerformanceProfile.yaml
     patches:
       - spec:
           # These must be tailored for the specific hardware platform
           cpu:
             isolated: "2-19,22-39"
             reserved: "0-1,20-21"
           hugepages:
             defaultHugepagesSize: 1G
             pages:
             - size: 1G
               count: 10
           globallyDisableIrqLoadBalancing: false

4. Commit the PolicyGenerator change in Git, and then push to the Git repository being monitored by the GitOps ZTP argo CD application.

   Example output

   The GitOps ZTP application generates an RHACM policy that contains the generated PerformanceProfile CR. The contents of that CR are derived by merging the metadata and spec contents from the PerformanceProfile entry in the PolicyGenerator onto the source CR. The resulting CR has the following content:

   ---
   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
       name: openshift-node-performance-profile
   spec:
       additionalKernelArgs:
           - idle=poll
           - rcupdate.rcu_normal_after_boot=0
       cpu:
           isolated: 2-19,22-39
           reserved: 0-1,20-21
       globallyDisableIrqLoadBalancing: false
       hugepages:
           defaultHugepagesSize: 1G
           pages:
               - count: 10
                 size: 1G
       machineConfigPoolSelector:
           pools.operator.machineconfiguration.openshift.io/master: ""
       net:
           userLevelNetworking: true
       nodeSelector:
           node-role.kubernetes.io/master: ""
       numa:
           topologyPolicy: restricted
       realTimeKernel:
           enabled: true

Procedure

1. Create a subdirectory named source-crs in the directory that contains the kustomization.yaml file for the PolicyGenerator custom resource (CR).

2. Add your user-provided CRs to the source-crs subdirectory, as shown in the following example:

   example
   └── acmpolicygenerator
       ├── dev.yaml
       ├── kustomization.yaml
       ├── mec-edge-sno1.yaml
       ├── sno.yaml
       └── source-crs 1
           ├── PaoCatalogSource.yaml
           ├── PaoSubscription.yaml
           ├── custom-crs
           |   ├── apiserver-config.yaml
           |   └── disable-nic-lldp.yaml
           └── elasticsearch
               ├── ElasticsearchNS.yaml
               └── ElasticsearchOperatorGroup.yaml

   1

   The source-crs subdirectory must be in the same directory as the kustomization.yaml file.

3. Update the required PolicyGenerator CRs to include references to the content you added in the source-crs/custom-crs and source-crs/elasticsearch directories. For example:

   apiVersion: policy.open-cluster-management.io/v1
   kind: PolicyGenerator
   metadata:
       name: group-dev
   placementBindingDefaults:
       name: group-dev-placement-binding
   policyDefaults:
       namespace: ztp-clusters
       placement:
           labelSelector:
               matchExpressions:
                   - key: dev
                     operator: In
                     values:
                       - "true"
       remediationAction: inform
       severity: low
       namespaceSelector:
           exclude:
               - kube-*
           include:
               - '*'
       evaluationInterval:
           compliant: 10m
           noncompliant: 10s
   policies:
       - name: group-dev-group-dev-cluster-log-ns
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/ClusterLogNS.yaml
       - name: group-dev-group-dev-cluster-log-operator-group
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/ClusterLogOperGroup.yaml
       - name: group-dev-group-dev-cluster-log-sub
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/ClusterLogSubscription.yaml
       - name: group-dev-group-dev-lso-ns
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/StorageNS.yaml
       - name: group-dev-group-dev-lso-operator-group
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/StorageOperGroup.yaml
       - name: group-dev-group-dev-lso-sub
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/StorageSubscription.yaml
       - name: group-dev-group-dev-pao-cat-source
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "1"
         manifests:
           - path: source-crs/PaoSubscriptionCatalogSource.yaml
             patches:
               - spec:
                   image: <container_image_url>
       - name: group-dev-group-dev-pao-ns
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/PaoSubscriptionNS.yaml
       - name: group-dev-group-dev-pao-sub
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: source-crs/PaoSubscription.yaml
       - name: group-dev-group-dev-elasticsearch-ns
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: elasticsearch/ElasticsearchNS.yaml 1
       - name: group-dev-group-dev-elasticsearch-operator-group
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: elasticsearch/ElasticsearchOperatorGroup.yaml
       - name: group-dev-group-dev-apiserver-config
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: custom-crs/apiserver-config.yaml 2
       - name: group-dev-group-dev-disable-nic-lldp
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "2"
         manifests:
           - path: custom-crs/disable-nic-lldp.yaml

   1 2

   Set policies.manifests.path to include the relative path to the file from the /source-crs parent directory.

4. Commit the PolicyGenerator change in Git, and then push to the Git repository that is monitored by the GitOps ZTP Argo CD policies application.

5. Update the ClusterGroupUpgrade CR to include the changed PolicyGenerator and save it as cgu-test.yaml. The following example shows a generated cgu-test.yaml file.

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: custom-source-cr
     namespace: ztp-clusters
   spec:
     managedPolicies:
       - group-dev-config-policy
     enable: true
     clusters:
     - cluster1
     remediationStrategy:
       maxConcurrency: 2
       timeout: 240

6. Apply the updated ClusterGroupUpgrade CR by running the following command:

   $ oc apply -f cgu-test.yaml

Procedure

1. To configure the evaluation interval for all policies in a PolicyGenerator CR, set appropriate compliant and noncompliant values for the evaluationInterval field. For example:

   policyDefaults:
     evaluationInterval:
       compliant: 30m
       noncompliant: 45s

   Note

   You can also set compliant and noncompliant fields to never to stop evaluating the policy after it reaches particular compliance state.

2. To configure the evaluation interval for an individual policy object in a PolicyGenerator CR, add the evaluationInterval field and set appropriate values. For example:

   policies:
     - name: "sriov-sub-policy"
       manifests:
         - path: "SriovSubscription.yaml"
           evaluationInterval:
             compliant: never
             noncompliant: 10s

3. Commit the PolicyGenerator CRs files in the Git repository and push your changes.

1. Log in as a user with cluster-admin privileges on the managed cluster.

2. Get the pods that are running in the open-cluster-management-agent-addon namespace. Run the following command:

   $ oc get pods -n open-cluster-management-agent-addon

   Example output

   NAME                                         READY   STATUS    RESTARTS        AGE
   config-policy-controller-858b894c68-v4xdb    1/1     Running   22 (5d8h ago)   10d

3. Check the applied policies are being evaluated at the expected interval in the logs for the config-policy-controller pod:

   $ oc logs -n open-cluster-management-agent-addon config-policy-controller-858b894c68-v4xdb

   Example output

   2022-05-10T15:10:25.280Z       info   configuration-policy-controller controllers/configurationpolicy_controller.go:166      Skipping the policy evaluation due to the policy not reaching the evaluation interval  {"policy": "compute-1-config-policy-config"}
   2022-05-10T15:10:25.280Z       info   configuration-policy-controller controllers/configurationpolicy_controller.go:166      Skipping the policy evaluation due to the policy not reaching the evaluation interval  {"policy": "compute-1-common-compute-1-catalog-policy-config"}

Procedure

1. Create a standalone PolicyGenerator custom resource (CR) that contains the source file validatorCRs/informDuValidator.yaml. You only need one standalone PolicyGenerator CR for each cluster type. For example, this CR applies a validator inform policy for single-node OpenShift clusters:

   Example single-node cluster validator inform policy CR (acm-group-du-sno-validator-ranGen.yaml)

   apiVersion: policy.open-cluster-management.io/v1
   kind: PolicyGenerator
   metadata:
       name: group-du-sno-validator-latest
   placementBindingDefaults:
       name: group-du-sno-validator-latest-placement-binding
   policyDefaults:
       namespace: ztp-group
       placement:
           labelSelector:
               matchExpressions:
                   - key: du-profile
                     operator: In
                     values:
                       - latest
                   - key: group-du-sno
                     operator: Exists
                   - key: ztp-done
                     operator: DoesNotExist
       remediationAction: inform
       severity: low
       namespaceSelector:
           exclude:
               - kube-*
           include:
               - '*'
       evaluationInterval:
           compliant: 10m
           noncompliant: 10s
   policies:
       - name: group-du-sno-validator-latest-du-policy
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "10000"
         evaluationInterval:
           compliant: 5s
         manifests:
           - path: source-crs/validatorCRs/informDuValidator-MCP-master.yaml

2. Commit the PolicyGenerator CR file in your Git repository and push the changes.

Procedure

1. To configure LVM Storage for new managed clusters, add the following YAML to policies.manifests in the acm-common-ranGen.yaml file:

   - name: subscription-policies
     policyAnnotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     manifests:
       - path: source-crs/StorageLVMOSubscriptionNS.yaml
       - path: source-crs/StorageLVMOSubscriptionOperGroup.yaml
       - path: source-crs/StorageLVMOSubscription.yaml
         spec:
           name: lvms-operator
           channel: stable-4.16

   Note

   The Storage LVMO subscription is deprecated. In future releases of OpenShift Container Platform, the storage LVMO subscription will not be available. Instead, you must use the Storage LVMS subscription.

   In OpenShift Container Platform 4.16, you can use the Storage LVMS subscription instead of the LVMO subscription. The LVMS subscription does not require manual overrides in the acm-common-ranGen.yaml file. Add the following YAML to policies.manifests in the acm-common-ranGen.yaml file to use the Storage LVMS subscription:

   - path: source-crs/StorageLVMSubscriptionNS.yaml
   - path: source-crs/StorageLVMSubscriptionOperGroup.yaml
   - path: source-crs/StorageLVMSubscription.yaml

2. Add the LVMCluster CR to policies.manifests in your specific group or individual site configuration file. For example, in the acm-group-du-sno-ranGen.yaml file, add the following:

   - fileName: StorageLVMCluster.yaml
     policyName: "lvms-config"
       metadata:
         name: "lvms-storage-cluster-config"
           spec:
             storage:
               deviceClasses:
               - name: vg1
                 thinPoolConfig:
                   name: thin-pool-1
                   sizePercent: 90
                   overprovisionRatio: 10

   This example configuration creates a volume group (vg1) with all the available devices, except the disk where OpenShift Container Platform is installed. A thin-pool logical volume is also created.

3. Merge any other required changes and files with your custom site repository.
4. Commit the PolicyGenerator changes in Git, and then push the changes to your site configuration repository to deploy LVM Storage to new sites using GitOps ZTP.

Procedure

1. Create a PolicyGenerator CR for the platform update:

   1. Save the following PolicyGenerator CR in the du-upgrade.yaml file:

      Example of PolicyGenerator for platform update

      apiVersion: policy.open-cluster-management.io/v1
      kind: PolicyGenerator
      metadata:
          name: du-upgrade
      placementBindingDefaults:
          name: du-upgrade-placement-binding
      policyDefaults:
          namespace: ztp-group-du-sno
          placement:
              labelSelector:
                  matchExpressions:
                      - key: group-du-sno
                        operator: Exists
          remediationAction: inform
          severity: low
          namespaceSelector:
              exclude:
                  - kube-*
              include:
                  - '*'
          evaluationInterval:
              compliant: 10m
              noncompliant: 10s
      policies:
          - name: du-upgrade-platform-upgrade
            policyAnnotations:
              ran.openshift.io/ztp-deploy-wave: "100"
            manifests:
              - path: source-crs/ClusterVersion.yaml 1
                patches:
                  - metadata:
                      name: version
                    spec:
                      channel: stable-4.16
                      desiredUpdate:
                          version: 4.16.4
                      upstream: http://upgrade.example.com/images/upgrade-graph_stable-4.16
                    status:
                      history:
                          - state: Completed
                            version: 4.16.4
          - name: du-upgrade-platform-upgrade-prep
            policyAnnotations:
              ran.openshift.io/ztp-deploy-wave: "1"
            manifests:
              - path: source-crs/ImageSignature.yaml 2
              - path: source-crs/DisconnectedICSP.yaml
                patches:
                  - metadata:
                      name: disconnected-internal-icsp-for-ocp
                    spec:
                      repositoryDigestMirrors: 3
                          - mirrors:
                              - quay-intern.example.com/ocp4/openshift-release-dev
                            source: quay.io/openshift-release-dev/ocp-release
                          - mirrors:
                              - quay-intern.example.com/ocp4/openshift-release-dev
                            source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

      1

      Shows the ClusterVersion CR to trigger the update. The channel, upstream, and desiredVersion fields are all required for image pre-caching.

      2

      ImageSignature.yaml contains the image signature of the required release image. The image signature is used to verify the image before applying the platform update.

      3

      Shows the mirror repository that contains the required OpenShift Container Platform image. Get the mirrors from the imageContentSources.yaml file that you saved when following the procedures in the "Setting up the environment" section.

      The PolicyGenerator CR generates two policies:

      • The du-upgrade-platform-upgrade-prep policy does the preparation work for the platform update. It creates the ConfigMap CR for the desired release image signature, creates the image content source of the mirrored release image repository, and updates the cluster version with the desired update channel and the update graph reachable by the managed cluster in the disconnected environment.
      • The du-upgrade-platform-upgrade policy is used to perform platform upgrade.

   2. Add the du-upgrade.yaml file contents to the kustomization.yaml file located in the GitOps ZTP Git repository for the PolicyGenerator CRs and push the changes to the Git repository.

      ArgoCD pulls the changes from the Git repository and generates the policies on the hub cluster.

   3. Check the created policies by running the following command:

      $ oc get policies -A | grep platform-upgrade

2. Create the ClusterGroupUpdate CR for the platform update with the spec.enable field set to false.

   1. Save the content of the platform update ClusterGroupUpdate CR with the du-upgrade-platform-upgrade-prep and the du-upgrade-platform-upgrade policies and the target clusters to the cgu-platform-upgrade.yml file, as shown in the following example:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: cgu-platform-upgrade
        namespace: default
      spec:
        managedPolicies:
        - du-upgrade-platform-upgrade-prep
        - du-upgrade-platform-upgrade
        preCaching: false
        clusters:
        - spoke1
        remediationStrategy:
          maxConcurrency: 1
        enable: false

   2. Apply the ClusterGroupUpdate CR to the hub cluster by running the following command:

      $ oc apply -f cgu-platform-upgrade.yml

3. Optional: Pre-cache the images for the platform update.

   1. Enable pre-caching in the ClusterGroupUpdate CR by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-platform-upgrade \
      --patch '{"spec":{"preCaching": true}}' --type=merge

   2. Monitor the update process and wait for the pre-caching to complete. Check the status of pre-caching by running the following command on the hub cluster:

      $ oc get cgu cgu-platform-upgrade -o jsonpath='{.status.precaching.status}'

4. Start the platform update:

   1. Enable the cgu-platform-upgrade policy and disable pre-caching by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-platform-upgrade \
      --patch '{"spec":{"enable":true, "preCaching": false}}' --type=merge

   2. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command:

      $ oc get policies --all-namespaces

Procedure

1. Update the PolicyGenerator CR for the Operator update.

   1. Update the du-upgrade PolicyGenerator CR with the following additional contents in the du-upgrade.yaml file:

      apiVersion: policy.open-cluster-management.io/v1
      kind: PolicyGenerator
      metadata:
          name: du-upgrade
      placementBindingDefaults:
          name: du-upgrade-placement-binding
      policyDefaults:
          namespace: ztp-group-du-sno
          placement:
              labelSelector:
                  matchExpressions:
                      - key: group-du-sno
                        operator: Exists
          remediationAction: inform
          severity: low
          namespaceSelector:
              exclude:
                  - kube-*
              include:
                  - '*'
          evaluationInterval:
              compliant: 10m
              noncompliant: 10s
      policies:
          - name: du-upgrade-operator-catsrc-policy
            policyAnnotations:
              ran.openshift.io/ztp-deploy-wave: "1"
            manifests:
              - path: source-crs/DefaultCatsrc.yaml
                patches:
                  - metadata:
                      name: redhat-operators
                    spec:
                      displayName: Red Hat Operators Catalog
                      image: registry.example.com:5000/olm/redhat-operators:v4.16 1
                      updateStrategy: 2
                          registryPoll:
                              interval: 1h
                    status:
                      connectionState:
                          lastObservedState: READY 3

      1

      Contains the required Operator images. If the index images are always pushed to the same image name and tag, this change is not needed.

      2

      Sets how frequently the Operator Lifecycle Manager (OLM) polls the index image for new Operator versions with the registryPoll.interval field. This change is not needed if a new index image tag is always pushed for y-stream and z-stream Operator updates. The registryPoll.interval field can be set to a shorter interval to expedite the update, however shorter intervals increase computational load. To counteract this, you can restore registryPoll.interval to the default value once the update is complete.

      3

      Displays the observed state of the catalog connection. The READY value ensures that the CatalogSource policy is ready, indicating that the index pod is pulled and is running. This way, TALM upgrades the Operators based on up-to-date policy compliance states.

   2. This update generates one policy, du-upgrade-operator-catsrc-policy, to update the redhat-operators catalog source with the new index images that contain the desired Operators images.

      Note

      If you want to use the image pre-caching for Operators and there are Operators from a different catalog source other than redhat-operators, you must perform the following tasks:

      • Prepare a separate catalog source policy with the new index image or registry poll interval update for the different catalog source.
      • Prepare a separate subscription policy for the desired Operators that are from the different catalog source.

      For example, the desired SRIOV-FEC Operator is available in the certified-operators catalog source. To update the catalog source and the Operator subscription, add the following contents to generate two policies, du-upgrade-fec-catsrc-policy and du-upgrade-subscriptions-fec-policy:

      apiVersion: policy.open-cluster-management.io/v1
      kind: PolicyGenerator
      metadata:
          name: du-upgrade
      placementBindingDefaults:
          name: du-upgrade-placement-binding
      policyDefaults:
          namespace: ztp-group-du-sno
          placement:
              labelSelector:
                  matchExpressions:
                      - key: group-du-sno
                        operator: Exists
          remediationAction: inform
          severity: low
          namespaceSelector:
              exclude:
                  - kube-*
              include:
                  - '*'
          evaluationInterval:
              compliant: 10m
              noncompliant: 10s
      policies:
          - name: du-upgrade-fec-catsrc-policy
            policyAnnotations:
              ran.openshift.io/ztp-deploy-wave: "1"
            manifests:
              - path: source-crs/DefaultCatsrc.yaml
                patches:
                  - metadata:
                      name: certified-operators
                    spec:
                      displayName: Intel SRIOV-FEC Operator
                      image: registry.example.com:5000/olm/far-edge-sriov-fec:v4.10
                      updateStrategy:
                          registryPoll:
                              interval: 10m
          - name: du-upgrade-subscriptions-fec-policy
            policyAnnotations:
              ran.openshift.io/ztp-deploy-wave: "2"
            manifests:
              - path: source-crs/AcceleratorsSubscription.yaml
                patches:
                  - spec:
                      channel: stable
                      source: certified-operators

   3. Remove the specified subscriptions channels in the common PolicyGenerator CR, if they exist. The default subscriptions channels from the GitOps ZTP image are used for the update.

      Note

      The default channel for the Operators applied through GitOps ZTP 4.16 is stable, except for the performance-addon-operator. As of OpenShift Container Platform 4.11, the performance-addon-operator functionality was moved to the node-tuning-operator. For the 4.10 release, the default channel for PAO is v4.10. You can also specify the default channels in the common PolicyGenerator CR.

   4. Push the PolicyGenerator CRs updates to the GitOps ZTP Git repository.

      ArgoCD pulls the changes from the Git repository and generates the policies on the hub cluster.

   5. Check the created policies by running the following command:

      $ oc get policies -A | grep -E "catsrc-policy|subscription"

2. Apply the required catalog source updates before starting the Operator update.

   1. Save the content of the ClusterGroupUpgrade CR named operator-upgrade-prep with the catalog source policies and the target managed clusters to the cgu-operator-upgrade-prep.yml file:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: cgu-operator-upgrade-prep
        namespace: default
      spec:
        clusters:
        - spoke1
        enable: true
        managedPolicies:
        - du-upgrade-operator-catsrc-policy
        remediationStrategy:
          maxConcurrency: 1

   2. Apply the policy to the hub cluster by running the following command:

      $ oc apply -f cgu-operator-upgrade-prep.yml

   3. Monitor the update process. Upon completion, ensure that the policy is compliant by running the following command:

      $ oc get policies -A | grep -E "catsrc-policy"

3. Create the ClusterGroupUpgrade CR for the Operator update with the spec.enable field set to false.

   1. Save the content of the Operator update ClusterGroupUpgrade CR with the du-upgrade-operator-catsrc-policy policy and the subscription policies created from the common PolicyGenerator and the target clusters to the cgu-operator-upgrade.yml file, as shown in the following example:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: cgu-operator-upgrade
        namespace: default
      spec:
        managedPolicies:
        - du-upgrade-operator-catsrc-policy 1
        - common-subscriptions-policy 2
        preCaching: false
        clusters:
        - spoke1
        remediationStrategy:
          maxConcurrency: 1
        enable: false

      1

      The policy is needed by the image pre-caching feature to retrieve the operator images from the catalog source.

      2

      The policy contains Operator subscriptions. If you have followed the structure and content of the reference PolicyGenTemplates, all Operator subscriptions are grouped into the common-subscriptions-policy policy.

      Note

      One ClusterGroupUpgrade CR can only pre-cache the images of the desired Operators defined in the subscription policy from one catalog source included in the ClusterGroupUpgrade CR. If the desired Operators are from different catalog sources, such as in the example of the SRIOV-FEC Operator, another ClusterGroupUpgrade CR must be created with du-upgrade-fec-catsrc-policy and du-upgrade-subscriptions-fec-policy policies for the SRIOV-FEC Operator images pre-caching and update.

   2. Apply the ClusterGroupUpgrade CR to the hub cluster by running the following command:

      $ oc apply -f cgu-operator-upgrade.yml

4. Optional: Pre-cache the images for the Operator update.

   1. Before starting image pre-caching, verify the subscription policy is NonCompliant at this point by running the following command:

      $ oc get policy common-subscriptions-policy -n <policy_namespace>

      Example output

      NAME                          REMEDIATION ACTION   COMPLIANCE STATE     AGE
      common-subscriptions-policy   inform               NonCompliant         27d

   2. Enable pre-caching in the ClusterGroupUpgrade CR by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-operator-upgrade \
      --patch '{"spec":{"preCaching": true}}' --type=merge

   3. Monitor the process and wait for the pre-caching to complete. Check the status of pre-caching by running the following command on the managed cluster:

      $ oc get cgu cgu-operator-upgrade -o jsonpath='{.status.precaching.status}'

   4. Check if the pre-caching is completed before starting the update by running the following command:

      $ oc get cgu -n default cgu-operator-upgrade -ojsonpath='{.status.conditions}' | jq

      Example output

      [
          {
            "lastTransitionTime": "2022-03-08T20:49:08.000Z",
            "message": "The ClusterGroupUpgrade CR is not enabled",
            "reason": "UpgradeNotStarted",
            "status": "False",
            "type": "Ready"
          },
          {
            "lastTransitionTime": "2022-03-08T20:55:30.000Z",
            "message": "Precaching is completed",
            "reason": "PrecachingCompleted",
            "status": "True",
            "type": "PrecachingDone"
          }
      ]

5. Start the Operator update.

   1. Enable the cgu-operator-upgrade ClusterGroupUpgrade CR and disable pre-caching to start the Operator update by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-operator-upgrade \
      --patch '{"spec":{"enable":true, "preCaching": false}}' --type=merge

   2. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command:

      $ oc get policies --all-namespaces

Procedure

1. Add a catalog source configuration in the PolicyGenerator resource:

   manifests:
   - path: source-crs/DefaultCatsrc.yaml
     patches:
       - metadata:
           name: redhat-operators
         spec:
           displayName: Red Hat Operators Catalog
           image: registry.example.com:5000/olm/redhat-operators:v{product-version}
           updateStrategy:
               registryPoll:
                   interval: 1h
         status:
           connectionState:
               lastObservedState: READY
   - path: source-crs/DefaultCatsrc.yaml
     patches:
       - metadata:
           name: redhat-operators-v2 1
         spec:
           displayName: Red Hat Operators Catalog v2 2
           image: registry.example.com:5000/olredhat-operators:<version> 3
           updateStrategy:
               registryPoll:
                   interval: 1h
         status:
           connectionState:
               lastObservedState: READY

   1

   Update the name for the new configuration.

   2

   Update the display name for the new configuration.

   3

   Update the index image URL. This policies.manifests.patches.spec.image field overrides any configuration in the DefaultCatsrc.yaml file.

2. Update the Subscription resource to point to the new configuration for Operators that require an update:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: operator-subscription
     namespace: operator-namspace
   # ...
   spec:
     source: redhat-operators-v2 1
   # ...

   1

   Enter the name of the additional catalog source configuration that you defined in the PolicyGenerator resource.

Procedure

1. Create the PolicyGenerator CR for the updates by following the steps described in the "Performing a platform update" and "Performing an Operator update" sections.

2. Apply the prep work for the platform and the Operator update.

   1. Save the content of the ClusterGroupUpgrade CR with the policies for platform update preparation work, catalog source updates, and target clusters to the cgu-platform-operator-upgrade-prep.yml file, for example:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: cgu-platform-operator-upgrade-prep
        namespace: default
      spec:
        managedPolicies:
        - du-upgrade-platform-upgrade-prep
        - du-upgrade-operator-catsrc-policy
        clusterSelector:
        - group-du-sno
        remediationStrategy:
          maxConcurrency: 10
        enable: true

   2. Apply the cgu-platform-operator-upgrade-prep.yml file to the hub cluster by running the following command:

      $ oc apply -f cgu-platform-operator-upgrade-prep.yml

   3. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command:

      $ oc get policies --all-namespaces

3. Create the ClusterGroupUpdate CR for the platform and the Operator update with the spec.enable field set to false.

   1. Save the contents of the platform and Operator update ClusterGroupUpdate CR with the policies and the target clusters to the cgu-platform-operator-upgrade.yml file, as shown in the following example:

      apiVersion: ran.openshift.io/v1alpha1
      kind: ClusterGroupUpgrade
      metadata:
        name: cgu-du-upgrade
        namespace: default
      spec:
        managedPolicies:
        - du-upgrade-platform-upgrade 1
        - du-upgrade-operator-catsrc-policy 2
        - common-subscriptions-policy 3
        preCaching: true
        clusterSelector:
        - group-du-sno
        remediationStrategy:
          maxConcurrency: 1
        enable: false

      1

      This is the platform update policy.

      2

      This is the policy containing the catalog source information for the Operators to be updated. It is needed for the pre-caching feature to determine which Operator images to download to the managed cluster.

      3

      This is the policy to update the Operators.

   2. Apply the cgu-platform-operator-upgrade.yml file to the hub cluster by running the following command:

      $ oc apply -f cgu-platform-operator-upgrade.yml

4. Optional: Pre-cache the images for the platform and the Operator update.

   1. Enable pre-caching in the ClusterGroupUpgrade CR by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-du-upgrade \
      --patch '{"spec":{"preCaching": true}}' --type=merge

   2. Monitor the update process and wait for the pre-caching to complete. Check the status of pre-caching by running the following command on the managed cluster:

      $ oc get jobs,pods -n openshift-talm-pre-cache

   3. Check if the pre-caching is completed before starting the update by running the following command:

      $ oc get cgu cgu-du-upgrade -ojsonpath='{.status.conditions}'

5. Start the platform and Operator update.

   1. Enable the cgu-du-upgrade ClusterGroupUpgrade CR to start the platform and the Operator update by running the following command:

      $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-du-upgrade \
      --patch '{"spec":{"enable":true, "preCaching": false}}' --type=merge

   2. Monitor the process. Upon completion, ensure that the policy is compliant by running the following command:

      $ oc get policies --all-namespaces

      Note

      The CRs for the platform and Operator updates can be created from the beginning by configuring the setting to spec.enable: true. In this case, the update starts immediately after pre-caching completes and there is no need to manually enable the CR.

      Both pre-caching and the update create extra resources, such as policies, placement bindings, placement rules, managed cluster actions, and managed cluster view, to help complete the procedures. Setting the afterCompletion.deleteObjects field to true deletes all these resources after the updates complete.

Procedure

1. Change the complianceType to mustnothave for the Performance Addon Operator namespace, Operator group, and subscription in the acm-common-ranGen.yaml file.

   - name: group-du-sno-pg-subscriptions-policy
     policyAnnotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     manifests:
       - path: source-crs/PaoSubscriptionNS.yaml
       - path: source-crs/PaoSubscriptionOperGroup.yaml
       - path: source-crs/PaoSubscription.yaml

2. Merge the changes with your custom site repository and wait for the ArgoCD application to synchronize the change to the hub cluster. The status of the common-subscriptions-policy policy changes to Non-Compliant.
3. Apply the change to your target clusters by using the Topology Aware Lifecycle Manager. For more information about rolling out configuration changes, see the "Additional resources" section.

4. Monitor the process. When the status of the common-subscriptions-policy policy for a target cluster is Compliant, the Performance Addon Operator has been removed from the cluster. Get the status of the common-subscriptions-policy by running the following command:

   $ oc get policy -n ztp-common common-subscriptions-policy

5. Delete the Performance Addon Operator namespace, Operator group and subscription CRs from policies.manifests in the acm-common-ranGen.yaml file.
6. Merge the changes with your custom site repository and wait for the ArgoCD application to synchronize the change to the hub cluster. The policy remains compliant.