Chapter 5. Configuring managed clusters with policies and PolicyGenTemplate resources

Procedure

1. Create a PolicyGenTemplate CR for site-specific configuration CRs.

   1. Choose the appropriate example for your CR from the out/argocd/example/policygentemplates folder, for example, example-sno-site.yaml or example-multinode-site.yaml.

   2. Change the bindingRules field in the example file to match the site-specific label included in the SiteConfig CR. In the example SiteConfig file, the site-specific label is sites: example-sno.

      Note

      Ensure that the labels defined in your PolicyGenTemplate bindingRules field correspond to the labels that are defined in the related managed clusters SiteConfig CR.

   3. Change the content in the example file to match the desired configuration.

2. Optional: Create a PolicyGenTemplate CR for any common configuration CRs that apply to the entire fleet of clusters.

   1. Select the appropriate example for your CR from the out/argocd/example/policygentemplates folder, for example, common-ranGen.yaml.
   2. Change the content in the example file to match the desired configuration.

3. Optional: Create a PolicyGenTemplate CR for any group configuration CRs that apply to the certain groups of clusters in the fleet.

   Ensure that the content of the overlaid spec files matches your desired end state. As a reference, the out/source-crs directory contains the full list of source-crs available to be included and overlaid by your PolicyGenTemplate templates.

   Note

   Depending on the specific requirements of your clusters, you might need more than a single group policy per cluster type, especially considering that the example group policies each have a single PerformancePolicy.yaml file that can only be shared across a set of clusters if those clusters consist of identical hardware configurations.

   1. Select the appropriate example for your CR from the out/argocd/example/policygentemplates folder, for example, group-du-sno-ranGen.yaml.
   2. Change the content in the example file to match the desired configuration.
4. Optional. Create a validator inform policy PolicyGenTemplate CR to signal when the GitOps ZTP installation and configuration of the deployed cluster is complete. For more information, see "Creating a validator inform policy".

5. Define all the policy namespaces in a YAML file similar to the example out/argocd/example/policygentemplates/ns.yaml file.

   Important

   Do not include the Namespace CR in the same file with the PolicyGenTemplate CR.

6. Add the PolicyGenTemplate CRs and Namespace CR to the kustomization.yaml file in the generators section, similar to the example shown in out/argocd/example/policygentemplates/kustomization.yaml.

7. Commit the PolicyGenTemplate CRs, Namespace CR, and associated kustomization.yaml file in your Git repository and push the changes.

   The ArgoCD pipeline detects the changes and begins the managed cluster deployment. You can push the changes to the SiteConfig CR and the PolicyGenTemplate CR simultaneously.

Procedure

1. The Topology Aware Lifecycle Manager (TALM) applies the configuration policies that are bound to the cluster.

   After the cluster installation is complete and the cluster becomes Ready, a ClusterGroupUpgrade CR corresponding to this cluster, with a list of ordered policies defined by the ran.openshift.io/ztp-deploy-wave annotations, is automatically created by the TALM. The cluster’s policies are applied in the order listed in ClusterGroupUpgrade CR.

   You can monitor the high-level progress of configuration policy reconciliation by using the following commands:

   $ export CLUSTER=<clusterName>

   Copy to Clipboard Toggle word wrap

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER -o jsonpath='{.status.conditions[-1:]}' | jq

   Copy to Clipboard Toggle word wrap

   Example output

   {
     "lastTransitionTime": "2022-11-09T07:28:09Z",
     "message": "Remediating non-compliant policies",
     "reason": "InProgress",
     "status": "True",
     "type": "Progressing"
   }

   Copy to Clipboard Toggle word wrap

2. You can monitor the detailed cluster policy compliance status by using the RHACM dashboard or the command line.

   1. To check policy compliance by using oc, run the following command:

      $ oc get policies -n $CLUSTER

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                                     REMEDIATION ACTION   COMPLIANCE STATE   AGE
      ztp-common.common-config-policy                          inform               Compliant          3h42m
      ztp-common.common-subscriptions-policy                   inform               NonCompliant       3h42m
      ztp-group.group-du-sno-config-policy                     inform               NonCompliant       3h42m
      ztp-group.group-du-sno-validator-du-policy               inform               NonCompliant       3h42m
      ztp-install.example1-common-config-policy-pjz9s          enforce              Compliant          167m
      ztp-install.example1-common-subscriptions-policy-zzd9k   enforce              NonCompliant       164m
      ztp-site.example1-config-policy                          inform               NonCompliant       3h42m
      ztp-site.example1-perf-policy                            inform               NonCompliant       3h42m

      Copy to Clipboard Toggle word wrap

   2. To check policy status from the RHACM web console, perform the following actions:

      1. Click Governance Find policies.
      2. Click on a cluster policy to check it’s status.

Procedure

1. To display detailed information about the policies, run the following command:

   $ oc describe -n openshift-gitops application policies

   Copy to Clipboard Toggle word wrap

2. Check for Status: Conditions: to show the error logs. For example, setting an invalid sourceFile→fileName: generates the error shown below:

   Status:
     Conditions:
       Last Transition Time:  2021-11-26T17:21:39Z
       Message:               rpc error: code = Unknown desc = `kustomize build /tmp/https___git.com/ran-sites/policies/ --enable-alpha-plugins` failed exit status 1: 2021/11/26 17:21:40 Error could not find test.yaml under source-crs/: no such file or directory Error: failure in plugin configured via /tmp/kust-plugin-config-52463179; exit status 1: exit status 1
       Type:  ComparisonError

   Copy to Clipboard Toggle word wrap

3. Check for Status: Sync:. If there are log errors at Status: Conditions:, the Status: Sync: shows Unknown or Error:

   Status:
     Sync:
       Compared To:
         Destination:
           Namespace:  policies-sub
           Server:     https://kubernetes.default.svc
         Source:
           Path:             policies
           Repo URL:         https://git.com/ran-sites/policies/.git
           Target Revision:  master
       Status:               Error

   Copy to Clipboard Toggle word wrap

4. When Red Hat Advanced Cluster Management (RHACM) recognizes that policies apply to a ManagedCluster object, the policy CR objects are applied to the cluster namespace. Check to see if the policies were copied to the cluster namespace:

   $ oc get policy -n $CLUSTER

   Copy to Clipboard Toggle word wrap

   Example output:

   NAME                                         REMEDIATION ACTION   COMPLIANCE STATE   AGE
   ztp-common.common-config-policy              inform               Compliant          13d
   ztp-common.common-subscriptions-policy       inform               Compliant          13d
   ztp-group.group-du-sno-config-policy         inform               Compliant          13d
   Ztp-group.group-du-sno-validator-du-policy   inform               Compliant          13d
   ztp-site.example-sno-config-policy           inform               Compliant          13d

   Copy to Clipboard Toggle word wrap

   RHACM copies all applicable policies into the cluster namespace. The copied policy names have the format: <policyGenTemplate.Namespace>.<policyGenTemplate.Name>-<policyName>.

5. Check the placement rule for any policies not copied to the cluster namespace. The matchSelector in the PlacementRule for those policies should match labels on the ManagedCluster object:

   $ oc get placementrule -n $NS

   Copy to Clipboard Toggle word wrap

6. Note the PlacementRule name appropriate for the missing policy, common, group, or site, using the following command:

   $ oc get placementrule -n $NS <placementRuleName> -o yaml

   Copy to Clipboard Toggle word wrap
   • The status-decisions should include your cluster name.
   • The key-value pair of the matchSelector in the spec must match the labels on your managed cluster.

7. Check the labels on the ManagedCluster object using the following command:

   $ oc get ManagedCluster $CLUSTER -o jsonpath='{.metadata.labels}' | jq

   Copy to Clipboard Toggle word wrap

8. Check to see which policies are compliant using the following command:

   $ oc get policy -n $CLUSTER

   Copy to Clipboard Toggle word wrap

   If the Namespace, OperatorGroup, and Subscription policies are compliant but the Operator configuration policies are not, it is likely that the Operators did not install on the managed cluster. This causes the Operator configuration policies to fail to apply because the CRD is not yet applied to the spoke.

Procedure

1. A ClusterGroupUpgrade CR is generated in the namespace ztp-install by the Topology Aware Lifecycle Manager after the managed cluster becomes Ready:

   $ export CLUSTER=<clusterName>

   Copy to Clipboard Toggle word wrap

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER

   Copy to Clipboard Toggle word wrap

2. If there are unexpected issues and the policies fail to become complaint within the configured timeout (the default is 4 hours), the status of the ClusterGroupUpgrade CR shows UpgradeTimedOut:

   $ oc get clustergroupupgrades -n ztp-install $CLUSTER -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

   Copy to Clipboard Toggle word wrap

3. A ClusterGroupUpgrade CR in the UpgradeTimedOut state automatically restarts its policy reconciliation every hour. If you have changed your policies, you can start a retry immediately by deleting the existing ClusterGroupUpgrade CR. This triggers the automatic creation of a new ClusterGroupUpgrade CR that begins reconciling the policies immediately:

   $ oc delete clustergroupupgrades -n ztp-install $CLUSTER

   Copy to Clipboard Toggle word wrap

Procedure

1. Remove the content that you no longer need from the affected CRs. In this example, the disableDrain: false line was removed from the SriovOperatorConfig CR.

   Example CR

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovOperatorConfig
   metadata:
     name: default
     namespace: openshift-sriov-network-operator
   spec:
     configDaemonNodeSelector:
       "node-role.kubernetes.io/$mcp": ""
     disableDrain: true
     enableInjector: true
     enableOperatorWebhook: true

   Copy to Clipboard Toggle word wrap

2. Change the complianceType of the affected policies to mustonlyhave in the group-du-sno-ranGen.yaml file.

   Example YAML

   # ...
   - fileName: SriovOperatorConfig.yaml
     policyName: "config-policy"
     complianceType: mustonlyhave
   # ...

   Copy to Clipboard Toggle word wrap

3. Create a ClusterGroupUpdates CR and specify the clusters that must receive the CR changes::

   Example ClusterGroupUpdates CR

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-remove
     namespace: default
   spec:
     managedPolicies:
       - ztp-group.group-du-sno-config-policy
     enable: false
     clusters:
     - spoke1
     - spoke2
     remediationStrategy:
       maxConcurrency: 2
       timeout: 240
     batchTimeoutAction:

   Copy to Clipboard Toggle word wrap

4. Create the ClusterGroupUpgrade CR by running the following command:

   $ oc create -f cgu-remove.yaml

   Copy to Clipboard Toggle word wrap

5. When you are ready to apply the changes, for example, during an appropriate maintenance window, change the value of the spec.enable field to true by running the following command:

   $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-remove \
   --patch '{"spec":{"enable":true}}' --type=merge

   Copy to Clipboard Toggle word wrap

Verification

1. Check the status of the policies by running the following command:

   $ oc get <kind> <changed_cr_name>

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE   NAME                                                   REMEDIATION ACTION   COMPLIANCE STATE   AGE
   default     cgu-ztp-group.group-du-sno-config-policy               enforce                                 17m
   default     ztp-group.group-du-sno-config-policy                   inform               NonCompliant       15h

   Copy to Clipboard Toggle word wrap

   When the COMPLIANCE STATE of the policy is Compliant, it means that the CR is updated and the unwanted content is removed.

2. Check that the policies are removed from the targeted clusters by running the following command on the managed clusters:

   $ oc get <kind> <changed_cr_name>

   Copy to Clipboard Toggle word wrap

   If there are no results, the CR is removed from the managed cluster.