Chapter 8. Advanced managed cluster configuration with SiteConfig resources

Procedure

1. Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

2. In your custom /siteconfig directory, create a subdirectory /custom-manifest for your extra manifests. The following example illustrates a sample /siteconfig with /custom-manifest folder:

   siteconfig
   ├── site1-sno-du.yaml
   ├── site2-standard-du.yaml
   ├── extra-manifest/
   └── custom-manifest
       └── 01-example-machine-config.yaml

   Copy to Clipboard Toggle word wrap
   Note

   The subdirectory names /custom-manifest and /extra-manifest used throughout are example names only. There is no requirement to use these names and no restriction on how you name these subdirectories. In this example /extra-manifest refers to the Git subdirectory that stores the contents of /extra-manifest from the ztp-site-generate container.

3. Add your custom extra manifest CRs to the siteconfig/custom-manifest directory.

4. In your SiteConfig CR, enter the directory name in the extraManifests.searchPaths field, for example:

   clusters:
   - clusterName: "example-sno"
     networkType: "OVNKubernetes"
     extraManifests:
       searchPaths:
         - extra-manifest/ 

   1

   
         - custom-manifest/ 

   2

   Copy to Clipboard Toggle word wrap

   1

   Folder for manifests copied from the ztp-site-generate container.

   2

   Folder for custom manifests.

5. Save the SiteConfig, /extra-manifest, and /custom-manifest CRs, and push them to the site configuration repo.

Procedure

1. To prevent the GitOps ZTP pipeline from applying the 03-sctp-machine-config-worker.yaml CR file, apply the following YAML in the SiteConfig CR:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     baseDomain: "example.com"
     pullSecretRef:
       name: "assisted-deployment-pull-secret"
     clusterImageSetNameRef: "openshift-4.17"
     sshPublicKey: "<ssh_public_key>"
     clusters:
   - clusterName: "site1-sno-du"
     extraManifests:
       filter:
         exclude:
           - 03-sctp-machine-config-worker.yaml

   Copy to Clipboard Toggle word wrap

   The GitOps ZTP pipeline skips the 03-sctp-machine-config-worker.yaml CR during installation. All other CRs in /source-crs/extra-manifest are applied.

2. Save the SiteConfig CR and push the changes to the site configuration repository.

   The GitOps ZTP pipeline monitors and adjusts what CRs it applies based on the SiteConfig filter instructions.

3. Optional: To prevent the GitOps ZTP pipeline from applying all the /source-crs/extra-manifest CRs during cluster installation, apply the following YAML in the SiteConfig CR:

   - clusterName: "site1-sno-du"
     extraManifests:
       filter:
         inclusionDefault: exclude

   Copy to Clipboard Toggle word wrap

4. Optional: To exclude all the /source-crs/extra-manifest RAN CRs and instead include a custom CR file during installation, edit the custom SiteConfig CR to set the custom manifests folder and the include file, for example:

   clusters:
   - clusterName: "site1-sno-du"
     extraManifestPath: "<custom_manifest_folder>" 

   1

   
     extraManifests:
       filter:
         inclusionDefault: exclude  

   2

   
         include:
           - custom-sctp-machine-config-worker.yaml

   Copy to Clipboard Toggle word wrap

   1

   Replace <custom_manifest_folder> with the name of the folder that contains the custom installation CRs, for example, user-custom-manifest/.

   2

   Set inclusionDefault to exclude to prevent the GitOps ZTP pipeline from applying the files in /source-crs/extra-manifest during installation.

   The following example illustrates the custom folder structure:

   siteconfig
     ├── site1-sno-du.yaml
     └── user-custom-manifest
           └── custom-sctp-machine-config-worker.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Update the SiteConfig CR to include the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation and push the changes to the Git repository:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "cnfdf20"
     namespace: "cnfdf20"
   spec:
     clusters:
       nodes:
       - hostname: node6
         role: "worker"
         crAnnotations:
           add:
             BareMetalHost:
               bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: true
   # ...

   Copy to Clipboard Toggle word wrap

2. Verify that the BareMetalHost object is annotated by running the following command:

   oc get bmh -n <managed-cluster-namespace> <bmh-object> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

   Copy to Clipboard Toggle word wrap

   Example output

   true

   Copy to Clipboard Toggle word wrap

3. Suppress the generation of the BareMetalHost CR by updating the SiteConfig CR to include the crSuppression.BareMetalHost annotation:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "cnfdf20"
     namespace: "cnfdf20"
   spec:
     clusters:
     - nodes:
       - hostName: node6
         role: "worker"
         crSuppression:
         - BareMetalHost
   # ...

   Copy to Clipboard Toggle word wrap
4. Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verification

1. Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

   $ oc get bmh -n <cluster-ns>

   Copy to Clipboard Toggle word wrap

   $ oc get agent -n <cluster-ns>

   Copy to Clipboard Toggle word wrap

2. Verify that the node record has been deleted from the spoke cluster by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap
   Note

   If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete.