Chapter 9. Advanced managed cluster configuration with ClusterInstance resources

Procedure

1. Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

2. In your /clusterinstance directory, create a subdirectory with your extra manifests. The following example illustrates a sample folder structure:

   clusterinstance/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   │   ├── 01-example-machine-config.yaml
   │   ├── enable-crun-master.yaml
   │   └── enable-crun-worker.yaml
   └── kustomization.yaml

   Copy to Clipboard Toggle word wrap

3. Create or update the kustomization.yaml file to use configMapGenerator to package your extra manifests into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: extra-manifests-cm
       namespace: site1-sno-du 

   1

   
       files:
         - extra-manifest/01-example-machine-config.yaml
         - extra-manifest/enable-crun-master.yaml
         - extra-manifest/enable-crun-worker.yaml
   generatorOptions:
     disableNameSuffixHash: true 

   2

   Copy to Clipboard Toggle word wrap

   1

   The namespace must match the ClusterInstance namespace.

   2

   Disables the hash suffix so the ConfigMap name is predictable.

4. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: extra-manifests-cm 

   1

   
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Reference to the ConfigMap containing the extra manifests.

5. Commit the ClusterInstance CR, extra manifest files, and kustomization.yaml to your Git repository and push the changes.

Procedure

1. Update the ClusterInstance CR to add the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation to the BareMetalHost resource for the node, and push the changes to the Git repository:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         extraAnnotations:
           BareMetalHost:
             bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
   # ...

   Copy to Clipboard Toggle word wrap

2. Verify that the BareMetalHost object is annotated by running the following command:

   $ oc get bmh -n <cluster_namespace> <bmh_name> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

   Copy to Clipboard Toggle word wrap

   Example output

   true

   Copy to Clipboard Toggle word wrap

3. Delete the BareMetalHost CR by configuring the pruneManifests field in the ClusterInstance CR to remove the target BareMetalHost resource:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         pruneManifests:
           - apiVersion: metal3.io/v1alpha1
             kind: BareMetalHost
   # ...

   Copy to Clipboard Toggle word wrap
4. Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verification

1. Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

   $ oc get bmh -n <cluster_namespace>

   Copy to Clipboard Toggle word wrap

   $ oc get agent -n <cluster_namespace>

   Copy to Clipboard Toggle word wrap

2. Verify that the node record has been deleted from the spoke cluster by running the following command:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap
   Note

   If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete.

3. After the BareMetalHost object is successfully deleted, remove the worker node definition from the spec.nodes section in the ClusterInstance CR and push the changes to the Git repository.