Chapter 9. Advanced managed cluster configuration with ClusterInstance resources

Procedure

1. Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

2. In your /clusterinstance directory, create a subdirectory with your extra manifests. The following example illustrates a sample folder structure:

   clusterinstance/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   │   ├── 01-example-machine-config.yaml
   │   ├── enable-crun-master.yaml
   │   └── enable-crun-worker.yaml
   └── kustomization.yaml

3. Create or update the kustomization.yaml file to use configMapGenerator to package your extra manifests into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: extra-manifests-cm
       namespace: site1-sno-du
       files:
         - extra-manifest/01-example-machine-config.yaml
         - extra-manifest/enable-crun-master.yaml
         - extra-manifest/enable-crun-worker.yaml
   generatorOptions:
     disableNameSuffixHash: true

   • The configMapGenerator.namespace value must match the ClusterInstance namespace.
   • Setting generatorOptions.disableNameSuffixHash to true disables the hash suffix so the ConfigMap name is predictable.

4. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: extra-manifests-cm
     # ...

   • The extraManifestsRefs field references the ConfigMap containing the extra manifests.

5. Commit the ClusterInstance CR, extra manifest files, and kustomization.yaml to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the CRs contained in the referenced ConfigMap resources as extra manifests.

   Note

   You can reference multiple ConfigMap resources in extraManifestsRefs to organize your manifests logically. For example, you might have separate ConfigMap resources for crun configuration, custom MachineConfig CRs, and other Day 0 configurations.

Procedure

1. In your ClusterInstance CR, set the machine network MTU on the network interface for all nodes.

   The following example configures a VLAN interface with MTU 9100:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     nodes:
       - hostName: "node1.example.com"
         nodeNetwork:
           interfaces:
             - name: "bond0.120"
               macAddress: "00:00:00:00:00:00"
           config:
             interfaces:
               - name: bond0.120
                 type: vlan
                 state: up
                 mtu: 9100
                 ipv4:
                   enabled: true
                   dhcp: false
                   address:
                     - ip: "192.168.120.15"
                       prefix-length: 25
                 vlan:
                   base-iface: bond0
                   id: 120
     # ...

2. Create a Network CR manifest file named set-cluster-mtu.yaml that sets the cluster network MTU:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         mtu: 8900

   where:

   mtu

   Specifies the cluster network MTU value. This value must be at least 100 bytes less than the machine network MTU. In this example, the value is 200 bytes less than the machine network MTU of 9100 to allow headroom for IPsec and other future requirements.

3. In your clusterinstance directory, place the manifest file in an extra manifests subdirectory:

   clusterinstance/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   │   └── set-cluster-mtu.yaml
   └── kustomization.yaml

4. Create or update the kustomization.yaml file to package the extra manifest into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: cluster-mtu-extra-manifests
       namespace: site1-sno-du
       files:
         - extra-manifest/set-cluster-mtu.yaml
   generatorOptions:
     disableNameSuffixHash: true

5. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: cluster-mtu-extra-manifests
     # ...

6. Commit the ClusterInstance CR, the set-cluster-mtu.yaml manifest, and the kustomization.yaml to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the Network CR as an extra manifest, and the CNO uses the specified MTU value instead of auto-calculating it.

Procedure

1. Update the ClusterInstance CR to add the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation to the BareMetalHost resource for the node, and push the changes to the Git repository:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         extraAnnotations:
           BareMetalHost:
             bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
   # ...

2. Verify that the BareMetalHost object is annotated by running the following command:

   $ oc get bmh -n <cluster_namespace> <bmh_name> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

   Example output

   true

3. Delete the BareMetalHost CR by configuring the pruneManifests field in the ClusterInstance CR to remove the target BareMetalHost resource:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         pruneManifests:
           - apiVersion: metal3.io/v1alpha1
             kind: BareMetalHost
   # ...

4. Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verification

1. Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

   $ oc get bmh -n <cluster_namespace>

   $ oc get agent -n <cluster_namespace>

2. Verify that the node record has been deleted from the spoke cluster by running the following command:

   $ oc get nodes

   Note

   If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete.

3. After the BareMetalHost object is successfully deleted, remove the worker node definition from the spec.nodes section in the ClusterInstance CR and push the changes to the Git repository.