Chapter 9. Advanced managed cluster configuration with ClusterInstance resources

Procedure

1. Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

2. In your /clusterinstance directory, create a subdirectory with your extra manifests. The following example illustrates a sample folder structure:

   clusterinstance/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   │   ├── 01-example-machine-config.yaml
   │   ├── enable-crun-master.yaml
   │   └── enable-crun-worker.yaml
   └── kustomization.yaml

3. Create or update the kustomization.yaml file to use configMapGenerator to package your extra manifests into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: extra-manifests-cm
       namespace: site1-sno-du
       files:
         - extra-manifest/01-example-machine-config.yaml
         - extra-manifest/enable-crun-master.yaml
         - extra-manifest/enable-crun-worker.yaml
   generatorOptions:
     disableNameSuffixHash: true

   • The configMapGenerator.namespace value must match the ClusterInstance namespace.
   • Setting generatorOptions.disableNameSuffixHash to true disables the hash suffix so the ConfigMap name is predictable.

4. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: extra-manifests-cm
     # ...

   • The extraManifestsRefs field references the ConfigMap containing the extra manifests.

5. Commit the ClusterInstance CR, extra manifest files, and kustomization.yaml to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the CRs contained in the referenced ConfigMap resources as extra manifests.

   Note

   You can reference multiple ConfigMap resources in extraManifestsRefs to organize your manifests logically. For example, you might have separate ConfigMap resources for crun configuration, custom MachineConfig CRs, and other Day 0 configurations.

Procedure

1. In your /clusterinstance directory, create an extra-manifest subdirectory and save the MachineConfig object to a file in this directory, for example, extra-manifest/10-br-ex-master.yaml.

2. Create or update the kustomization.yaml file to use configMapGenerator to package the MachineConfig object into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: br-ex-extra-manifest
       namespace: site1-sno-du
       files:
         - extra-manifest/10-br-ex-master.yaml
   generatorOptions:
     disableNameSuffixHash: true

   • The configMapGenerator.namespace value must match the ClusterInstance namespace.
   • Setting generatorOptions.disableNameSuffixHash to true disables the hash suffix so the ConfigMap name is predictable.

3. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: br-ex-extra-manifest
     # ...

4. Commit the ClusterInstance CR, MachineConfig object, and kustomization.yaml file to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the MachineConfig object as an extra manifest, and the customized br-ex bridge configuration is applied to each node without requiring a reboot.

Procedure

1. In your ClusterInstance CR, set the machine network MTU on the network interface for all nodes.

   The following example configures a VLAN interface with MTU 9100:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     nodes:
       - hostName: "node1.example.com"
         nodeNetwork:
           interfaces:
             - name: "bond0.120"
               macAddress: "00:00:00:00:00:00"
           config:
             interfaces:
               - name: bond0.120
                 type: vlan
                 state: up
                 mtu: 9100
                 ipv4:
                   enabled: true
                   dhcp: false
                   address:
                     - ip: "192.168.120.15"
                       prefix-length: 25
                 vlan:
                   base-iface: bond0
                   id: 120
     # ...

2. Create a Network CR manifest file named set-cluster-mtu.yaml that sets the cluster network MTU:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         mtu: 8900

   where:

   mtu

   Specifies the cluster network MTU value. This value must be at least 100 bytes less than the machine network MTU. In this example, the value is 200 bytes less than the machine network MTU of 9100 to allow headroom for IPsec and other future requirements.

3. In your clusterinstance directory, place the manifest file in an extra manifests subdirectory:

   clusterinstance/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   │   └── set-cluster-mtu.yaml
   └── kustomization.yaml

4. Create or update the kustomization.yaml file to package the extra manifest into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: cluster-mtu-extra-manifests
       namespace: site1-sno-du
       files:
         - extra-manifest/set-cluster-mtu.yaml
   generatorOptions:
     disableNameSuffixHash: true

5. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: cluster-mtu-extra-manifests
     # ...

6. Commit the ClusterInstance CR, the set-cluster-mtu.yaml manifest, and the kustomization.yaml to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the Network CR as an extra manifest, and the CNO uses the specified MTU value instead of auto-calculating it.

Procedure

1. Update the ClusterInstance CR to add the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation to the BareMetalHost resource for the node, and push the changes to the Git repository:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         extraAnnotations:
           BareMetalHost:
             bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
   # ...

2. Verify that the BareMetalHost object is annotated by running the following command:

   $ oc get bmh -n <cluster_namespace> <bmh_name> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

   Example output

   true

3. Delete the BareMetalHost CR by configuring the pruneManifests field in the ClusterInstance CR to remove the target BareMetalHost resource:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "worker-node2.example.com"
         role: "worker"
         pruneManifests:
           - apiVersion: metal3.io/v1alpha1
             kind: BareMetalHost
   # ...

4. Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verification

1. Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

   $ oc get bmh -n <cluster_namespace>

   $ oc get agent -n <cluster_namespace>

2. Verify that the node record has been deleted from the spoke cluster by running the following command:

   $ oc get nodes

   Note

   If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete.

3. After the BareMetalHost object is successfully deleted, remove the worker node definition from the spec.nodes section in the ClusterInstance CR and push the changes to the Git repository.

Procedure

1. Identify the failed control plane node on the managed cluster by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> get nodes

   where:

   <managed_cluster_kubeconfig>

   Specifies the kubeconfig file for the managed cluster.

   Example output

   NAME                           STATUS     ROLES                  AGE   VERSION
   control-plane-0.example.com    Ready      control-plane,master   10d   v1.33.0
   control-plane-1.example.com    Ready      control-plane,master   10d   v1.33.0
   control-plane-2.example.com    NotReady   control-plane,master   10d   v1.33.0
   compute-1.example.com          Ready      worker                 10d   v1.33.0

   The failed node reports NotReady or is missing from the list. Note the hostName value that matches that node in the ClusterInstance CR.

2. In the ClusterInstance CR, add the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete annotation to the failed node and apply the updated CR to the hub cluster:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "control-plane-2.example.com"
         role: "master"
         extraAnnotations:
           BareMetalHost:
             bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
   # ...

3. Verify that the BareMetalHost resource on the hub cluster has the annotation by running the following command:

   $ oc --kubeconfig <hub_cluster_kubeconfig> get bmh -n <cluster_namespace> <bmh_name> \
     -o jsonpath='{.metadata.annotations.bmac\.agent-install\.openshift.io/remove-agent-and-node-on-delete}{"\n"}'

   Example output

   true

   where:

   <cluster_namespace>

   Specifies the namespace of the ClusterInstance CR.

   <bmh_name>

   Specifies the name of the BareMetalHost resource, which matches the node hostName value.

4. Configure the pruneManifests field for the failed node so that the BareMetalHost resource is deleted. Keep the deletion annotation and apply the updated CR to the hub cluster:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "example-cluster"
     namespace: "example-cluster"
   spec:
     # ...
     nodes:
       - hostName: "control-plane-2.example.com"
         role: "master"
         extraAnnotations:
           BareMetalHost:
             bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true"
         pruneManifests:
           - apiVersion: metal3.io/v1alpha1
             kind: BareMetalHost
   # ...

   The status of the BareMetalHost resource changes to deprovisioning. If the BMC is unreachable, deprovisioning can take significantly longer than a graceful removal. Wait until the BareMetalHost resource is fully deleted.

5. Verify that the BareMetalHost resource is deleted from the hub cluster by running the following command:

   $ oc --kubeconfig <hub_cluster_kubeconfig> get bmh -n <cluster_namespace>

   where:

   <hub_cluster_kubeconfig>

   Specifies the kubeconfig file for the hub cluster.

   <cluster_namespace>

   Specifies the namespace of the ClusterInstance CR.

6. Verify that the Agent resource for the failed node is deleted from the hub cluster by running the following command:

   $ oc --kubeconfig <hub_cluster_kubeconfig> get agent -n <cluster_namespace> \
     -o jsonpath='{range .items[*]}{.spec.hostname}{"\n"}{end}' | grep <failed_hostname>

   where:

   <failed_hostname>

   Specifies the hostName value of the failed node, for example control-plane-2.example.com.

   If the command returns no output, the Agent resource for the failed node has been deleted. If other Agent resources remain in the namespace for other nodes, that is expected.

   Important

   Wait for the Agent resource for the failed node to be deleted before adding the replacement node.

7. Verify that the failed node is no longer listed on the managed cluster by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> get nodes

8. Configure the replacement node in spec.nodes. Remove the pruneManifests field and the deletion annotation. Then complete one of the following actions:

   • To reuse the same hardware, keep the existing node entry and the same hostName value. Leave the BMC address, boot MAC address, and node network configuration unchanged.

   • To install replacement hardware, update at least the BMC address, BMC secret name, boot MAC address, and node network configuration. If the replacement uses a different hostName value, delete the failed node entry and add a new entry.

     The following example reuses the control-plane-2.example.com hostname on replacement hardware:

     apiVersion: siteconfig.open-cluster-management.io/v1alpha1
     kind: ClusterInstance
     metadata:
       name: "example-cluster"
       namespace: "example-cluster"
     spec:
       # ...
       nodes:
         - hostName: "control-plane-2.example.com"
           role: "master"
           bmcAddress: "redfish-virtualmedia+https://192.0.2.22/redfish/v1/Systems/System.Embedded.1"
           bmcCredentialsName:
             name: "control-plane-2-bmc-secret"
           bootMACAddress: "AA:BB:CC:DD:EE:22"
           bootMode: "UEFI"
           nodeNetwork:
             interfaces:
               - name: eno1
                 macAddress: "AA:BB:CC:DD:EE:22"
             config:
               interfaces:
                 - name: eno1
                   type: ethernet
                   state: up
                   macAddress: "AA:BB:CC:DD:EE:22"
                   ipv4:
                     enabled: true
                     address:
                       - ip: 192.0.2.12
                         prefix-length: 24
                   ipv6:
                     enabled: false
     # ...

9. Create the BMC secret for the replacement node in the same namespace as the ClusterInstance CR. The BareMetalHost finalizer deletes the referenced bmcSecret when the old BareMetalHost resource is removed, so this step is required for both same-hardware and new-hardware replacements.

   apiVersion: v1
   kind: Secret
   metadata:
     name: "control-plane-2-bmc-secret"
     namespace: example-cluster
   stringData:
     username: "username"
     password: "password"
   type: Opaque

10. Apply the updated ClusterInstance CR and any new secret to the hub cluster. The SiteConfig Operator creates a new BareMetalHost resource. The Assisted Installer provisions the host and joins it to the cluster.

11. Monitor provisioning on the hub cluster by running the following command:

    $ oc --kubeconfig <hub_cluster_kubeconfig> get bmh -n <cluster_namespace>

    where:

    <hub_cluster_kubeconfig>

    Specifies the kubeconfig file for the hub cluster.

    <cluster_namespace>

    Specifies the namespace of the ClusterInstance CR.

    Example output

    NAME                           STATE          CONSUMER   ONLINE   ERROR   AGE
    control-plane-0.example.com    provisioned               true             10d
    control-plane-1.example.com    provisioned               true             10d
    control-plane-2.example.com    provisioning              true             5m

Verification

1. Verify that the replacement node is Ready on the managed cluster by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> get nodes -l node-role.kubernetes.io/control-plane

   where:

   <managed_cluster_kubeconfig>

   Specifies the kubeconfig file for the managed cluster.

2. Identify an etcd pod on a control plane node by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> get pods -n openshift-etcd

3. Verify that etcd has three healthy members by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> rsh -n openshift-etcd <etcd_pod> \
     etcdctl member list -w table

   where:

   <etcd_pod>

   Specifies the name of an etcd pod on a control plane node.

4. Verify that every etcd endpoint is healthy by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> rsh -n openshift-etcd <etcd_pod> \
     etcdctl endpoint health --cluster

5. Verify that the etcd and kube-apiserver Operators have finished reconciling by running the following command:

   $ oc --kubeconfig <managed_cluster_kubeconfig> get clusteroperators etcd kube-apiserver

   Wait until both Operators report True in the AVAILABLE column and False in the PROGRESSING column.