Chapter 5. Managing the hosting cluster for virtualized control planes

Procedure

1. Verify that all target cluster control plane VMs are healthy before starting the update by running the following command:

   $ oc get vm -A -l <vm_selector_label>=<value>

   Replace <vm_selector_label>=<value> with the label you configured in your KubeVirt Redfish vm_selector, for example redfish-enabled=true.

   All control plane VMs must show a Running status.

2. Verify etcd health on each target cluster by running the following command:

   $ oc --kubeconfig=<target_cluster_kubeconfig> get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{end}'

   • target_cluster_kubeconfig is the path to the target cluster’s kubeconfig file.

3. Start the hosting cluster update by following the standard OpenShift Container Platform update process.

   The Machine Config Operator drains and reboots each node in sequence. The PodDisruptionBudget CR prevents more than one control plane VM for each target cluster from being evicted at a time.

4. Monitor the update progress by running the following commands:

   $ oc get clusterversion
   $ oc get nodes

5. After the hosting cluster update completes, update OpenShift Virtualization if a new version is available.

6. Verify that all control plane VMs are running after the update by running the following command:

   $ oc get vm -A -l <vm_selector_label>=<value>

   Replace <vm_selector_label>=<value> with the label you configured in your KubeVirt Redfish vm_selector, for example redfish-enabled=true.

   All control plane VMs must show a Running status.

Verification

1. Verify etcd health on each target cluster after the update by running the following command:

   $ oc --kubeconfig=<target_cluster_kubeconfig> get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{end}'

   • target_cluster_kubeconfig is the path to the target cluster’s kubeconfig file.

2. Verify that the target cluster API is responsive by running the following command:

   $ oc --kubeconfig=<target_cluster_kubeconfig> get nodes

   • target_cluster_kubeconfig is the path to the target cluster’s kubeconfig file.

Procedure

1. Identify which control plane VMs are running on the node you plan to maintain by running the following command:

   $ oc get pods --field-selector spec.nodeName=<node_name> -n <vm_namespace> -l vcp-cluster=<cluster_name>

   • <node_name> is the name of the node you plan to maintain.
   • <vm_namespace> is the namespace of the control plane VMs.

   • <cluster_name> is the name of the target cluster.

     Note

     Use oc get pods with the spec.nodeName field selector to filter by node. The oc get vmi command does not support the status.nodeName field selector.

2. Verify that all target clusters affected by this node have healthy etcd quorum before proceeding by running the following command:

   $ oc --kubeconfig=<target_cluster_kubeconfig> get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{end}'

   • target_cluster_kubeconfig is the path to the target cluster’s kubeconfig file.

3. Cordon the node to prevent new workloads from being scheduled by running the following command:

   $ oc adm cordon <node_name>

   • <node_name> is the name of the node you plan to maintain.

4. Drain the node to evict all pods, including control plane VM pods, by running the following command:

   $ oc adm drain <node_name> --ignore-daemonsets --delete-emptydir-data

   • <node_name> is the name of the node you plan to maintain.

     The PodDisruptionBudget ensures that only one control plane VM per target cluster is evicted. The drain operation blocks until the PDB allows eviction.

5. Verify that the remaining control plane VMs maintain etcd quorum on each affected target cluster by running the following command:

   $ oc --kubeconfig=<target_cluster_kubeconfig> get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{end}'

   • target_cluster_kubeconfig is the path to the target cluster’s kubeconfig file.
6. Perform the required maintenance on the physical node.

7. Uncordon the node to allow workloads to be scheduled again by running the following command:

   $ oc adm uncordon <node_name>

   • <node_name> is the name of the node you plan to maintain.

8. Verify that the control plane VM restarts on the node and rejoins the target cluster by running the following command:

   $ oc get vmi -A -l <vm_selector_label>=<value>

   Replace <vm_selector_label>=<value> with the label you configured in your KubeVirt Redfish vm_selector, for example redfish-enabled=true.

   All control plane VMs must show a Running phase.

   Note

   Use oc get vmi (VirtualMachineInstance) instead of oc get vm (VirtualMachine) to verify the actual runtime state. The VirtualMachine resource reflects the desired state, while the VirtualMachineInstance resource reflects whether the VM is currently running.