Chapter 1. Virtualized control planes
A virtualized control plane deployment is an OpenShift Container Platform cluster that runs its control plane nodes as virtual machines (VMs) on a hosting cluster with OpenShift Virtualization.
1.1. How virtualized control planes work
A virtualized control plane deployment involves two distinct clusters:
- Hosting cluster
- An existing OpenShift Container Platform cluster running OpenShift Virtualization that hosts the control plane VMs. A single hosting cluster can support multiple target clusters by running each cluster’s control plane VMs in separate namespaces.
- Target cluster
- The OpenShift Container Platform cluster whose control plane nodes run as VMs on the hosting cluster. The target cluster’s worker nodes run on separate infrastructure, either physical servers or VMs on different hosts.
KubeVirt Redfish runs on the hosting cluster and exposes the control plane VMs through standard Redfish API endpoints. This enables standard installation workflows such as Agent-based Installer, installer-provisioned infrastructure, or GitOps Zero Touch Provisioning (ZTP) to treat the VMs the same as physical servers with baseboard management controllers (BMCs).
Virtualized control planes differ from Hosted control planes. With virtualized control planes, the control plane runs as VMs with hypervisor-level isolation. With Hosted control planes, the control plane runs as pods with container-level isolation.
1.2. When to use a virtualized control plane
You can use virtualized control planes to address infrastructure challenges where physical hardware constraints, compliance requirements, or provisioning timelines limit how you deploy and manage clusters.
- Regulatory isolation
- You must meet compliance mandates that require hypervisor-level isolation for control plane components. A virtualized control plane runs each control plane node as a separate VM, providing stronger isolation boundaries than container-based alternatives.
- Infrastructure consolidation
- You want to reduce hardware costs by hosting multiple cluster control planes on shared infrastructure. A single hosting cluster can run control plane VMs for multiple target clusters in separate namespaces, reducing the number of physical servers you need to maintain.
- Faster cluster provisioning
- You need to deploy new clusters faster than physical bare-metal provisioning allows. Provisioning control plane VMs on an existing hosting cluster reduces the lead times associated with physical hardware provisioning.
You can deploy virtualized control planes by using standard installation workflows such as Agent-based Installer, installer-provisioned infrastructure, or GitOps Zero Touch Provisioning (ZTP). These workflows treat the VMs the same as physical servers with baseboard management controllers (BMCs), so your existing deployment processes work without modification.
1.3. Virtualized control plane architecture
A single hosting cluster runs control plane VMs for multiple target clusters across its nodes.
The following diagram shows a multitenant virtualized control plane deployment.
The hosting cluster runs each control plane VM as a VirtualMachine CR managed by OpenShift Virtualization. KubeVirt Redfish exposes these VMs through standard Redfish API endpoints, which the installation workflow uses to manage VM power state and boot order.
Each target cluster’s control plane VMs are distributed across hosting cluster nodes by using anti-affinity rules, so that a single node failure affects only one VM for each target cluster. The VMs run in a dedicated namespace on the hosting cluster, providing resource and access isolation between target clusters.
Each target cluster has its own machine network that provides L2 connectivity between its control plane VMs. The target cluster’s worker nodes run on a separate L2 network segment on bare-metal infrastructure, connected to the control plane through routed L3 connectivity. The apiVIPs must reside on the same L2 segment as the control plane VMs, and the ingressVIPs must reside on the same L2 segment as the worker nodes.
Network address translation (NAT) is not supported between the control plane and worker networks.
The following diagram shows how network traffic flows inside a single hosting cluster node. Control plane VM traffic and hosting cluster pod traffic use separate network paths.
Control plane VMs connect to the cluster machine network through a Multus NetworkAttachmentDefinition resource and a secondary bridge interface. Hosting cluster pods use the default OVN-Kubernetes network on a separate bridge and NIC.
1.4. Virtualized control plane deployment workflow
A virtualized control plane deployment involves installing KubeVirt Redfish on the hosting cluster, configuring it to expose VMs, and running a supported installation method.
Virtualized control planes require an OpenShift Container Platform cluster with OpenShift Virtualization installed and operational, which operates as the hosting cluster.
The following high-level steps outline the deployment process:
Install and configure KubeVirt Redfish on the hosting cluster.
This includes defining which VMs to expose through the Redfish API, configuring authentication credentials, and creating a
RouteCR to expose the endpoint externally.Create the control plane VMs on the hosting cluster.
Configure the VMs with appropriate resources and network settings, and ensure they remain powered off until the installation begins.
Configure high availability for the control plane VMs.
Set anti-affinity rules to distribute VMs across different physical nodes, create a
PodDisruptionBudgetto prevent concurrent eviction, and set the VM eviction strategy toNone.Configure your installation method to use KubeVirt Redfish.
In your configuration files, specify BMC addresses using the KubeVirt Redfish route URL for the virtualized control plane nodes, for example:
redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.<vm_name>.Run the installation.
The VMs boot from the installation media and communicate with each other to form the cluster. Depending on the installation method, this process is either fully automated or requires manual intervention to boot each node.
After installation completes, a new OpenShift Container Platform cluster is deployed with its control plane running on VMs hosted by the original OpenShift Virtualization cluster.
1.5. High availability for virtualized control planes
A virtualized control plane relies on etcd quorum to remain operational. With three control plane VMs, the cluster tolerates the loss of one VM but becomes unavailable if two VMs fail simultaneously.
To protect quorum, distribute VMs across physical nodes and prevent disruptive operations from affecting more than one control plane VM at a time.
Live migration is not supported for virtualized control plane VMs. The eviction strategy for control plane VMs must be set to None to prevent the hosting cluster from attempting to migrate them during node drain operations.
1.5.1. Anti-affinity for virtualized control planes
Anti-affinity rules ensure that control plane VMs are distributed across different physical nodes on the hosting cluster so that a single node failure affects only one control plane VM.
The following methods are available for configuring this placement:
-
Pod anti-affinity rules in the
VirtualMachinespecification, which prevent multiple control plane VMs from being scheduled on the same node. - Topology spread constraints, which evenly distribute VMs across failure domains.
A hosting cluster requires at least three physical nodes to support anti-affinity placement for the three control plane VMs of a target cluster.
1.5.2. Pod disruption budgets for virtualized control planes
A PodDisruptionBudget (PDB) prevents the hosting cluster from evicting more than one control plane VM at a time during node drain or maintenance operations. You can configure a PDB to ensure that at least two control plane VMs remain running, which preserves etcd quorum.
Without a PDB, draining multiple hosting cluster nodes concurrently could evict more than one control plane VM at a time and cause quorum loss.
1.5.3. QEMU guest agent for virtualized control planes
The QEMU guest agent is required on each control plane VM. The hosting cluster uses a readiness probe to ping the guest agent and verify that the VM is healthy. This health check enables the hosting cluster to safely coordinate node drain operations while maintaining etcd quorum.
Without the guest agent, the readiness probe cannot confirm that a VM is healthy, and the pod disruption budget cannot manage drain operations correctly.
1.5.4. Readiness probes for virtualized control planes
Readiness probes verify that the VM operating system and critical services are operational. The hosting cluster uses readiness probe results to determine whether a VM is healthy before allowing operations that affect other VMs in the same target cluster.
Readiness probes work together with pod disruption budgets to ensure that the hosting cluster does not proceed with draining a node until the previously affected control plane VM is confirmed healthy on another node.
1.5.5. Hosting cluster maintenance for virtualized control planes
Hosting cluster maintenance, such as operating system updates or hardware repairs, can disrupt control plane VMs on the affected node. To preserve etcd quorum during these operations, maintenance must follow a sequential, one-node-at-a-time approach.
Before any node is cordoned, all control plane VMs must be healthy and etcd quorum must be intact. Only one hosting cluster node is drained at a time. After the drained node’s control plane VM shuts down, the remaining control plane VMs must still maintain etcd quorum before maintenance proceeds. The node is returned to service and the VM and etcd member are verified healthy before the next node is drained.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}