Chapter 4. Deploying a virtualized control plane

Procedure

1. Create an install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       provisioningNetwork: Disabled
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • platform: baremetal specifies full hardware lifecycle management. Set to none for infrastructure-agnostic installation.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled for agent-based installations on virtual media. Nodes boot from the agent ISO, not PXE.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Create an agent-config.yaml YAML file similar to the following example:

   apiVersion: v1alpha1
   kind: AgentConfig
   metadata:
     name: my-vcp-cluster
   rendezvousIP: 10.0.0.20

   where: * The rendezvousIP field specifies the IP address of the first control plane node. This node coordinates the installation.

3. Generate the agent ISO by running the following command:

   $ openshift-install agent create image --dir <installation_directory>

4. Host the generated agent.x86_64.iso on an HTTP server accessible from your hosting cluster.

5. Boot each node from the agent ISO:

   1. For virtualized control plane nodes, use KubeVirt Redfish to mount the ISO and power on the VMs.
   2. For baremetal worker nodes, use the server’s BMC interface to mount the ISO and boot.

6. Monitor the installation progress by running the following command:

   $ openshift-install agent wait-for install-complete --dir <installation_directory>

   • Replace <installation_directory> with the path to the directory where the agent ISO was generated.

Procedure

1. Create Secret custom resources (CRs) that contain the BMC credentials for each node. The following example shows a secret for a control plane node:

   apiVersion: v1
   kind: Secret
   metadata:
     name: master-0-bmc-secret
     namespace: my-vcp-cluster
   type: Opaque
   data:
     username: <base64_encoded_username>
     password: <base64_encoded_password>

   Create similar Secret CRs for each node, for example master-1-bmc-secret, worker-0-bmc-secret and so on. For virtualized control plane nodes, use the KubeVirt Redfish credentials. For baremetal worker nodes, use the physical server’s BMC credentials.

2. Create a ClusterInstance custom resource (CR) that defines the cluster. Specify BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: my-vcp-cluster
     namespace: my-vcp-cluster
   spec:
     baseDomain: example.com
     clusterImageSetNameRef: "openshift-4.22"
     clusterName: my-vcp-cluster
     clusterType: HighlyAvailable
     platformType: BareMetal
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - cidr: 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
     apiVIPs:
       - 10.0.0.10
     ingressVIPs:
       - 10.0.0.11
     pullSecretRef:
       name: assisted-deployment-pull-secret
     sshPublicKey: "<ssh_public_key>"
     extraLabels:
       ManagedCluster:
         common: "true"
         sites: "my-vcp-cluster"
     templateRefs:
       - name: ai-cluster-templates-v1
         namespace: open-cluster-management
     nodes:
       - hostName: master-0.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
         bmcCredentialsName:
           name: master-0-bmc-secret
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-1.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
         bmcCredentialsName:
           name: master-1-bmc-secret
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-2.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
         bmcCredentialsName:
           name: master-2-bmc-secret
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-0.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-0-bmc-secret
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-1.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-1-bmc-secret
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management

   where:

   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bmcAddress for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • bmcCredentialsName specifies a reference to a Secret containing BMC credentials.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmcAddress for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
3. Commit the manifests to your Git repository.

4. Apply the configuration through ArgoCD or your GitOps tooling.

   ZTP generates the required resources and uses KubeVirt Redfish to provision the VMs automatically.

Procedure

1. Create the install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
     platform:
       baremetal: {}
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
       platform:
         baremetal: {}
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       provisioningNetwork: Disabled
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled when using virtual media.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Run the installation by running the following command:

   $ openshift-install create cluster --dir <installation_directory>

   The installation process uses KubeVirt Redfish to manage VM power states and boot configuration automatically.