Virtualized control planes

1. Install and configure KubeVirt Redfish on the hosting cluster. This includes defining which VMs to expose through the Redfish API, configuring authentication credentials, and creating a Route CR to expose the endpoint externally.
2. Create the control plane VMs on the hosting cluster. Configure the VMs with appropriate resources and network settings, and ensure they remain powered off until the installation begins.
3. Configure your installation method to use KubeVirt Redfish. In your configuration files, specify BMC addresses using the KubeVirt Redfish route URL for the virtualized control plane nodes, for example: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.<vm_name>.
4. Run the installation. The VMs boot from the installation media and communicate with each other to form the cluster. Depending on the installation method, this process is either fully automated or requires manual intervention to boot each node.
5. After installation completes, a new OpenShift Container Platform cluster is deployed with its control plane running on VMs hosted by the original OpenShift Virtualization cluster.

Procedure

1. Create the Namespace CR for KubeVirt Redfish by creating a YAML file with content such as the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: kubevirt-redfish
     labels:
       app.kubernetes.io/name: kubevirt-redfish

2. Apply the resource by running the following command:

   $ oc apply -f namespace.yaml

3. Create the ServiceAccount CR by creating a YAML file with content such as the following example:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: kubevirt-redfish
     namespace: kubevirt-redfish
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac

4. Apply the resource by running the following command:

   $ oc apply -f serviceaccount.yaml

5. Create the ClusterRole CR with required permissions by creating a YAML file with content such as the following example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: kubevirt-redfish-role
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac
   rules:
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines", "virtualmachineinstances"]
       verbs: ["get", "list", "watch", "update", "patch"]
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines/status", "virtualmachineinstances/status"]
       verbs: ["get", "list", "watch", "patch"]
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines/restart", "virtualmachines/start", "virtualmachines/stop"]
       verbs: ["create"]
     - apiGroups: ["subresources.kubevirt.io"]
       resources: ["virtualmachineinstances/pause", "virtualmachineinstances/unpause"]
       verbs: ["create", "update"]
     - apiGroups: [""]
       resources: ["pods", "services", "configmaps", "secrets"]
       verbs: ["get", "list", "watch", "create", "update", "delete"]
     - apiGroups: [""]
       resources: ["namespaces"]
       verbs: ["get", "list"]
     - apiGroups: ["cdi.kubevirt.io"]
       resources: ["datavolumes", "volumeimportsources"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
     - apiGroups: [""]
       resources: ["persistentvolumeclaims"]
       verbs: ["get", "list", "create", "update", "patch", "delete", "watch"]
     - apiGroups: ["storage.k8s.io"]
       resources: ["storageclasses"]
       verbs: ["get", "list"]

6. Apply the resource by running the following command:

   $ oc apply -f clusterrole.yaml

7. Create the ClusterRoleBinding CR by creating a YAML file with content such as the following example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kubevirt-redfish-binding
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: kubevirt-redfish-role
   subjects:
     - kind: ServiceAccount
       name: kubevirt-redfish
       namespace: kubevirt-redfish

8. Apply the resource by running the following command:

   $ oc apply -f clusterrolebinding.yaml

9. Create the Secret CR containing the configuration by creating a YAML file with content such as the following example. Edit the config.yaml section to match your environment:

   apiVersion: v1
   kind: Secret
   metadata:
     name: kubevirt-redfish-secret
     namespace: kubevirt-redfish
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: config
   type: Opaque
   stringData:
     config.yaml: |
       server:
         host: "0.0.0.0"
         port: 8443
         tls:
           enabled: false
       system_id_convention: "enhanced"
       chassis:
         - name: "<chassis_name>"
           namespace: "<vm_namespace>"
           service_account: "kubevirt-redfish"
           vm_selector:
             labels:
               redfish-enabled: "true"
       authentication:
         users:
           - username: "admin"
             password: "<password>"
             chassis: ["<chassis_name>"]
       datavolume:
         storage_class: "<storage_class>"
         storage_size: "3Gi"

   where:

   • system_id_convention specifies the format for Redfish system IDs. The recommended setting is enhanced to use <namespace>.<vm-name> format. The legacy setting uses <vm-name> only.
   • chassis specifies the namespaces where VMs are deployed. Replace <chassis_name> with a name for this chassis configuration and <vm_namespace> with the namespace containing your VMs. The vm_selector labels identify which VMs in the namespace are exposed through Redfish. Only VMs with matching labels are visible. You can configure multiple chassis entries to expose different subsets of VMs in the same namespace, each with different authentication users.
   • authentication specifies the username and password required to access the Redfish API. These credentials enable full management control over exposed VMs, independently of any OpenShift Container Platform privileges. Replace <password> with a secure password.
   • datavolume specifies storage for VirtualMedia operations. Replace <storage_class> with a storage class available on your cluster, such as lvms-vg1 or ocs-storagecluster-ceph-rbd-virtualization. For more information about storage options, see Storage requirements in "Prerequisites for virtualized control planes".

10. Apply the resource by running the following command:

    $ oc apply -f secret.yaml

    Warning

    The credentials defined in this Secret CR enable full management control over the VMs exposed through KubeVirt Redfish, independently of any OpenShift Container Platform privileges.

11. Create the Deployment CR by creating a YAML file with content such as the following example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: kubevirt-redfish
          app.kubernetes.io/component: server
      template:
        metadata:
          labels:
            app.kubernetes.io/name: kubevirt-redfish
            app.kubernetes.io/component: server
        spec:
          serviceAccountName: kubevirt-redfish
          securityContext:
            runAsNonRoot: true
          containers:
            - name: kubevirt-redfish
              image: registry.redhat.io/container-native-virtualization/kubevirt-redfish-rhel9:v4.22
              imagePullPolicy: Always
              ports:
                - name: http
                  containerPort: 8443
                  protocol: TCP
              env:
                - name: CONFIG_PATH
                  value: "/app/config/config.yaml"
                - name: LOG_LEVEL
                  value: "info"
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "100m"
                limits:
                  memory: "2Gi"
                  cpu: "500m"
              livenessProbe:
                httpGet:
                  path: /redfish/v1/
                  port: 8443
                  scheme: HTTP
                initialDelaySeconds: 30
                periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /redfish/v1/
                  port: 8443
                  scheme: HTTP
                initialDelaySeconds: 5
                periodSeconds: 5
              securityContext:
                runAsNonRoot: true
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
              volumeMounts:
                - name: config-volume
                  mountPath: /app/config
                  readOnly: true
          volumes:
            - name: config-volume
              secret:
                secretName: kubevirt-redfish-secret

    where:

    • The image field specifies the KubeVirt Redfish container image.

12. Apply the resource by running the following command:

    $ oc apply -f deployment.yaml

13. Create the Service CR by creating a YAML file with content such as the following example:

    apiVersion: v1
    kind: Service
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      type: ClusterIP
      ports:
        - name: http
          port: 8443
          targetPort: 8443
          protocol: TCP
      selector:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server

14. Apply the resource by running the following command:

    $ oc apply -f service.yaml

15. Create the Route CR to expose the Redfish API externally by creating a YAML file with content such as the following example:

    apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      port:
        targetPort: http
      to:
        kind: Service
        name: kubevirt-redfish
        weight: 100
      tls:
        termination: edge
        insecureEdgeTerminationPolicy: Redirect

16. Apply the resource by running the following command:

    $ oc apply -f route.yaml

Verification

1. Verify that the pods are running by running the following command:

   $ oc get pods -n kubevirt-redfish

   Example output

   NAME                                READY   STATUS    RESTARTS   AGE
   kubevirt-redfish-587cd94988-xthml   1/1     Running   0          2m

2. Get the route hostname by running the following command:

   $ oc get route kubevirt-redfish -n kubevirt-redfish -o jsonpath='{.spec.host}'

3. Test the Redfish endpoint by running the following command:

   $ curl -sk -u "admin:<password>" https://<route_hostname>/redfish/v1/

   A successful response returns JSON with the Redfish service root:

   {
     "@odata.id": "/redfish/v1",
     "@odata.type": "#ServiceRoot.v1_0_0.ServiceRoot",
     "Id": "RootService",
     "Name": "Root Service",
     "Systems": {
       "@odata.id": "/redfish/v1/Systems"
     }
   }

Procedure

1. Enable the RebootPolicy feature gate on the hosting cluster by running the following command:

   $ oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
       kubevirt.kubevirt.io/jsonpatch='[{"op":"add","path":"/spec/configuration/developerConfiguration/featureGates/-","value":"RebootPolicy"}]'

   Note

   The RebootPolicy feature gate enables the rebootPolicy field in VirtualMachine specifications. This configuration is required when using KubeVirt Redfish for cluster installation. The feature gate is enabled through an annotation on the HyperConverged resource, which propagates the configuration to the underlying KubeVirt CR.

2. Enable the declarativeHotplugVolumes feature gate on the hosting cluster by running the following command:

   $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
       --type merge \
       -p '{"spec": {"featureGates": {"declarativeHotplugVolumes": true}}}'

   Note

   The declarativeHotplugVolumes feature gate enables KubeVirt Redfish to dynamically attach boot media to VMs through the Redfish API. This configuration is required when using KubeVirt Redfish for cluster installation.

3. Create a VirtualMachine CR for each control plane node by creating a YAML file with content such as the following example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: master-0
     namespace: <vm_namespace>
     labels:
       redfish-enabled: "true"
   spec:
     runStrategy: Halted
     template:
       metadata:
         labels:
           redfish-enabled: "true"
       spec:
         domain:
           rebootPolicy: Terminate
           cpu:
             cores: 8
           memory:
             guest: 16Gi
           devices:
             disks:
               - name: rootdisk
                 disk:
                   bus: virtio
               - name: cloudinitdisk
                 disk:
                   bus: virtio
             interfaces:
               - name: default
                 bridge: {}
         networks:
           - name: default
             multus:
               networkName: <network_attachment_definition>
         volumes:
           - name: rootdisk
             dataVolume:
               name: master-0-disk
           - name: cloudinitdisk
             cloudInitNoCloud:
               userData: |
                 #cloud-config
                 hostname: master-0
                 user: core

   where:

   • <vm_namespace> specifies the namespace for the VMs. Must match the namespace specified in the KubeVirt Redfish chassis configuration.
   • redfish-enabled: "true" specifies the label that must match the vm_selector labels in the KubeVirt Redfish configuration so the VM is exposed through the Redfish API.
   • runStrategy: Halted specifies that VMs must be powered off initially. The installation powers them on by using the Redfish API.
   • rebootPolicy: Terminate specifies the reboot behavior required for Redfish API boot override operations. Ensures the VM terminates cleanly when boot media changes.
   • cores: 8 and guest: 16Gi specify the minimum recommended resources for control plane nodes.

   • <network_attachment_definition> specifies the name of a NetworkAttachmentDefinition configured on your hosting cluster. All control plane VMs must share the same L2 network segment. Common options include localnet, Linux bridge, or OVN Layer 2 networks.

     Important

     For production deployments, configure anti-affinity rules to ensure control plane VMs are distributed across different physical nodes. This prevents a single node failure from affecting multiple control plane VMs simultaneously. Add pod anti-affinity rules or topology spread constraints to the VM specification based on your environment requirements.

4. Apply the resource by running the following command:

   $ oc apply -f master-0.yaml

Procedure

1. Create an install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       provisioningNetwork: Disabled
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • platform: baremetal specifies full hardware lifecycle management. Set to none for infrastructure-agnostic installation.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled for agent-based installations on virtual media. Nodes boot from the agent ISO, not PXE.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Create an agent-config.yaml YAML file similar to the following example:

   apiVersion: v1alpha1
   kind: AgentConfig
   metadata:
     name: my-vcp-cluster
   rendezvousIP: 10.0.0.20

   where: * The rendezvousIP field specifies the IP address of the first control plane node. This node coordinates the installation.

3. Generate the agent ISO by running the following command:

   $ openshift-install agent create image --dir <installation_directory>

4. Host the generated agent.x86_64.iso on an HTTP server accessible from your hosting cluster.

5. Boot each node from the agent ISO:

   1. For virtualized control plane nodes, use KubeVirt Redfish to mount the ISO and power on the VMs.
   2. For baremetal worker nodes, use the server’s BMC interface to mount the ISO and boot.

6. Monitor the installation progress by running the following command:

   $ openshift-install agent wait-for install-complete --dir <installation_directory>

   • Replace <installation_directory> with the path to the directory where the agent ISO was generated.

Procedure

1. Create Secret custom resources (CRs) that contain the BMC credentials for each node. The following example shows a secret for a control plane node:

   apiVersion: v1
   kind: Secret
   metadata:
     name: master-0-bmc-secret
     namespace: my-vcp-cluster
   type: Opaque
   data:
     username: <base64_encoded_username>
     password: <base64_encoded_password>

   Create similar Secret CRs for each node, for example master-1-bmc-secret, worker-0-bmc-secret and so on. For virtualized control plane nodes, use the KubeVirt Redfish credentials. For baremetal worker nodes, use the physical server’s BMC credentials.

2. Create a ClusterInstance custom resource (CR) that defines the cluster. Specify BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: my-vcp-cluster
     namespace: my-vcp-cluster
   spec:
     baseDomain: example.com
     clusterImageSetNameRef: "openshift-4.22"
     clusterName: my-vcp-cluster
     clusterType: HighlyAvailable
     platformType: BareMetal
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - cidr: 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
     apiVIPs:
       - 10.0.0.10
     ingressVIPs:
       - 10.0.0.11
     pullSecretRef:
       name: assisted-deployment-pull-secret
     sshPublicKey: "<ssh_public_key>"
     extraLabels:
       ManagedCluster:
         common: "true"
         sites: "my-vcp-cluster"
     templateRefs:
       - name: ai-cluster-templates-v1
         namespace: open-cluster-management
     nodes:
       - hostName: master-0.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
         bmcCredentialsName:
           name: master-0-bmc-secret
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-1.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
         bmcCredentialsName:
           name: master-1-bmc-secret
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: master-2.my-vcp-cluster.example.com
         role: master
         bmcAddress: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
         bmcCredentialsName:
           name: master-2-bmc-secret
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/vda
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-0.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-0-bmc-secret
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management
       - hostName: worker-1.my-vcp-cluster.example.com
         role: worker
         bmcAddress: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
         bmcCredentialsName:
           name: worker-1-bmc-secret
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         automatedCleaningMode: disabled
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         templateRefs:
           - name: ai-node-templates-v1
             namespace: open-cluster-management

   where:

   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • bmcAddress for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • bmcCredentialsName specifies a reference to a Secret containing BMC credentials.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmcAddress for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
3. Commit the manifests to your Git repository.

4. Apply the configuration through ArgoCD or your GitOps tooling.

   ZTP generates the required resources and uses KubeVirt Redfish to provision the VMs automatically.

Procedure

1. Create the install-config.yaml file with BMC addresses pointing to KubeVirt Redfish for the virtualized control plane nodes and real Redfish endpoints for the baremetal worker nodes:

   apiVersion: v1
   metadata:
     name: my-vcp-cluster
   baseDomain: example.com
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     replicas: 3
     platform:
       baremetal: {}
   compute:
     - name: worker
       architecture: amd64
       hyperthreading: Enabled
       replicas: 2
       platform:
         baremetal: {}
   networking:
     networkType: OVNKubernetes
     clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
     serviceNetwork:
       - 172.30.0.0/16
     machineNetwork:
       - cidr: 10.0.0.0/24
   platform:
     baremetal:
       apiVIPs:
         - 10.0.0.10
       ingressVIPs:
         - 10.0.0.11
       provisioningNetwork: Disabled
       hosts:
       - name: master-0
         role: master
         bootMACAddress: 52:54:00:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-0
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-1
         role: master
         bootMACAddress: 52:54:00:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-1
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: master-2
         role: master
         bootMACAddress: 52:54:00:00:00:03
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/vda
         bmc:
           address: redfish-virtualmedia+https://<kubevirt_redfish_route>/redfish/v1/Systems/<vm_namespace>.master-2
           username: admin
           password: <password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: enp1s0
               mac-address: 52:54:00:00:00:03
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-0
         role: worker
         bootMACAddress: e4:43:4b:00:00:01
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:01
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
       - name: worker-1
         role: worker
         bootMACAddress: e4:43:4b:00:00:02
         bootMode: UEFI
         rootDeviceHints:
           deviceName: /dev/nvme0n1
         bmc:
           address: redfish-virtualmedia+https://<worker_bmc_ip>/redfish/v1/Systems/System.Embedded.1
           username: <bmc_username>
           password: <bmc_password>
           disableCertificateVerification: false
         networkConfig:
           interfaces:
             - name: ens1f0
               mac-address: e4:43:4b:00:00:02
               type: ethernet
               state: up
               ipv4:
                 enabled: true
                 dhcp: true
                 auto-dns: true
               ipv6:
                 enabled: true
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_public_key>'

   where:

   • replicas specifies the number of worker node replicas to match the number of baremetal worker hosts defined in the hosts section.
   • apiVIPs and ingressVIPs specifies the virtual IP addresses that must be allocated from the machine network. The apiVIPs must reside on the same L2 network segment as the control plane VMs, and the ingressVIPs must reside on the same L2 network segment as the worker nodes.
   • provisioningNetwork: Disabled specifies that the provisioning network is disabled when using virtual media.
   • bootMACAddress specifies the MAC address used for network boot. When using DHCP, ensure this MAC has a reserved IP address configured in your DHCP server.
   • deviceName: /dev/vda specifies the installation disk for virtualized control plane nodes.
   • bmc.address for control plane nodes specifies the KubeVirt Redfish route. The <vm_namespace>.<vm_name> format corresponds to the enhanced system ID convention configured in KubeVirt Redfish. Replace <kubevirt_redfish_route> with your route hostname and <vm_namespace> with the namespace containing your VMs.
   • disableCertificateVerification specifies whether to skip TLS certificate validation. For production deployments, configure properly signed TLS certificates and set to false. Set to true only for lab or development environments.
   • networkConfig specifies the host network configuration for each node. This example uses DHCP.
   • deviceName: /dev/nvme0n1 specifies the installation disk for baremetal worker nodes.
   • bmc.address for worker nodes specifies the real Redfish endpoint of the physical server. Replace <worker_bmc_ip> with the BMC IP address.

2. Run the installation by running the following command:

   $ openshift-install create cluster --dir <installation_directory>

   The installation process uses KubeVirt Redfish to manage VM power states and boot configuration automatically.