Chapter 5. Postinstallation configuration

Procedure

1. Edit the object in your default editor by running the following command:

   $ oc edit <resource_type> <resource_name> -n openshift-cnv

2. Save the file to apply the changes.

Procedure

1. Verify that the NFD Operator has labeled the LPAR nodes by running the following command:

   $ oc get nodes -L feature.node.kubernetes.io/cpu-model.hypervisor

   Example output

   worker-lpar                PR_SM
   worker-zvm                 z_VM_7.4.0
   worker-kvm                 KVM_Linux

   Nodes labeled PR_SM are valid VM targets. Nodes labeled KVM_Linux or z_VM_7.4.0 are not supported VM targets.

2. Patch the HyperConverged CR to restrict OpenShift Virtualization infrastructure components and VM workloads to PR_SM nodes by running the following command:

   $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
     --type=merge -p '
   spec:
     infra:
       nodePlacement:
         nodeSelector:
           feature.node.kubernetes.io/cpu-model.hypervisor: "PR_SM"
     workloads:
       nodePlacement:
         nodeSelector:
           feature.node.kubernetes.io/cpu-model.hypervisor: "PR_SM"
   '

   where:

   spec.infra.nodePlacement

   Specifies the node placement for OpenShift Virtualization infrastructure components, including virt-handler. Setting this field prevents virt-handler rollout failures on non-LPAR worker nodes.

   spec.workloads.nodePlacement

   Specifies the default scheduling placement for all VM pods (virt-launcher) across the cluster, ensuring that VMs start only on PR_SM LPAR nodes without requiring per-VM nodeSelector definitions.

   Note

   In environments where spec.infra.nodePlacement does not fully prevent virt-handler scheduling on non-LPAR nodes, you can protect those nodes by applying a taint.

   $ oc adm taint nodes <non_lpar_node> kubevirt.io/no-virt=true:NoSchedule

Verification

1. Verify that virt-handler pods are running only on PR_SM LPAR nodes by running the following command:

   $ oc get pods -n openshift-cnv -l kubevirt.io=virt-handler -o wide

   Confirm that no virt-handler pods are running on nodes labeled KVM_Linux or z_VM_7.4.0.

2. Verify that the HyperConverged Operator is healthy by running the following command:

   $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv

   The output must show Available: True and Degraded: False.

3. Optional: Deploy a test VM without a nodeSelector to confirm cluster-wide placement by running the following command:

   $ oc apply -f - <<EOF
   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: test-vm-placement
     namespace: default
   spec:
     runStrategy: Always
     template:
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: containerdisk
           resources:
             requests:
               memory: 512Mi
         volumes:
         - containerDisk:
             image: quay.io/kubevirt/cirros-container-disk-demo:latest
           name: containerdisk
   EOF

4. Verify that the test VM runs on a PR_SM LPAR node by running the following command:

   $ NODE=$(oc get pod -n default -l vm.kubevirt.io/name=test-vm-placement \
       -o jsonpath='{.items[0].spec.nodeName}') && \
     echo "Scheduled node: $NODE" && \
     oc get node "$NODE" -L feature.node.kubernetes.io/cpu-model.hypervisor

   The node must report PR_SM in the CPU-MODEL.HYPERVISOR column.

5. Delete the test VM by running the following command:

   $ oc delete vm test-vm-placement -n default

Procedure

1. Configure the NNCP manifest. This example includes sample values that you must replace with your own information.

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: br1-eth1-policy
   spec:
     desiredState:
       interfaces:
         - name: br1
           description: Linux bridge with eth1 as a port
           type: linux-bridge
           state: up
           ipv4:
             enabled: false
           bridge:
             options:
               stp:
                 enabled: false
             port:
               - name: eth1

   • metadata.name: Defines the name of the node network configuration policy.
   • spec.desiredState.interfaces.name: Defines the name of the new Linux bridge.
   • spec.desiredState.interfaces.description: Optional field that can be used to define a human-readable description for the bridge.
   • spec.desiredState.interfaces.type: Defines the interface type. In this example, the type is a Linux bridge.
   • spec.desiredState.interfaces.state: Defines the requested state for the interface after creation.
   • spec.desiredState.interfaces.ipv4.enabled: Defines whether the ipv4 protocol is active. Setting this to false disables IPv4 addressing on this bridge.
   • spec.desiredState.interfaces.bridge.options.stp.enabled: Defines whether Spanning Tree Protocol (STP) is active. Setting this to false disables STP on this bridge.

   • spec.desiredState.interfaces.bridge.port.name: Defines the node NIC that the bridge is attached to.

     Note

     To create the NNCP manifest for a Linux bridge using Open Systems Adapter (OSA) with IBM Z®, you must disable VLAN filtering by the setting the rx-vlan-filter to false in the NodeNetworkConfigurationPolicy manifest.

     Alternatively, if you have SSH access to the node, you can disable VLAN filtering by running the following command:

     $ sudo ethtool -K <osa-interface-name> rx-vlan-filter off

2. Create the NNCP manifest:

   $ oc apply -f linux-bridge-nncp.yaml

Procedure

1. In the web console, click Networking NetworkAttachmentDefinitions.

2. Click Create Network Attachment Definition.

   Note

   The network attachment definition must be in the same namespace as the pod or virtual machine.

3. Enter a unique Name and optional Description.
4. Select CNV Linux bridge from the Network Type list.
5. Enter the name of the bridge in the Bridge Name field.

6. Optional: If the resource has VLAN IDs configured, enter the ID numbers in the VLAN Tag Number field.

   Note

   Open Systems Adapter (OSA) interfaces on IBM Z® do not support VLAN filtering and drop VLAN-tagged traffic. Avoid using VLAN-tagged NetworkAttachmentDefinitions with OSA interfaces.

7. Optional: Select MAC Spoof Check to enable MAC spoof filtering. This feature provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod.

8. Optional: In the YAML tab, add the spec.config.disableContainerInterface field. When set to true, the Bridge CNI plug-in skips creating a standard container virtual ethernet (veth) interface inside the pod’s network namespace. This enables the virtualization networking backend to attach the layer-2 interface directly to the guest VM instead of a standard container.

   Example YAML:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-network
     annotations:
       k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1
   spec:
     config: |
       {
   # ...
         "disableContainerInterface": true,
   # ...
       }

9. Click Create.

Procedure

1. Create a NetworkAttachmentDefinition manifest according to the following example:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: my-secondary-network
     namespace: openshift-cnv
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "migration-bridge",
       "type": "macvlan",
       "master": "eth1",
       "mode": "bridge",
       "ipam": {
         "type": "whereabouts",
         "range": "10.200.5.0/24"
       }
     }'

   • metadata.name: Defines the name of the NetworkAttachmentDefinition object.
   • config.master: Defines the name of the NIC to be used for live migration.
   • config.type: Defines the name of the CNI plugin that provides the network for the NetworkAttachmentDefinition.
   • config.range: Defines an IP address range for the secondary network. This range must not overlap the IP addresses of the main network.

2. Open the HyperConverged CR in your default editor by running the following command:

   $ oc edit hco kubevirt-hyperconverged -n openshift-cnv

3. Add the name of the NetworkAttachmentDefinition object to the spec.virtualization.liveMigrationConfig stanza of the HyperConverged CR.

   Example HyperConverged manifest:

   apiVersion: hco.kubevirt.io/v1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     virtualization:
       liveMigrationConfig:
         completionTimeoutPerGiB: 800
         network: <network>
         parallelMigrationsPerCluster: 5
         parallelOutboundMigrationsPerNode: 2
         progressTimeout: 150
   # ...

   • spec.virtualization.liveMigrationConfig.network: Defines the name of the Multus NetworkAttachmentDefinition object to be used for live migrations.
4. Save your changes and exit the editor. The virt-handler pods restart and connect to the secondary network.

Procedure

1. Go to Virtualization Settings in the OpenShift Container Platform web console.
2. On the Cluster tab, click General settiings.
3. Click Live Migration.
4. Select the network from the Live migration network list.

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: <name>
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: <sriov_resource_name>
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     mtu: <mtu>
     numVfs: <num>
     nicSelector:
       vendor: "<vendor_code>"
       deviceID: "<device_id>"
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: vfio-pci
     isRdma: false

   • metadata.name defines a name for the SriovNetworkNodePolicy object.
   • metadata.namespace defines the namespace where the SR-IOV Network Operator is installed.
   • spec.resourceName defines the resource name of the SR-IOV device plugin. You can create multiple SriovNetworkNodePolicy objects for a resource name.
   • spec.nodeSelector.feature.node.kubernetes.io/network-sriov.capable defines the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed only on selected nodes.
   • spec.priority is an optional field that defines an integer value between 0 and 99. A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99. The default value is 99.
   • spec.mtu is an optional field that defines a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.
   • spec.numVfs defines the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 127.

   • spec.nicSelector defines the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters.

     Note

     It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames.

     If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device.

   • spec.nicSelector.vendor is an optional field that defines the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3.
   • spec.nicSelector.deviceID is an optional field that defines the device hex code of SR-IOV network device. The only allowed values are 158b, 1015, 1017.
   • spec.nicSelector.pfNames is an optional field that defines an array of one or more physical function (PF) names for the Ethernet device.
   • spec.nicSelector.rootDevices is an optional field that defines an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1.
   • spec.deviceType defines the driver type. The vfio-pci driver type is required for virtual functions in OpenShift Virtualization.

   • spec.isRdma is an optional field that defines whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false. The default value is false.

     Note

     If isRDMA flag is set to true, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.

2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".

3. Create the SriovNetworkNodePolicy object. When running the following command, replace <name> with the name for this configuration:

   $ oc create -f <name>-sriov-node-network.yaml

   After applying the configuration update, all the pods in the sriov-network-operator namespace change to the Running status.

4. To verify your SR-IOV network device configuration, enter the following command and replace <node_name> with the name of the node where you configured the device.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Procedure

1. Go to Virtualization Settings.
2. Click Cluster.
3. Expand General settings and SSH configuration.
4. Set SSH over LoadBalancer service to on.

Procedure

1. Configure the route to the external host by running the following command:

   $ oc create route reencrypt <route_name> -n openshift-cnv \
       --insecure-policy=Redirect \
       --hostname=<host_name_or_address> \
       --service=cdi-uploadproxy

   where:

   <route_name>

   Specifies the name to assign to this custom route.

   <host_name_or_address>

   Specifies the fully qualified domain name or IP address of the external host providing image upload access.

2. Run the following command to annotate the route. This ensures that the correct Containerized Data Importer (CDI) CA certificate is injected when certificates are rotated:

   $ oc annotate route <route_name> -n openshift-cnv \
       operator.cdi.kubevirt.io/injectUploadProxyCert="true"

   where:

   <route_name>

   Specifies the name of the route you created.

Procedure

1. In the OpenShift Container Platform web console, go to Virtualization Checkups.
2. Go to the Self validation tab.
3. Click Run checkup.
4. Configure settings for the test that you want to run.
5. Optional: You can enable a dry run test by clicking Advanced settings and then toggling the Dry run button.

6. Click Run. The Self validation checkup details page is displayed.

   You can observe the self validation checkup running in real time. The test can take several hours to complete.

7. After the test is complete, you can view high-level results in the Self validation checkup details page, including the names of any failing tests.
8. Optional: You can download detailed results as a ZIP file by clicking Download results in the Self validation checkup details page.

Procedure

1. Edit the KubeletConfig CR to add the required cpuManagerPolicy and cpuManagerPolicyOptions configurations:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   # ...
   cpuManagerPolicy: static
   cpuManagerPolicyOptions:
     full-pcpus-only: true
   kubeReserved:
     cpu: "1"
   # ...

   • You must set the cpuManagerPolicy: static policy to enable exclusive CPU allocation. This setting is a prerequisite for configuring the cpuManagerPolicyOptions settings.
   • You must set the full-pcpus-only: true policy option so that the static CPU Manager policy only allocates full physical cores.
   • You must reserve 1 CPU for the system by setting cpu: "1" in the kubeReserved settings. This ensures that the cluster remains stable, by requiring that the system’s core functions always have access to the CPU that they need to work correctly.

2. Run the following command to apply the changes to the KubeletConfig CR:

   $ oc apply -f <filename>.yaml

Procedure

1. Select a worker node where you want to enable memory overcommitment and start a debug session by running the following command:

   $ oc debug node/<node_name>

   Replace <node_name> with the name of the worker node.

2. Enter the host context by running the following command:

   $ chroot /host

3. List the available NVMe disks and their World Wide Names (WWNs) by running the following command:

   $ lsblk -p -o +WWN

   Example output

   NAME         TYPE MODEL                  SERIAL            REV TRAN   RQ-SIZE  MQ WWN
   /dev/nvme0n1 disk KXG8AZNV512G LA KIOXIA 63QFXXXXXXEU 5108APLA nvme      1023  12 eui.8ce38e0500e775a4

4. Find the device path by using its WWN. The path follows the /dev/disk/by-id/nvme-<WWN> format. Run the following command, replacing <WWN> with the value from the previous command:

   $ ls /dev/disk/by-id/nvme-<WWN>

   Example output

   /dev/disk/by-id/nvme-eui.8ce38e0500e775a4

5. Create a GUID Partition Table (GPT) partition table and an OCPSWAP partition by running the following command, replacing <WWN> with the WWN of the selected device:

   Warning

   The following command erases all data on the selected NVMe device. Back up any data that you need to retain before you continue.

   $ echo -e 'label: gpt\nname="OCPSWAP", type=swap' | sfdisk --wipe always /dev/disk/by-id/nvme-<WWN>

   The partition name must be OCPSWAP so that the virt-platform-autopilot controller can discover and enable the partition.

   Warning

   Only one partition on the node can use the OCPSWAP label. If more than one partition uses this label, the /dev/disk/by-partlabel/OCPSWAP path does not resolve deterministically, and later commands can format or enable the wrong partition.

6. Format the partition as Linux swap and set its filesystem label to OCPSWAP by running the following command:

   $ mkswap -L OCPSWAP /dev/disk/by-partlabel/OCPSWAP

   The /dev/disk/by-partlabel/OCPSWAP path resolves to the partition that you created in the previous step. The -L OCPSWAP option sets the swap filesystem label, which the virt-platform-autopilot controller uses to identify the swap partition. You must format the partition, not the whole NVMe device.

7. Verify the swap partition before rebooting the node by running the following commands:

   $ swapon /dev/disk/by-partlabel/OCPSWAP
   $ free -m

   Example output

                  total        used        free      shared  buff/cache   available
   Mem:           31728       20964         906        4586       12269       10764
   Swap:           8191          95        8096

8. Reboot the node by running the following command:

   $ systemctl reboot

Verification

1. After the node reboots, start a debug session by running the following command:

   $ oc debug node/<node_name>

2. Enter the host context by running the following command:

   $ chroot /host

3. Verify that the virt-platform-autopilot controller discovered the swap partition by running the following command:

   $ ls /dev/disk/by-partlabel/OCPSWAP

4. Verify that swap space is available by running the following command:

   $ free -m

   Example output

                  total        used        free      shared  buff/cache   available
   Mem:           31728       20964         906        4586       12269       10764
   Swap:           8191          95        8096

Procedure

1. Create a MachineConfig file with the parameters shown in the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 90-worker-swap
   spec:
     config:
       ignition:
         version: 3.5.0
       systemd:
         units:
           - contents: |
               [Unit]
               Description=Provision and enable swap
               ConditionFirstBoot=no
               ConditionPathExists=!/var/tmp/swapfile
   
               [Service]
               Type=oneshot
               Environment=SWAP_SIZE_MB=5000
               ExecStart=/bin/sh -c "sudo fallocate -l ${SWAP_SIZE_MB}M /var/tmp/swapfile && \
               sudo chmod 600 /var/tmp/swapfile && \
               sudo mkswap /var/tmp/swapfile && \
               sudo swapon /var/tmp/swapfile && \
               free -h"
   
               [Install]
               RequiredBy=kubelet-dependencies.target
             enabled: true
             name: swap-provision.service
           - contents: |
               [Unit]
               Description=Enable swap
               ConditionFirstBoot=no
               ConditionPathExists=/var/tmp/swapfile
   
               [Service]
               Type=oneshot
               ExecStart=/bin/sh -c "sudo swapon /var/tmp/swapfile"
   
               [Install]
               RequiredBy=kubelet-dependencies.target
             enabled: true
             name: swap-enable.service

   Set the SWAP_SIZE_MB value to the amount of swap space to provision on the node, in MB.

2. Wait for the worker nodes to sync with the new configuration by running the following command:

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

Verification

1. View a list of worker nodes by running the following command:

   $ oc get nodes -l node-role.kubernetes.io/worker

2. Select a node from the list and display its memory usage by running the following command:

   $ oc debug node/<selected_node> -- free -m

   Replace <selected_node> with the node name.

   The Swap: row displays an amount greater than zero if the node has swap enabled.

   Example output

                  total        used        free      shared  buff/cache   available
   Mem:           31846       23155        1044        6014       14483        8690
   Swap:           8191        2337        5854

Procedure

1. Place the node in maintenance mode by using the oc adm cordon and oc adm drain commands.

2. Verify that all pod and VM workloads are evicted from the node by running the following commands:

   $ oc get pods --all-namespaces --field-selector spec.nodeName=<node_name>

   $ oc get vmi -A --field-selector spec.nodeName=<node_name>

   Only DaemonSet pods should remain on the node.

3. Start a debug session by running the following command:

   $ oc debug node/<node_name>

   Replace <node_name> with the name of the worker node.

4. Enter the host context by running the following command:

   $ chroot /host

5. Disable the swap partition by running the following command:

   $ swapoff /dev/disk/by-partlabel/OCPSWAP

6. Verify that swap is disabled by running the following command:

   $ free -m

   Example output

                  total        used        free      shared  buff/cache   available
   Mem:           31728       20964         906        4586       12269       10764
   Swap:              0           0           0

   The Swap: row displays zero values when swap is disabled.

   Note

   If the Swap: row shows non-zero values, the swap partition is still active. Verify that all workloads are evacuated and retry the swapoff command. If the issue persists, reboot the node to force swap deactivation.

7. Remove the swap partition:

   1. Set the $DISK variable to the disk device path by running the following command:

      $ DISK="/dev/$(lsblk -no PKNAME /dev/disk/by-partlabel/OCPSWAP)"

   2. Delete the partition by running the following command:

      Warning

      This command removes all partitions on the device, not just the OCPSWAP partition. If you manually created additional partitions on this device, they will be deleted.

      $ sfdisk --delete $DISK

8. Verify that the OCPSWAP partition label is removed by running the following command:

   $ ls /dev/disk/by-partlabel/OCPSWAP

   Example output

   ls: cannot access '/dev/disk/by-partlabel/OCPSWAP': No such file or directory

   Note

   If the partition label still exists after running sfdisk --delete, verify the correct disk device was used and retry the deletion command. Reboot the node if the partition table was not updated.

9. Exit the debug session by running the following commands:

   $ exit

   $ exit

   The first exit command leaves the chroot environment. The second exit command terminates the debug pod.

10. Remove the node from maintenance mode by using the oc adm uncordon command.

Procedure

1. Choose one of the following methods to enable memory overcommitment in OpenShift Virtualization:

   After you enable memory overcommitment, only virtual machine workloads (VM pods) of the Burstable Quality of Service (QoS) class can use swap resources. VM pods of the Guaranteed QoS class and pods of any QoS class that do not belong to VMs cannot use swap resources. For descriptions of QoS classes, see Configure Quality of Service for Pods (Kubernetes documentation).

   When you configure virtual machines for memory overcommitment, use spec.domain.memory.guest in the VM manifest. Using spec.domain.resources.requests.memory disables the memory overcommit configuration.

   • To use the web console:

     1. In the OpenShift Container Platform web console, go to Virtualization Settings.
     2. Click the Cluster tab.
     3. Expand General settings.
     4. Expand Memory request ratio.

     5. In the Request ratio field, enter a value from 25 to 100.

        The request ratio is the percentage of each virtual machine’s configured memory that the cluster requests. A lower ratio increases overcommitment. At a ratio of 25%, a VM configured with 4 GiB of memory requests 1 GiB on the cluster. At a ratio of 100%, it requests the full 4 GiB.

        The web console indicates the level of risk for the value you enter. It shows Recommended at 75% or greater, Use with caution from 50% to 74.9%, and High overcommit risk from 25% to 49.9%.

        The Active ratio line shows the ratio currently calculated across all virtual machines on the cluster, which can differ from the ratio you request. It shows No running VMs if no virtual machines are running.

     6. Click Save.

   • To use the CLI, run the following command to enable higher memory density and set the overcommit rate:

     $ oc patch -n openshift-cnv hco kubevirt-hyperconverged --type='json' -p='[ \
       { \
       "op": "replace", \
       "path": "/spec/virtualization/higherWorkloadDensity/memoryOvercommitPercentage", \
       "value": 150 \
       } \
     ]'

     Example output

     hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

Procedure

1. Revert the memory overcommitment configuration by running the following command:

   $ oc patch -n openshift-cnv hco kubevirt-hyperconverged \
     --type='json' \
     -p='[{"op": "remove", "path": "/spec/virtualization/higherWorkloadDensity"}]'

2. Delete the MachineConfig objects that provision and configure swap by running the following commands:

   $ oc delete machineconfig 90-worker-swap

   $ oc delete machineconfig 90-worker-swap-online

3. Wait for the worker nodes to sync with the new configuration by running the following command:

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

Procedure

1. Create a NodeHealthCheck custom resource (CR) to define the health check criteria and remediation strategy:

   apiVersion: remediation.medik8s.io/v1alpha1
   kind: NodeHealthCheck
   metadata:
     name: nodehealthcheck-sample
   spec:
     minHealthy: <count>
     pauseRequests:
       - <pause_test_cluster>
     remediationTemplate:
       apiVersion: self-node-remediation.medik8s.io/v1alpha1
       name: self-node-remediation-automatic-strategy-template
       namespace: openshift-workload-availability
       kind: SelfNodeRemediationTemplate
     escalatingRemediations:
       - remediationTemplate:
           apiVersion: self-node-remediation.medik8s.io/v1alpha1
           name: self-node-remediation-resource-deletion-template
           namespace: openshift-workload-availability
           kind: SelfNodeRemediationTemplate
         order: 1
         timeout: 300s
     selector:
       matchExpressions:
         - key: node-role.kubernetes.io/worker
           operator: Exists
     unhealthyConditions:
       - type: Ready
         status: "False"
         duration: 30s
       - type: Ready
         status: Unknown
         duration: 30s

   where:

   • spec.minHealthy defines the number of worker nodes required to host VMs that migrate from failed nodes.

     • For critical environments, set this value to the minimum number of nodes that you require to maintain the cluster workload.
     • For hyperconverged storage, set this value to one less than your number of nodes to limit remediation to a single worker-node failure and maintain storage stability.
   • spec.remediationTemplate defines the remediation template to use when the Node Health Check Operator detects an unhealthy node. This example uses the Self Node Remediation Operator.
   • spec.escalatingRemediations defines escalating remediation strategies. If the initial remediation does not resolve the issue within the specified timeout, the next remediation strategy runs.
   • spec.selector defines the nodes to monitor. This example monitors all worker nodes.
   • spec.unhealthyConditions defines the parameters to identify an unhealthy node.

   • spec.unhealthyConditions.duration defines the duration that a condition must persist before remediation starts. Set lower values for faster recovery. The following table shows recommended values:

     Table 5.2. Recommended unhealthyConditions duration values

     Environment	Recommended duration
     Critical	30-60 seconds
     Standard	60-180 seconds
     Conservative	180-360 seconds

2. Apply the NodeHealthCheck CR by running the following command:

   $ oc apply -f nodehealthcheck-sample.yaml

Procedure

1. Open the HyperConverged CR by running the following command:

   $ oc edit hco kubevirt-hyperconverged -n openshift-cnv

2. Edit the spec.security.certConfig fields as shown in the following example. To avoid overloading the system, ensure that all values are greater than or equal to 10 minutes. Express all values as strings that comply with the golang ParseDuration format.

   apiVersion: hco.kubevirt.io/v1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     security:
       certConfig:
         ca:
           duration: 48h0m0s
           renewBefore: 24h0m0s
         server:
           duration: 24h0m0s
           renewBefore: 12h0m0s

   • ca.renewBefore: Set to a value less than or equal to the value of ca.duration.
   • server.duration: Set to a value less than or equal to the value of ca.duration.
   • server.renewBefore; Set to a value less than or equal to the value of server.duration.

3. Apply updates to the HyperConverged CR by running the following command:

   $ oc apply -f <filename>.yaml

   For example:

   $ oc apply -f kubevirt-hyperconverged.yaml

Procedure

1. Create the Namespace CR for KubeVirt Redfish by creating a YAML file with content such as the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: kubevirt-redfish
     labels:
       app.kubernetes.io/name: kubevirt-redfish

2. Apply the resource by running the following command:

   $ oc apply -f namespace.yaml

3. Create the ServiceAccount CR by creating a YAML file with content such as the following example:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: kubevirt-redfish
     namespace: kubevirt-redfish
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac

4. Apply the resource by running the following command:

   $ oc apply -f serviceaccount.yaml

5. Create the ClusterRole CR with required permissions by creating a YAML file with content such as the following example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: kubevirt-redfish-role
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac
   rules:
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines", "virtualmachineinstances"]
       verbs: ["get", "list", "watch", "update", "patch"]
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines/status", "virtualmachineinstances/status"]
       verbs: ["get", "list", "watch", "patch"]
     - apiGroups: ["kubevirt.io"]
       resources: ["virtualmachines/restart", "virtualmachines/start", "virtualmachines/stop"]
       verbs: ["create"]
     - apiGroups: ["subresources.kubevirt.io"]
       resources: ["virtualmachineinstances/pause", "virtualmachineinstances/unpause"]
       verbs: ["create", "update"]
     - apiGroups: [""]
       resources: ["pods", "services", "configmaps", "secrets"]
       verbs: ["get", "list", "watch", "create", "update", "delete"]
     - apiGroups: [""]
       resources: ["namespaces"]
       verbs: ["get", "list"]
     - apiGroups: ["cdi.kubevirt.io"]
       resources: ["datavolumes", "volumeimportsources"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
     - apiGroups: [""]
       resources: ["persistentvolumeclaims"]
       verbs: ["get", "list", "create", "update", "patch", "delete", "watch"]
     - apiGroups: ["storage.k8s.io"]
       resources: ["storageclasses"]
       verbs: ["get", "list"]

6. Apply the resource by running the following command:

   $ oc apply -f clusterrole.yaml

7. Create the ClusterRoleBinding CR by creating a YAML file with content such as the following example:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kubevirt-redfish-binding
     labels:
       app.kubernetes.io/name: kubevirt-redfish
       app.kubernetes.io/component: rbac
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: kubevirt-redfish-role
   subjects:
     - kind: ServiceAccount
       name: kubevirt-redfish
       namespace: kubevirt-redfish

8. Apply the resource by running the following command:

   $ oc apply -f clusterrolebinding.yaml

9. Create a Secret CR to enable the Redfish API to manage specific VMs:

   1. Generate a bcrypt hashed password for the Redfish API by running the following command in a Linux terminal:

      $ htpasswd -nbB "" '<password>' | cut -d: -f2

      • <password> is the password you want to use for the Redfish API.

        The output is a bcrypt hashed password beginning with $2a$, $2b$, or $2y$.

        Note

        In earlier Technology Preview releases, plain text passwords were used. Use bcrypt hashed passwords for improved security.

   2. Create the Secret CR containing the configuration with content such as the following example. Edit the config.yaml section to match your environment:

      apiVersion: v1
      kind: Secret
      metadata:
        name: kubevirt-redfish-secret
        namespace: kubevirt-redfish
        labels:
          app.kubernetes.io/name: kubevirt-redfish
          app.kubernetes.io/component: config
      type: Opaque
      stringData:
        config.yaml: |
          server:
            host: "0.0.0.0"
            port: 8443
            tls:
              enabled: false
          system_id_convention: "enhanced"
          chassis:
            - name: "<chassis_name>"
              namespace: "<vm_namespace>"
              service_account: "kubevirt-redfish"
              vm_selector:
                labels:
                  redfish-enabled: "true"
          authentication:
            users:
              - username: "admin"
                password:
                  hash: "<bcrypt_password>"
                chassis: ["<chassis_name>"]
          datavolume:
            storage_class: "<storage_class>"
            storage_size: "3Gi"

      where:

      system_id_convention

      Specifies the format for Redfish system IDs. The recommended setting is enhanced to use <namespace>.<vm-name> format. The legacy setting uses <vm-name> only.

      chassis

      Specifies the namespaces where VMs are deployed. Replace <chassis_name> with a name for this chassis configuration and <vm_namespace> with the namespace containing your VMs. The vm_selector labels identify which VMs in the namespace are exposed through Redfish. Only VMs with matching labels are visible. You can configure multiple chassis entries to expose different subsets of VMs in the same namespace, each with different authentication users.

      authentication

      Specifies the username and password required to validate requests to the Redfish API. Replace <bcrypt_password> with the bcrypt hashed password that you generated.

      datavolume

      Specifies storage for virtual media operations. KubeVirt Redfish uses this storage to create a data volume, download the installation ISO into that volume, and attach the volume to the VM as virtual media. Replace <storage_class> with a storage class that points to a shared storage location accessible from all nodes in the hosting cluster. Do not use a node-local storage class such as LVM Storage for this volume. For more information about storage options, see Storage requirements in "Prerequisites for virtualized control planes".

10. Apply the resource by running the following command:

    $ oc apply -f secret.yaml

    Warning

    The credentials defined in this Secret CR enable full management control over the VMs exposed through KubeVirt Redfish, independently of any OpenShift Container Platform privileges.

11. Create the Deployment CR by creating a YAML file with content such as the following example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: kubevirt-redfish
          app.kubernetes.io/component: server
      template:
        metadata:
          labels:
            app.kubernetes.io/name: kubevirt-redfish
            app.kubernetes.io/component: server
        spec:
          serviceAccountName: kubevirt-redfish
          securityContext:
            runAsNonRoot: true
          containers:
            - name: kubevirt-redfish
              image: registry.redhat.io/container-native-virtualization/kubevirt-redfish-rhel9:v5.0
              imagePullPolicy: Always
              ports:
                - name: http
                  containerPort: 8443
                  protocol: TCP
              env:
                - name: CONFIG_PATH
                  value: "/app/config/config.yaml"
                - name: LOG_LEVEL
                  value: "info"
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "100m"
                limits:
                  memory: "2Gi"
                  cpu: "500m"
              livenessProbe:
                httpGet:
                  path: /redfish/v1/
                  port: 8443
                  scheme: HTTP
                initialDelaySeconds: 30
                periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /redfish/v1/
                  port: 8443
                  scheme: HTTP
                initialDelaySeconds: 5
                periodSeconds: 5
              securityContext:
                runAsNonRoot: true
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
              volumeMounts:
                - name: config-volume
                  mountPath: /app/config
                  readOnly: true
          volumes:
            - name: config-volume
              secret:
                secretName: kubevirt-redfish-secret

    where:

    • The image field specifies the KubeVirt Redfish container image.

12. Apply the resource by running the following command:

    $ oc apply -f deployment.yaml

13. Create the Service CR by creating a YAML file with content such as the following example:

    apiVersion: v1
    kind: Service
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      type: ClusterIP
      ports:
        - name: http
          port: 8443
          targetPort: 8443
          protocol: TCP
      selector:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server

14. Apply the resource by running the following command:

    $ oc apply -f service.yaml

15. Create the Route CR to expose the Redfish API externally by creating a YAML file with content such as the following example:

    apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      name: kubevirt-redfish
      namespace: kubevirt-redfish
      labels:
        app.kubernetes.io/name: kubevirt-redfish
        app.kubernetes.io/component: server
    spec:
      port:
        targetPort: http
      to:
        kind: Service
        name: kubevirt-redfish
        weight: 100
      tls:
        termination: edge
        insecureEdgeTerminationPolicy: Redirect

16. Apply the resource by running the following command:

    $ oc apply -f route.yaml

Verification

1. Verify that the pods are running by running the following command:

   $ oc get pods -n kubevirt-redfish

   Example output

   NAME                                READY   STATUS    RESTARTS   AGE
   kubevirt-redfish-587cd94988-xthml   1/1     Running   0          2m

2. Get the route hostname by running the following command:

   $ oc get route kubevirt-redfish -n kubevirt-redfish -o jsonpath='{.spec.host}'

3. Test the Redfish endpoint by running the following command:

   $ curl -sk -u "admin:<password>" https://<route_hostname>/redfish/v1/

   A successful response returns JSON with the Redfish service root:

   {
     "@odata.id": "/redfish/v1",
     "@odata.type": "#ServiceRoot.v1_0_0.ServiceRoot",
     "Id": "RootService",
     "Name": "Root Service",
     "Systems": {
       "@odata.id": "/redfish/v1/Systems"
     }
   }