Chapter 7. Virtual machines

Procedure

1. Navigate to Virtualization Templates in the web console.
2. From the list of VM templates, click the template marked as deprecated.
3. Click Edit next to the pencil icon beside Labels.

4. Remove the following two labels:

   • template.kubevirt.io/type: "base"
   • template.kubevirt.io/version: "version"
5. Click Save.
6. Click the pencil icon beside the number of existing Annotations.

7. Remove the following annotation:

   • template.kubevirt.io/deprecated
8. Click Save.

Procedure

1. Stop the RHEL VM if it is running, by entering the following command:

   $ virtctl stop <my_vm_name>

2. Optional: Clone the virtual machine to avoid losing the data from your original VM. You can then generalize the cloned VM.

3. Retrieve the dataVolume that stores the root filesystem for the VM by running the following command:

   $ oc get vm <my_vm_name> -o jsonpath="{.spec.template.spec.volumes}{'\n'}"

   Example output

   [{"dataVolume":{"name":"<my_vm_volume>"},"name":"rootdisk"},{"cloudInitNoCloud":{...}]

4. Retrieve the persistent volume claim (PVC) that matches the listed dataVolume by running the followimg command:

   $ oc get pvc

   Example output

   NAME            STATUS   VOLUME  CAPACITY   ACCESS MODES  STORAGECLASS     AGE
   <my_vm_volume> Bound  …

   Note

   If your cluster configuration does not enable you to clone a VM, to avoid losing the data from your original VM, you can clone the VM PVC to a data volume instead. You can then use the cloned PVC to create a golden image.

   If you are creating a golden image by cloning a PVC, continue with the next steps, using the cloned PVC.

5. Deploy a new interactive container with libguestfs-tools and attach the PVC to it by running the following command:

   $ virtctl guestfs <my-vm-volume> --uid 107

   This command opens a shell for you to run the next command.

6. Remove all configurations specific to your system by running the following command:

   $ virt-sysprep -a disk.img

7. In the OpenShift Container Platform console, click Virtualization Catalog.
8. Click Add volume.

9. In the Add volume window:

   1. From the Source type list, select Use existing Volume.
   2. From the Volume project list, select your project.
   3. From the Volume name list, select the correct PVC.
   4. In the Volume name field, enter a name for the new golden image.
   5. From the Preference list, select the RHEL version you are using.
   6. From the Default Instance Type list, select the instance type with the correct CPU and memory requirements for the version of RHEL you selected previously.
   7. Click Save.

Procedure

1. In the OpenShift Container Platform console, click Virtualization VirtualMachines.
2. Select a Windows VM to open the VirtualMachine details page.
3. Click Configuration Disks.
4. Click the Options menu beside the sysprep disk and select Detach.
5. Click Detach.
6. Rename C:\Windows\Panther\unattend.xml to avoid detection by the sysprep tool.

7. Start the sysprep program by running the following command:

   %WINDIR%\System32\Sysprep\sysprep.exe /generalize /shutdown /oobe /mode:vm

8. After the sysprep tool completes, the Windows VM shuts down. The disk image of the VM is now available to use as an installation image for Windows VMs.

Procedure

1. In the OpenShift Container Platform console, click Virtualization Catalog.
2. Select a Windows template and click Customize VirtualMachine.
3. Select PVC (clone PVC) from the Disk source list.
4. Select the PVC project and PVC name of the generalized Windows image.
5. Click Customize VirtualMachine parameters.
6. Click the Scripts tab.
7. In the Sysprep section, click Edit, browse to the unattend.xml answer file, and click Save.
8. Click Create VirtualMachine.

Procedure

1. Log in to the VM by using a console or SSH.

2. Install the QEMU guest agent by running the following command:

   $ yum install -y qemu-guest-agent

3. Ensure the service is persistent and start it:

   $ systemctl enable --now qemu-guest-agent

Procedure

1. In the Windows guest operating system, use the File Explorer to navigate to the guest-agent directory in the virtio-win CD drive.
2. Run the qemu-ga-x86_64.msi installer.

Verification

1. Obtain a list of network services by running the following command:

   $ net start

2. Verify that the output contains the QEMU Guest Agent.

Procedure

1. When creating a Windows VM from a template, click Customize VirtualMachine.
2. Select Mount Windows drivers disk.
3. Click the Customize VirtualMachine parameters.
4. Click Create VirtualMachine.

Procedure

1. Navigate to the existing Windows VM, and click Actions Stop.
2. Go to VM Details Configuration Disks and click Add disk.
3. Add windows-driver-disk from container source, set the Type to CD-ROM, and then set the Interface to SATA.
4. Click Save.
5. Start the VM, and connect to a graphical console.

Procedure

1. In the Windows operating system, use the File Explorer to navigate to the virtio-win CD drive.

2. Double-click the drive to run the appropriate installer for your VM.

   For a 64-bit vCPU, select the virtio-win-gt-x64 installer. 32-bit vCPUs are no longer supported.

3. Optional: During the Custom Setup step of the installer, select the device drivers you want to install. The recommended driver set is selected by default.
4. After the installation is complete, select Finish.
5. Reboot the VM.

Verification

1. Open the system disk on the PC. This is typically C:.
2. Navigate to Program Files Virtio-Win.

Procedure

1. Start the VM and connect to a graphical console.
2. Log in to a Windows user session.

3. Open Device Manager and expand Other devices to list any Unknown device.

   1. Open the Device Properties to identify the unknown device.
   2. Right-click the device and select Properties.
   3. Click the Details tab and select Hardware Ids in the Property list.
   4. Compare the Value for the Hardware Ids with the supported VirtIO drivers.
4. Right-click the device and select Update Driver Software.
5. Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
6. Click Next to install the driver.
7. Repeat this process for all the necessary VirtIO drivers.
8. After the driver installs, click Close to close the window.
9. Reboot the VM to complete the driver installation.

Procedure

1. Add the container-native-virtualization/virtio-win container disk as a CD drive by editing the VirtualMachine manifest:

   # ...
   spec:
     domain:
       devices:
         disks:
           - name: virtiocontainerdisk
             bootOrder: 2 1
             cdrom:
               bus: sata
   volumes:
     - containerDisk:
         image: container-native-virtualization/virtio-win
       name: virtiocontainerdisk

   1

   OpenShift Virtualization boots the VM disks in the order defined in the VirtualMachine manifest. You can either define other VM disks that boot before the container-native-virtualization/virtio-win container disk or use the optional bootOrder parameter to ensure the VM boots from the correct disk. If you configure the boot order for a disk, you must configure the boot order for the other disks.

2. Apply the changes:

   • If the VM is not running, run the following command:

     $ virtctl start <vm> -n <namespace>

   • If the VM is running, reboot the VM or run the following command:

     $ oc apply -f <vm.yaml>

3. After the VM has started, install the VirtIO drivers from the SATA CD drive.

Procedure

1. In the Windows Guest operating system, click the Windows key and select Settings.
2. Navigate to Windows Update Advanced Options Optional Updates.
3. Install all updates from Red Hat, Inc..
4. Reboot the VM.

Verification

1. On the Windows VM, navigate to the Device Manager.
2. Select a device.
3. Select the Driver tab.
4. Click Driver Details and confirm that the virtio driver details displays the correct version.

Procedure

1. Create a DataVolume manifest as shown in the following example:

   apiVersion: cdi.kubevirt.io/v1beta1
   kind: DataVolume
   metadata:
     name: <datavolume> 1
   spec:
     source:
       pvc:
         namespace: "<source_namespace>" 2
         name: "<my_vm_disk>" 3
     storage: {}

   1

   Specify the name of the new data volume.

   2

   Specify the namespace of the source PVC.

   3

   Specify the name of the source PVC.

2. Create the data volume by running the following command:

   $ oc create -f <datavolume>.yaml

   Note

   Data volumes prevent a VM from starting before the PVC is prepared. You can create a VM that references the new data volume while the PVC is being cloned.

Procedure

1. Create a VirtualMachine manifest as shown in the following example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm-dv-clone
     name: vm-dv-clone 1
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm-dv-clone
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: root-disk
           resources:
             requests:
               memory: 64M
         volumes:
         - dataVolume:
             name: favorite-clone
           name: root-disk
     dataVolumeTemplates:
     - metadata:
         name: favorite-clone
       spec:
         storage:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 2Gi
         source:
           pvc:
             namespace: <source_namespace> 2
             name: "<source_pvc>" 3

   1

   Specify the name of the VM.

   2

   Specify the namespace of the source PVC.

   3

   Specify the name of the source PVC.

2. Create the virtual machine with the PVC-cloned data volume:

   $ oc create -f <vm-clone-datavolumetemplate>.yaml

Procedure

1. Navigate to Virtualization Catalog in the web console.

2. Click a template tile.

   The guest operating system must support configuration from a cloud-init data source.

3. Click Customize VirtualMachine.
4. Click Next.
5. Click the Scripts tab.

6. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key and select one of the following options:

   • Use existing: Select a secret from the secrets list.

   • Add new:

     1. Browse to the SSH key file or paste the file in the key field.
     2. Enter the secret name.
     3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
7. Click Save.

8. Click Create VirtualMachine.

   The VirtualMachine details page displays the progress of the VM creation.

Procedure

1. In the web console, navigate to Virtualization Catalog.

   The InstanceTypes tab opens by default.

2. Select either of the following options:

   • Select a suitable bootable volume from the list. If the list is truncated, click the Show all button to display the entire list.

     Note

     The bootable volume table lists only those volumes in the openshift-virtualization-os-images namespace that have the instancetype.kubevirt.io/default-preference label.

     • Optional: Click the star icon to designate a bootable volume as a favorite. Starred bootable volumes appear first in the volume list.

   • Click Add volume to upload a new volume or to use an existing persistent volume claim (PVC), a volume snapshot, or a containerDisk volume. Click Save.

     Logos of operating systems that are not available in the cluster are shown at the bottom of the list. You can add a volume for the required operating system by clicking the Add volume link.

     In addition, there is a link to the Create a Windows boot source quick start. The same link appears in a popover if you hover the pointer over the question mark icon next to the Select volume to boot from line.

     Immediately after you install the environment or when the environment is disconnected, the list of volumes to boot from is empty. In that case, three operating system logos are displayed: Windows, RHEL, and Linux. You can add a new volume that meets your requirements by clicking the Add volume button.

3. Click an instance type tile and select the resource size appropriate for your workload.

4. Optional: Choose the virtual machine details, including the VM’s name, that apply to the volume you are booting from:

   • For a Linux-based volume, follow these steps to configure SSH:

     1. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key in the VirtualMachine details section.

     2. Select one of the following options:

        • Use existing: Select a secret from the secrets list.

        • Add new: Follow these steps:

          1. Browse to the public SSH key file or paste the file in the key field.
          2. Enter the secret name.
          3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
     3. Click Save.

   • For a Windows volume, follow either of these set of steps to configure sysprep options:

     • If you have not already added sysprep options for the Windows volume, follow these steps:

       1. Click the edit icon beside Sysprep in the VirtualMachine details section.
       2. Add the Autoattend.xml answer file.
       3. Add the Unattend.xml answer file.
       4. Click Save.

     • If you want to use existing sysprep options for the Windows volume, follow these steps:

       1. Click Attach existing sysprep.
       2. Enter the name of the existing sysprep Unattend.xml answer file.
       3. Click Save.

5. Optional: If you are creating a Windows VM, you can mount a Windows driver disk:

   1. Click the Customize VirtualMachine button.
   2. On the VirtualMachine details page, click Storage.
   3. Select the Mount Windows drivers disk checkbox.
6. Optional: Click View YAML & CLI to view the YAML file. Click CLI to view the CLI commands. You can also download or copy either the YAML file contents or the CLI commands.
7. Click Create VirtualMachine.

Procedure

1. Create a manifest file for a VirtualMachine object and a Secret object:

   Example manifest

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
     namespace: example-namespace
   spec:
     dataVolumeTemplates:
       - metadata:
           name: example-vm-volume
         spec:
           sourceRef:
             kind: DataSource
             name: rhel9
             namespace: openshift-virtualization-os-images
           storage:
             resources: {}
     instancetype:
       name: u1.medium
     preference:
       name: rhel.9
     running: true
     template:
       spec:
         domain:
           devices: {}
         volumes:
           - dataVolume:
               name: example-vm-volume
             name: rootdisk
           - cloudInitNoCloud: 1
               userData: |-
                 #cloud-config
                 user: cloud-user
             name: cloudinitdisk
         accessCredentials:
           - sshPublicKey:
               propagationMethod:
                 noCloud: {}
               source:
                 secret:
                   secretName: authorized-keys 2
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: authorized-keys
   data:
     key: c3NoLXJzYSB... 3

   1

   Specify the cloudInitNoCloud data source.

   2

   Specify the Secret object name.

   3

   Paste the public SSH key.

2. Create the VirtualMachine and Secret objects by running the following command:

   $ oc create -f <manifest_file>.yaml

3. Start the VM by running the following command:

   $ virtctl start vm example-vm -n example-namespace

Procedure

1. Navigate to Virtualization Catalog in the web console.
2. Click the Red Hat Enterprise Linux 9 VM tile.
3. Click Customize VirtualMachine.
4. Click Next.
5. Click the Scripts tab.

6. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key and select one of the following options:

   • Use existing: Select a secret from the secrets list.

   • Add new:

     1. Browse to the SSH key file or paste the file in the key field.
     2. Enter the secret name.
     3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
7. Set Dynamic SSH key injection to on.
8. Click Save.

9. Click Create VirtualMachine.

   The VirtualMachine details page displays the progress of the VM creation.

Procedure

1. In the web console, navigate to Virtualization Catalog.

   The InstanceTypes tab opens by default.

2. Select either of the following options:

   • Select a suitable bootable volume from the list. If the list is truncated, click the Show all button to display the entire list.

     Note

     The bootable volume table lists only those volumes in the openshift-virtualization-os-images namespace that have the instancetype.kubevirt.io/default-preference label.

     • Optional: Click the star icon to designate a bootable volume as a favorite. Starred bootable volumes appear first in the volume list.

   • Click Add volume to upload a new volume or to use an existing persistent volume claim (PVC), a volume snapshot, or a containerDisk volume. Click Save.

     Logos of operating systems that are not available in the cluster are shown at the bottom of the list. You can add a volume for the required operating system by clicking the Add volume link.

     In addition, there is a link to the Create a Windows boot source quick start. The same link appears in a popover if you hover the pointer over the question mark icon next to the Select volume to boot from line.

     Immediately after you install the environment or when the environment is disconnected, the list of volumes to boot from is empty. In that case, three operating system logos are displayed: Windows, RHEL, and Linux. You can add a new volume that meets your requirements by clicking the Add volume button.

3. Click an instance type tile and select the resource size appropriate for your workload.
4. Click the Red Hat Enterprise Linux 9 VM tile.

5. Optional: Choose the virtual machine details, including the VM’s name, that apply to the volume you are booting from:

   • For a Linux-based volume, follow these steps to configure SSH:

     1. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key in the VirtualMachine details section.

     2. Select one of the following options:

        • Use existing: Select a secret from the secrets list.

        • Add new: Follow these steps:

          1. Browse to the public SSH key file or paste the file in the key field.
          2. Enter the secret name.
          3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
     3. Click Save.

   • For a Windows volume, follow either of these set of steps to configure sysprep options:

     • If you have not already added sysprep options for the Windows volume, follow these steps:

       1. Click the edit icon beside Sysprep in the VirtualMachine details section.
       2. Add the Autoattend.xml answer file.
       3. Add the Unattend.xml answer file.
       4. Click Save.

     • If you want to use existing sysprep options for the Windows volume, follow these steps:

       1. Click Attach existing sysprep.
       2. Enter the name of the existing sysprep Unattend.xml answer file.
       3. Click Save.
6. Set Dynamic SSH key injection in the VirtualMachine details section to on.

7. Optional: If you are creating a Windows VM, you can mount a Windows driver disk:

   1. Click the Customize VirtualMachine button.
   2. On the VirtualMachine details page, click Storage.
   3. Select the Mount Windows drivers disk checkbox.
8. Optional: Click View YAML & CLI to view the YAML file. Click CLI to view the CLI commands. You can also download or copy either the YAML file contents or the CLI commands.
9. Click Create VirtualMachine.

Procedure

1. Navigate to Virtualization VirtualMachines in the web console.
2. Select a VM to open the VirtualMachine details page.
3. On the Configuration tab, click Scripts.

4. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key and select one of the following options:

   • Use existing: Select a secret from the secrets list.

   • Add new:

     1. Browse to the SSH key file or paste the file in the key field.
     2. Enter the secret name.
     3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
5. Set Dynamic SSH key injection to on.
6. Click Save.

Procedure

1. Create a manifest file for a VirtualMachine object and a Secret object:

   Example manifest

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
     namespace: example-namespace
   spec:
     dataVolumeTemplates:
       - metadata:
           name: example-vm-volume
         spec:
           sourceRef:
             kind: DataSource
             name: rhel9
             namespace: openshift-virtualization-os-images
           storage:
             resources: {}
     instancetype:
       name: u1.medium
     preference:
       name: rhel.9
     running: true
     template:
       spec:
         domain:
           devices: {}
         volumes:
           - dataVolume:
               name: example-vm-volume
             name: rootdisk
           - cloudInitNoCloud: 1
               userData: |-
                 #cloud-config
                 runcmd:
                 - [ setsebool, -P, virt_qemu_ga_manage_ssh, on ]
             name: cloudinitdisk
         accessCredentials:
           - sshPublicKey:
               propagationMethod:
                 qemuGuestAgent:
                   users: ["cloud-user"]
               source:
                 secret:
                   secretName: authorized-keys 2
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: authorized-keys
   data:
     key: c3NoLXJzYSB... 3

   1

   Specify the cloudInitNoCloud data source.

   2

   Specify the Secret object name.

   3

   Paste the public SSH key.

2. Create the VirtualMachine and Secret objects by running the following command:

   $ oc create -f <manifest_file>.yaml

3. Start the VM by running the following command:

   $ virtctl start vm example-vm -n example-namespace

Procedure

1. Navigate to Virtualization Overview.
2. On the Settings tab, click Cluster.
3. Expand General settings and SSH configuration.
4. Set SSH over LoadBalancer service to on.

Procedure

1. Navigate to VirtualMachines and select a virtual machine to view the VirtualMachine details page.
2. On the Details tab, select SSH over LoadBalancer from the SSH service type list.
3. Optional: Click the copy icon to copy the SSH command to your clipboard.

Procedure

1. Edit the VirtualMachine manifest to add the label for service creation:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm
     namespace: example-namespace
   spec:
     running: false
     template:
       metadata:
         labels:
           special: key 1
   # ...

   1

   Add special: key to the spec.template.metadata.labels stanza.

   Note

   Labels on a virtual machine are passed through to the pod. The special: key label must match the label in the spec.selector attribute of the Service manifest.

2. Save the VirtualMachine manifest file to apply your changes.

3. Create a Service manifest to expose the VM:

   apiVersion: v1
   kind: Service
   metadata:
     name: example-service
     namespace: example-namespace
   spec:
   # ...
     selector:
       special: key 1
     type: NodePort 2
     ports: 3
       protocol: TCP
       port: 80
       targetPort: 9376
       nodePort: 30000

   1

   Specify the label that you added to the spec.template.metadata.labels stanza of the VirtualMachine manifest.

   2

   Specify ClusterIP, NodePort, or LoadBalancer.

   3

   Specifies a collection of network ports and protocols that you want to expose from the virtual machine.

4. Save the Service manifest file.

5. Create the service by running the following command:

   $ oc create -f example-service.yaml

6. Restart the VM to apply the changes.

Verification

1. Verify that the label was added to the node by running the following command:

   $ oc describe node <node_name>

2. Optional: If GPU operands were previously deployed on the node, verify their removal.

   1. Check the status of the pods in the nvidia-gpu-operator namespace by running the following command:

      $ oc get pods -n nvidia-gpu-operator

      Example output

      NAME                             READY   STATUS        RESTARTS   AGE
      gpu-operator-59469b8c5c-hw9wj    1/1     Running       0          8d
      nvidia-sandbox-validator-7hx98   1/1     Running       0          8d
      nvidia-sandbox-validator-hdb7p   1/1     Running       0          8d
      nvidia-sandbox-validator-kxwj7   1/1     Terminating   0          9d
      nvidia-vfio-manager-7w9fs        1/1     Running       0          8d
      nvidia-vfio-manager-866pz        1/1     Running       0          8d
      nvidia-vfio-manager-zqtck        1/1     Terminating   0          9d

   2. Monitor the pod status until the pods with Terminating status are removed:

      $ oc get pods -n nvidia-gpu-operator

      Example output

      NAME                             READY   STATUS    RESTARTS   AGE
      gpu-operator-59469b8c5c-hw9wj    1/1     Running   0          8d
      nvidia-sandbox-validator-7hx98   1/1     Running   0          8d
      nvidia-sandbox-validator-hdb7p   1/1     Running   0          8d
      nvidia-vfio-manager-7w9fs        1/1     Running   0          8d
      nvidia-vfio-manager-866pz        1/1     Running   0          8d

Procedure

1. Create a MachineConfig object that identifies the kernel argument. The following example shows a kernel argument for an Intel CPU.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 1
     name: 100-worker-iommu 2
   spec:
     config:
       ignition:
         version: 3.2.0
     kernelArguments:
         - intel_iommu=on 3
   # ...

   1

   Applies the new kernel argument only to worker nodes.

   2

   The name indicates the ranking of this kernel argument (100) among the machine configs and its purpose. If you have an AMD CPU, specify the kernel argument as amd_iommu=on.

   3

   Identifies the kernel argument as intel_iommu for an Intel CPU.

2. Create the new MachineConfig object:

   $ oc create -f 100-worker-kernel-arg-iommu.yaml

Procedure

1. Run the lspci command to obtain the vendor-ID and the device-ID for the PCI device.

   $ lspci -nnv | grep -i nvidia

   Example output

   02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)

2. Create a Butane config file, 100-worker-vfiopci.bu, binding the PCI device to the VFIO driver.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Example

   variant: openshift
   version: 4.17.0
   metadata:
     name: 100-worker-vfiopci
     labels:
       machineconfiguration.openshift.io/role: worker 1
   storage:
     files:
     - path: /etc/modprobe.d/vfio.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           options vfio-pci ids=10de:1eb8 2
     - path: /etc/modules-load.d/vfio-pci.conf 3
       mode: 0644
       overwrite: true
       contents:
         inline: vfio-pci

   1

   Applies the new kernel argument only to worker nodes.

   2

   Specify the previously determined vendor-ID value (10de) and the device-ID value (1eb8) to bind a single device to the VFIO driver. You can add a list of multiple devices with their vendor and device information.

   3

   The file that loads the vfio-pci kernel module on the worker nodes.

3. Use Butane to generate a MachineConfig object file, 100-worker-vfiopci.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 100-worker-vfiopci.bu -o 100-worker-vfiopci.yaml

4. Apply the MachineConfig object to the worker nodes:

   $ oc apply -f 100-worker-vfiopci.yaml

5. Verify that the MachineConfig object was added.

   $ oc get MachineConfig

   Example output

   NAME                             GENERATEDBYCONTROLLER                      IGNITIONVERSION  AGE
   00-master                        d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   00-worker                        d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   01-master-container-runtime      d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   01-master-kubelet                d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   01-worker-container-runtime      d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   01-worker-kubelet                d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
   100-worker-iommu                                                            3.2.0            30s
   100-worker-vfiopci-configuration                                            3.2.0            30s

Procedure

1. Edit the HyperConverged CR in your default editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Add the PCI device information to the spec.permittedHostDevices.pciHostDevices array. For example:

   Example configuration file

   apiVersion: hco.kubevirt.io/v1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     permittedHostDevices: 1
       pciHostDevices: 2
       - pciDeviceSelector: "10DE:1DB6" 3
         resourceName: "nvidia.com/GV100GL_Tesla_V100" 4
       - pciDeviceSelector: "10DE:1EB8"
         resourceName: "nvidia.com/TU104GL_Tesla_T4"
       - pciDeviceSelector: "8086:6F54"
         resourceName: "intel.com/qat"
         externalResourceProvider: true 5
   # ...

   1

   The host devices that are permitted to be used in the cluster.

   2

   The list of PCI devices available on the node.

   3

   The vendor-ID and the device-ID required to identify the PCI device.

   4

   The name of a PCI host device.

   5

   Optional: Setting this field to true indicates that the resource is provided by an external device plugin. OpenShift Virtualization allows the usage of this device in the cluster but leaves the allocation and monitoring to an external device plugin.

   Note

   The above example snippet shows two PCI host devices that are named nvidia.com/GV100GL_Tesla_V100 and nvidia.com/TU104GL_Tesla_T4 added to the list of permitted host devices in the HyperConverged CR. These devices have been tested and verified to work with OpenShift Virtualization.

3. Save your changes and exit the editor.

Procedure

1. Edit the HyperConverged CR in your default editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

2. Remove the PCI device information from the spec.permittedHostDevices.pciHostDevices array by deleting the pciDeviceSelector, resourceName and externalResourceProvider (if applicable) fields for the appropriate device. In this example, the intel.com/qat resource has been deleted.

   Example configuration file

   apiVersion: hco.kubevirt.io/v1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     permittedHostDevices:
       pciHostDevices:
       - pciDeviceSelector: "10DE:1DB6"
         resourceName: "nvidia.com/GV100GL_Tesla_V100"
       - pciDeviceSelector: "10DE:1EB8"
         resourceName: "nvidia.com/TU104GL_Tesla_T4"
   # ...

3. Save your changes and exit the editor.

Procedure

1. Create a MachineConfig object that identifies the kernel argument. The following example shows a kernel argument for an Intel CPU.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker 1
     name: 100-worker-iommu 2
   spec:
     config:
       ignition:
         version: 3.2.0
     kernelArguments:
         - intel_iommu=on 3
   # ...

   1

   Applies the new kernel argument only to worker nodes.

   2

   The name indicates the ranking of this kernel argument (100) among the machine configs and its purpose. If you have an AMD CPU, specify the kernel argument as amd_iommu=on.

   3

   Identifies the kernel argument as intel_iommu for an Intel CPU.

2. Create the new MachineConfig object:

   $ oc create -f 100-worker-kernel-arg-iommu.yaml

Procedure

1. Open the HyperConverged CR in your default editor by running the following command:

   $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv

   Example 7.1. Example configuration file with mediated devices configured

   apiVersion: hco.kubevirt.io/v1
   kind: HyperConverged
   metadata:
     name: kubevirt-hyperconverged
     namespace: openshift-cnv
   spec:
     mediatedDevicesConfiguration:
       mediatedDeviceTypes:
       - nvidia-231
       nodeMediatedDeviceTypes:
       - mediatedDeviceTypes:
         - nvidia-233
         nodeSelector:
           kubernetes.io/hostname: node-11.redhat.com
     permittedHostDevices:
       mediatedDevices:
       - mdevNameSelector: GRID T4-2Q
         resourceName: nvidia.com/GRID_T4-2Q
       - mdevNameSelector: GRID T4-8Q
         resourceName: nvidia.com/GRID_T4-8Q
   # ...

2. Create mediated devices by adding them to the spec.mediatedDevicesConfiguration stanza:

   Example YAML snippet

   # ...
   spec:
     mediatedDevicesConfiguration:
       mediatedDeviceTypes: 1
       - <device_type>
       nodeMediatedDeviceTypes: 2
       - mediatedDeviceTypes: 3
         - <device_type>
         nodeSelector: 4
           <node_selector_key>: <node_selector_value>
   # ...

   1

   Required: Configures global settings for the cluster.

   2

   Optional: Overrides the global configuration for a specific node or group of nodes. Must be used with the global mediatedDeviceTypes configuration.

   3

   Required if you use nodeMediatedDeviceTypes. Overrides the global mediatedDeviceTypes configuration for the specified nodes.

   4

   Required if you use nodeMediatedDeviceTypes. Must include a key:value pair.

   Important

   Before OpenShift Virtualization 4.14, the mediatedDeviceTypes field was named mediatedDevicesTypes. Ensure that you use the correct field name when configuring mediated devices.

3. Identify the name selector and resource name values for the devices that you want to expose to the cluster. You will add these values to the HyperConverged CR in the next step.

   1. Find the resourceName value by running the following command:

      $ oc get $NODE -o json \
        | jq '.status.allocatable \
          | with_entries(select(.key | startswith("nvidia.com/"))) \
          | with_entries(select(.value != "0"))'

   2. Find the mdevNameSelector value by viewing the contents of /sys/bus/pci/devices/<slot>:<bus>:<domain>.<function>/mdev_supported_types/<type>/name, substituting the correct values for your system.

      For example, the name file for the nvidia-231 type contains the selector string GRID T4-2Q. Using GRID T4-2Q as the mdevNameSelector value allows nodes to use the nvidia-231 type.

4. Expose the mediated devices to the cluster by adding the mdevNameSelector and resourceName values to the spec.permittedHostDevices.mediatedDevices stanza of the HyperConverged CR:

   Example YAML snippet

   # ...
     permittedHostDevices:
       mediatedDevices:
       - mdevNameSelector: GRID T4-2Q 1
         resourceName: nvidia.com/GRID_T4-2Q 2
   # ...

   1

   Exposes the mediated devices that map to this value on the host.

   2

   Matches the resource name that is allocated on the node.

5. Save your changes and exit the editor.