Chapitre 15. Managing GPU devices in virtual machines
To enhance the graphical performance of your virtual machine (VMs) on a RHEL 9 host, you can assign a host GPU to a VM.
- You can detach the GPU from the host and pass full control of the GPU directly to the VM.
- You can create multiple mediated devices from a physical GPU, and assign these devices as virtual GPUs (vGPUs) to multiple guests. This is currently only supported on selected NVIDIA GPUs, and only one mediated device can be assigned to a single guest.
15.1. Assigning a GPU to a virtual machine
To access and control GPUs that are attached to the host system, you must configure the host system to pass direct control of the GPU to the virtual machine (VM).
If you are looking for information about assigning a virtual GPU, see Managing NVIDIA vGPU devices.
Conditions préalables
You must enable IOMMU support on the host machine kernel.
On an Intel host, you must enable VT-d:
Regenerate the GRUB configuration with the
intel_iommu=onandiommu=ptparameters:# grubby --args="intel_iommu=on iommu_pt" --update-kernel DEFAULT- Reboot the host.
On an AMD host, you must enable AMD-Vi.
Note that on AMD hosts, IOMMU is enabled by default, you can add
iommu=ptto switch it to pass-through mode:Regenerate the GRUB configuration with the
iommu=ptparameter:# grubby --args="iommu=pt" --update-kernel DEFAULTNoteThe
ptoption only enables IOMMU for devices used in pass-through mode and provides better host performance. However, not all hardware supports the option. You can still assign devices irrespective of whether this option is enabled.- Reboot the host.
Procédure
Prevent the driver from binding to the GPU.
Identify the PCI bus address to which the GPU is attached.
# lspci -Dnn | grep VGA 0000:02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106GL [Quadro K4000] [10de:11fa] (rev a1)
Prevent the host’s graphics driver from using the GPU. To do so, use the GPU’s PCI ID with the pci-stub driver.
For example, the following command prevents the driver from binding to the GPU attached at the 10de:11fa bus:
# grubby --args="pci-stub.ids=10de:11fa" --update-kernel DEFAULT- Reboot the host.
Optional: If certain GPU functions, such as audio, cannot be passed through to the VM due to support limitations, you can modify the driver bindings of the endpoints within an IOMMU group to pass through only the necessary GPU functions.
Convert the GPU settings to XML and note the PCI address of the endpoints that you want to prevent from attaching to the host drivers.
To do so, convert the GPU’s PCI bus address to a libvirt-compatible format by adding the
pci_prefix to the address, and converting the delimiters to underscores.For example, the following command displays the XML configuration of the GPU attached at the
0000:02:00.0bus address.# virsh nodedev-dumpxml pci_0000_02_00_0<device> <name>pci_0000_02_00_0</name> <path>/sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0</path> <parent>pci_0000_00_03_0</parent> <driver> <name>pci-stub</name> </driver> <capability type='pci'> <domain>0</domain> <bus>2</bus> <slot>0</slot> <function>0</function> <product id='0x11fa'>GK106GL [Quadro K4000]</product> <vendor id='0x10de'>NVIDIA Corporation</vendor> <iommuGroup number='13'> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='8' width='16'/> <link validity='sta' speed='2.5' width='16'/> </pci-express> </capability> </device>
Prevent the endpoints from attaching to the host driver.
In this example, to assign the GPU to a VM, prevent the endpoints that correspond to the audio function,
<address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>, from attaching to the host audio driver, and instead attach the endpoints to VFIO-PCI.# driverctl set-override 0000:02:00.1 vfio-pci
Attach the GPU to the VM
Create an XML configuration file for the GPU by using the PCI bus address.
For example, you can create the following XML file, GPU-Assign.xml, by using parameters from the GPU’s bus address.
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> </hostdev>
- Save the file on the host system.
Merge the file with the VM’s XML configuration.
For example, the following command merges the GPU XML file, GPU-Assign.xml, with the XML configuration file of the
System1VM.# virsh attach-device System1 --file /home/GPU-Assign.xml --persistent Device attached successfully.NoteThe GPU is attached as a secondary graphics device to the VM. Assigning a GPU as the primary graphics device is not supported, and Red Hat does not recommend removing the primary emulated graphics device in the VM’s XML configuration.
Vérification
-
The device appears under the
<devices>section in VM’s XML configuration. For more information, see Sample virtual machine XML configuration.
Problèmes connus
The number of GPUs that can be attached to a VM is limited by the maximum number of assigned PCI devices, which in RHEL 9 is currently 64. However, attaching multiple GPUs to a VM is likely to cause problems with memory-mapped I/O (MMIO) on the guest, which may result in the GPUs not being available to the VM.
To work around these problems, set a larger 64-bit MMIO space and configure the vCPU physical address bits to make the extended 64-bit MMIO space addressable.
- Attaching an NVIDIA GPU device to a VM that uses a RHEL 9 guest operating system currently disables the Wayland session on that VM, and loads an Xorg session instead. This is because of incompatibilities between NVIDIA drivers and Wayland.
