16.7. Assigning GPU Devices

Procedure 16.13. Enable IOMMU support in the host machine kernel

1. Edit the kernel command line

   For an Intel VT-d system, IOMMU is activated by adding the intel_iommu=on and iommu=pt parameters to the kernel command line. For an AMD-Vi system, the option needed is only iommu=pt. To enable this option, edit or add the GRUB_CMDLINX_LINUX line to the /etc/sysconfig/grub configuration file as follows:

   GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01
   vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root
   vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ]  &&
   /usr/sbin/rhcrashkernel-param || :) rhgb quiet intel_iommu=on iommu=pt"
   

   Note

   For further information on IOMMU, see Appendix E, Working with IOMMU Groups.

2. Regenerate the boot loader configuration

   For the changes to the kernel command line to apply, regenerate the boot loader configuration using the grub2-mkconfig command:

   # grub2-mkconfig -o /etc/grub2.cfg

   Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.

3. Reboot the host

   For the changes to take effect, reboot the host machine:

   # reboot

Procedure 16.14. Excluding the GPU device from binding to the host physical machine driver

1. Identify the PCI bus address

   To identify the PCI bus address and IDs of the device, run the following lspci command. In this example, a VGA controller such as an NVIDIA Quadro or GRID card is used:

   # lspci -Dnn | grep VGA
   0000:02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106GL [Quadro K4000] [10de:11fa] (rev a1)
   

   The resulting search reveals that the PCI bus address of this device is 0000:02:00.0 and the PCI IDs for the device are 10de:11fa.

2. Prevent the native host machine driver from using the GPU device

   To prevent the native host machine driver from using the GPU device, you can use a PCI ID with the pci-stub driver. To do this, append the pci-stub.ids option, with the PCI IDs as its value, to the GRUB_CMDLINX_LINUX line located in the /etc/sysconfig/grub configuration file, for example as follows:

   GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01
   vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root
   vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ]  &&
   /usr/sbin/rhcrashkernel-param || :) rhgb quiet intel_iommu=on iommu=pt pci-stub.ids=10de:11fa"
   

   To add additional PCI IDs for pci-stub, separate them with a comma.

3. Regenerate the boot loader configuration

   Regenerate the boot loader configuration using the grub2-mkconfig to include this option:

   # grub2-mkconfig -o /etc/grub2.cfg

   Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.

4. Reboot the host machine

   In order for the changes to take effect, reboot the host machine:

   # reboot

Procedure 16.15. Optional: Editing the GPU IOMMU configuration

1. Display the XML information of the GPU

   To display the settings of the GPU in XML form, you first need to convert its PCI bus address to libvirt-compatible format by appending pci_ and converting delimiters to underscores. In this example, the GPU PCI device identified with the 0000:02:00.0 bus address (as obtained in the previous procedure) becomes pci_0000_02_00_0. Use the libvirt address of the device with the virsh nodedev-dumpxml to display its XML configuration:

   # virsh nodedev-dumpxml pci_0000_02_00_0

   
   <device>
    <name>pci_0000_02_00_0</name>
    <path>/sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0</path>
    <parent>pci_0000_00_03_0</parent>
    <driver>
     <name>pci-stub</name>
    </driver>
    <capability type='pci'>
     <domain>0</domain>
     <bus>2</bus>
     <slot>0</slot>
     <function>0</function>
     <product id='0x11fa'>GK106GL [Quadro K4000]</product>
     <vendor id='0x10de'>NVIDIA Corporation</vendor>
        <!-- pay attention to the following lines -->
     <iommuGroup number='13'>
      <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
     </iommuGroup>
     <pci-express>
      <link validity='cap' port='0' speed='8' width='16'/>
      <link validity='sta' speed='2.5' width='16'/>
     </pci-express>
    </capability>
   </device>

   Note the <iommuGroup> element of the XML. The iommuGroup indicates a set of devices that are considered isolated from other devices due to IOMMU capabilities and PCI bus topologies. All of the endpoint devices within the iommuGroup (meaning devices that are not PCIe root ports, bridges, or switch ports) need to be unbound from the native host drivers in order to be assigned to a guest. In the example above, the group is composed of the GPU device (0000:02:00.0) as well as the companion audio device (0000:02:00.1). For more information, see Appendix E, Working with IOMMU Groups.

2. Adjust IOMMU settings

   In this example, assignment of NVIDIA audio functions is not supported due to hardware issues with legacy interrupt support. In addition, the GPU audio function is generally not useful without the GPU itself. Therefore, in order to assign the GPU to a guest, the audio function must first be detached from native host drivers. This can be done using one of the following:

   • Detect the PCI ID for the device and append it to the pci-stub.ids option in the /etc/sysconfig/grub file, as detailed in Procedure 16.14, “Excluding the GPU device from binding to the host physical machine driver”
   • Use the virsh nodedev-detach command, for example as follows:

     # virsh nodedev-detach pci_0000_02_00_1
     Device pci_0000_02_00_1 detached
     

Procedure 16.16. Attaching the GPU

1. Using the Virtual Machine Manager interface. For details, see Section 16.1.2, “Assigning a PCI Device with virt-manager”.
2. Creating an XML configuration fragment for the GPU and attaching it with the virsh attach-device:
   1. Create an XML for the device, similar to the following:

      
      <hostdev mode='subsystem' type='pci' managed='yes'>
       <driver name='vfio'/>
       <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
       </source>
      </hostdev>

   2. Save this to a file and run virsh attach-device [domain] [file] --persistent to include the XML in the guest configuration. Note that the assigned GPU is added in addition to the existing emulated graphics device in the guest machine. The assigned GPU is handled as a secondary graphics device in the virtual machine. Assignment as a primary graphics device is not supported and emulated graphics devices in the guest's XML should not be removed.
3. Editing the guest XML configuration using the virsh edit command and adding the appropriate XML segment manually.

Procedure 16.17. Ḿodifying the Xorg configuration on the guest

1. In the guest, use the lspci command to determine the PCI bus adress of the GPU:

   # lspci | grep VGA
   00:02.0 VGA compatible controller: Device 1234:111
   00:09.0 VGA compatible controller: NVIDIA Corporation GK106GL [Quadro K4000] (rev a1)
   

   In this example, the bus address is 00:09.0.
2. In the /etc/X11/xorg.conf file on the guest, add a BusID option with the detected address adjusted as follows:

   		Section "Device"
   		    Identifier     "Device0"
   		    Driver         "nvidia"
   		    VendorName     "NVIDIA Corporation"
   		    BusID          "PCI:0:9:0"
   		EndSection
   

   Important

   If the bus address detected in Step 1 is hexadecimal, you need to convert the values between delimiters to the decimal system. For example, 00:0a.0 should be converted into PCI:0:10:0.