17.2. Managing NVIDIA vGPU devices

Procedure

1. Download the NVIDIA vGPU drivers and install them on your system. For instructions, see the NVIDIA documentation.

2. If the NVIDIA software installer did not create the /etc/modprobe.d/nvidia-installer-disable-nouveau.conf file, create a conf file of any name in /etc/modprobe.d/, and add the following lines in the file:

   blacklist nouveau
   options nouveau modeset=0

3. Regenerate the initial RAM disk for the current kernel, then reboot.

   # dracut --force
   # reboot

4. Check that the kernel has loaded the nvidia_vgpu_vfio module and that the nvidia-vgpu-mgr.service service is running.

   # lsmod | grep nvidia_vgpu_vfio
   nvidia_vgpu_vfio 45011 0
   nvidia 14333621 10 nvidia_vgpu_vfio
   mdev 20414 2 vfio_mdev,nvidia_vgpu_vfio
   vfio 32695 3 vfio_mdev,nvidia_vgpu_vfio,vfio_iommu_type1
   
   # systemctl status nvidia-vgpu-mgr.service
   nvidia-vgpu-mgr.service - NVIDIA vGPU Manager Daemon
      Loaded: loaded (/usr/lib/systemd/system/nvidia-vgpu-mgr.service; enabled; vendor preset: disabled)
      Active: active (running) since Fri 2018-03-16 10:17:36 CET; 5h 8min ago
    Main PID: 1553 (nvidia-vgpu-mgr)
    [...]

   In addition, if creating vGPU on an NVIDIA GPU device that is based on the Ampere (or later) architecture, ensure that virtual functions are enabled for the physical GPU. For instructions, see the NVIDIA documentation.

5. Generate a device UUID.

   # uuidgen
   30820a6f-b1a5-4503-91ca-0c10ba58692a

6. Prepare an XML file with a configuration of the mediated device, based on the detected GPU hardware. For example, the following configures a mediated device of the nvidia-63 vGPU type on an NVIDIA Tesla P4 card that runs on the 0000:01:00.0 PCI bus and uses the UUID generated in the previous step.

   <device>
       <parent>pci_0000_01_00_0</parent>
       <capability type="mdev">
           <type id="nvidia-63"/>
           <uuid>30820a6f-b1a5-4503-91ca-0c10ba58692a</uuid>
       </capability>
   </device>

7. Define a vGPU mediated device based on the XML file you prepared. For example:

   # virsh nodedev-define vgpu-test.xml
   Node device mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0 created from vgpu-test.xml

8. Optional: Verify that the mediated device is listed as inactive.

   # virsh nodedev-list --cap mdev --inactive
   mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0

9. Start the vGPU mediated device you created.

   # virsh nodedev-start mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Device mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0 started

10. Optional: Ensure that the mediated device is listed as active.

    # virsh nodedev-list --cap mdev
    mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0

11. Set the vGPU device to start automatically after the host reboots

    # virsh nodedev-autostart mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
    Device mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0 marked as autostarted

12. Attach the mediated device to a VM that you want to share the vGPU resources. To do so, add the following lines, along with the previously genereated UUID, to the <devices/> sections in the XML configuration of the VM.

    • To attach a single vGPU to a VM:

      ...
      <video>
        <model type='none'/>
      </video>
      ...
      
      <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on' ramfb="on">
        <source>
          <address uuid='30820a6f-b1a5-4503-91ca-0c10ba58692a'/>
        </source>
      </hostdev>

      Note that each UUID can only be assigned to one VM at a time.

    • To attach multiple vGPUs to a VM:

      ...
      <video>
        <model type='none'/>
      </video>
      ...
      
      <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on' ramfb="on">
        <source>
          <address uuid='30820a6f-b1a5-4503-91ca-0c10ba58692a'/>
        </source>
      </hostdev>
      
      <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'>
        <source>
          <address uuid='751c2bd9-3322-4031-ad07-50a0d1367577'/>
        </source>
      </hostdev>

13. For full functionality of the vGPU mediated devices to be available on the assigned VMs, set up NVIDIA vGPU guest software licensing on the VMs. For further information and instructions, see the NVIDIA Virtual GPU Software License Server User Guide.

Verification

1. Query the capabilities of the vGPU you created, and ensure it is listed as active and persistent.

   # virsh nodedev-info mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Name:           virsh nodedev-autostart mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Parent:         pci_0000_01_00_0
   Active:         yes
   Persistent:     yes
   Autostart:      yes

2. Start the VM and verify that the guest operating system detects the mediated device as an NVIDIA GPU. For example, if the VM uses Linux:

   # lspci -d 10de: -k
   07:00.0 VGA compatible controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2 32GB] (rev a1)
           Subsystem: NVIDIA Corporation Device 12ce
           Kernel driver in use: nvidia
           Kernel modules: nouveau, nvidia_drm, nvidia

Procedure

1. Obtain the ID of the mediated device that you want to remove.

   # virsh nodedev-list --cap mdev
   mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0

2. Stop the running instance of the vGPU mediated device.

   # virsh nodedev-destroy mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Destroyed node device 'mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0'

3. Optional: Ensure the mediated device has been deactivated.

   # virsh nodedev-info mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Name:           virsh nodedev-autostart mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Parent:         pci_0000_01_00_0
   Active:         no
   Persistent:     yes
   Autostart:      yes

4. Remove the device from the XML configuration of the VM. To do so, use the virsh edit utility to edit the XML configuration of the VM, and remove the mdev’s configuration segment. The segment will look similar to the following:

   <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'>
     <source>
       <address uuid='30820a6f-b1a5-4503-91ca-0c10ba58692a'/>
     </source>
   </hostdev>

   Note that stopping and detaching the mediated device does not delete it, but rather keeps it as defined. As such, you can restart and attach the device to a different VM.

5. Optional: To delete the stopped mediated device, remove its definition.

   # virsh nodedev-undefine mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0
   Undefined node device 'mdev_30820a6f_b1a5_4503_91ca_0c10ba58692a_0000_01_00_0'