Chapter 10. Configuring virtual GPUs for instances

1. Designate Compute nodes for vGPU.
2. Configure the Compute node for vGPU.
3. Deploy the overcloud.
4. Optional: Create custom traits for vGPU types.
5. Optional: Create a custom GPU instance image.
6. Create a vGPU flavor for launching instances that have vGPU.

Procedure

1. Create a new trait:

   (overcloud)$ openstack --os-placement-api-version 1.6 trait \
    create CUSTOM_<TRAIT_NAME>

   • Replace <TRAIT_NAME> with the name of the trait. The name can contain only the letters A through Z, the numbers 0 through 9 and the underscore "_" character.

2. Collect the existing resource provider traits of each host:

   (overcloud)$ existing_traits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')

3. Check the existing resource provider traits for the traits you require a host or host aggregate to have:

   (overcloud)$ echo $existing_traits

4. If the traits you require are not already added to the resource provider, then add the existing traits and your required traits to the resource providers for each host:

   (overcloud)$ openstack --os-placement-api-version 1.6 \
    resource provider trait set $existing_traits \
    --trait CUSTOM_<TRAIT_NAME> \
    <host_uuid>

   • Replace <TRAIT_NAME> with the name of the trait that you want to add to the resource provider. You can use the --trait option more than once to add additional traits, as required.

     Note

     This command performs a full replacement of the traits for the resource provider. Therefore, you must retrieve the list of existing resource provider traits on the host and set them again to prevent them from being removed.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the overcloudrc credential file:

   $ source ~/overcloudrc

3. Create an instance with the hardware and software profile that your vGPU instances require:

   (overcloud)$ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require. For information about creating a vGPU flavor, see Creating a vGPU flavor for instances.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Creating RHEL KVM or RHOSP-compatible images in Creating and managing images.
4. Log in to the instance as a cloud-user.
5. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

6. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Image configuration parameters.

7. Create an image snapshot of the instance:

   (overcloud)$ openstack server image create \
    --name vgpu_image temp_vgpu_instance

8. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   (overcloud)$ openstack flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu

2. Assign a vGPU resource to the flavor:

   (overcloud)$ openstack flavor set m1.small-gpu \
    --property "resources:VGPU=1"

   Note

   You can assign only one vGPU for each instance.

3. Optional: To customize the flavor for a specific vGPU type, add a required trait to the flavor:

   (overcloud)$ openstack flavor set m1.small-gpu \
    --property trait:CUSTOM_NVIDIA_11=required

   For information on how to create custom resource provider traits for each vGPU type, see Creating a custom vGPU resource provider trait.

Procedure

1. Create an instance using a GPU flavor and image, for example:

   (overcloud)$ openstack server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

2. Log in to the instance as a cloud-user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

Procedure

1. To determine the vendor ID and product ID for each passthrough device type, enter the following command on the physical server that has the PCI cards:

   # lspci -nn | grep -i <gpu_name>

   For example, to determine the vendor and product ID for an NVIDIA GPU, enter the following command:

   # lspci -nn | grep -i nvidia
   3b:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
   d8:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1db4] (rev a1)

2. To determine if each PCI device has Single Root I/O Virtualization (SR-IOV) capabilities, enter the following command on the physical server that has the PCI cards:

   # lspci -v -s 3b:00.0
   3b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
             ...
             Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
             ...

3. To configure the Controller node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_controller.yaml.

4. Add PciPassthroughFilter to the NovaSchedulerEnabledFilters parameter in pci_passthru_controller.yaml:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

5. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthru_controller.yaml:

   • If the PCI device has SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     For more information on configuring the device_type field, see PCI passthrough device type field.

     Note

     If the nova-api service is running in a role other than the Controller, then replace ControllerExtraConfig with the user role, in the format <Role>ExtraConfig.

6. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthru_compute.yaml.

7. To specify the available PCIs for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   parameter_defaults:
     NovaPCIPassthrough:
       - vendor_id: "10de"
         product_id: "1eb8"

8. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the Compute node, add the following to pci_passthru_compute.yaml:

   • If the PCI device has SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
           device_type: "type-PF"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"
           device_type: "type-PF"

   • If the PCI device does not have SR-IOV capabilities:

     ComputeExtraConfig:
       nova::pci::aliases:
         - name: "t4"
           product_id: "1eb8"
           vendor_id: "10de"
         - name: "v100"
           product_id: "1db4"
           vendor_id: "10de"

     Note

     The Compute node aliases must be identical to the aliases on the Controller node.

9. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthru_compute.yaml:

   parameter_defaults:
     ...
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment. For more information, see Configuring manual node reboot to define KernelArgs.

10. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
      -e [your environment files] \
      -e /home/stack/templates/pci_passthru_controller.yaml \
      -e /home/stack/templates/pci_passthru_compute.yaml

11. Configure a flavor to request the PCI devices. The following example requests two devices, each with a vendor ID of 10de and a product ID of 13f2:

    # openstack flavor set m1.large \
     --property "pci_passthrough:alias"="t4:2"

Verification

1. Create an instance with a PCI passthrough device:

   # openstack server create --flavor m1.large \
    --image <custom_gpu> --wait test-pci

   Replace <custom_gpu> with the name of your custom instance image that has the required GPU driver installed.

2. Log in to the instance as a cloud user. For more information, see Connecting to an instance.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

4. To check the NVIDIA System Management Interface status, enter the following command from the instance:

   $ nvidia-smi

   Example output:

   -----------------------------------------------------------------------------
   | NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
   |---------------------------------------------------------------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |===========================================================================|
   |   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |
   | N/A   43C    P0    20W /  70W |      0MiB / 15109MiB |      0%      Default |
   ---------------------------------------------------------------------------
   
   -----------------------------------------------------------------------------
   | Processes:                                                       GPU Memory |
   |  GPU       PID   Type   Process name                             Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   -----------------------------------------------------------------------------