Chapter 11. Configuring virtual GPUs for instances

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh openstackclient

   Copy to Clipboard Toggle word wrap

2. Identify a node that you want to use for vGPU:

   1. Retrieve the IP address of the Compute node that you want to use for vGPU:

      $ openstack hypervisor list

      Copy to Clipboard Toggle word wrap

   2. Use SSH to connect to the data plane node:

      $ ssh <node_ipaddress>

      Copy to Clipboard Toggle word wrap
   3. Create the file /etc/modprobe.d/blacklist-nouveau.conf.

   4. Disable the nouveau driver by adding the following configuration to blacklist-nouveau.conf:

      blacklist nouveau
      options nouveau modeset=0

      Copy to Clipboard Toggle word wrap

   5. Regenerate the initramfs:

      $ dracut --force
      $ grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

      Copy to Clipboard Toggle word wrap
   6. Download and install the NVIDIA driver from the NVIDIA portal. For more information, see NVIDIA DOCS HUB.

   7. Reboot the node:

      $ sudo reboot

      Copy to Clipboard Toggle word wrap
3. Repeat this procedure for all nodes that you want to allocate for vGPU instances.

Procedure

1. Virtual GPUs are mediated devices. Retrieve the PCI address for each device that can create mediated devices on each Compute node:

   $ ls /sys/class/mdev_bus/

   Copy to Clipboard Toggle word wrap
   Note

   The PCI address of the GPU - or the GPU SR-IOV virtual function (VF) that can create vGPUs - is used as the device driver directory name, for example, 0000:84:00.0. In this procedure, the vGPU-capable resource is called an mdev device.

   Note

   Recent generations of NVIDIA cards now support SR-IOV. Refer to the NVIDIA documentation to discover if your GPU is SR-IOV-capable.

2. Review the supported mdev types for each available pGPU device on each Compute node to discover the available vGPU types:

   $ ls /sys/class/mdev_bus/<mdev_device>/mdev_supported_types

   Copy to Clipboard Toggle word wrap

   • Replace <mdev_device> with the PCI address for the mdev device, for example, 0000:84:00.0. For example, the following Compute node has 4 pGPUs, and each pGPU supports the same 11 vGPU types:

     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:84:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:85:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:86:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:87:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45

     Copy to Clipboard Toggle word wrap

3. Create or update the ConfigMap CR named nova-extra-config.yaml and set the values of the parameters under [devices]:

   apiVersion: v1
   kind: ConfigMap
   metadata:
      name: nova-extra-config
      namespace: openstack
   data:
      34-nova-vgpu.conf: |
         [devices]
         enabled_mdev_types = nvidia-35, nvidia-36

   Copy to Clipboard Toggle word wrap

   For more information about creating ConfigMap objects, see Creating and using config maps.

4. Optional: To configure more than one vGPU type, map the supported vGPU types to the pGPUs:

   [devices]
   enabled_mdev_types = nvidia-35, nvidia-36
   [mdev_nvidia-35]
   device_addresses = 0000:84:00.0,0000:85:00.0
   [vgpu_nvidia-36]
   device_addresses = 0000:86:00.0

   Copy to Clipboard Toggle word wrap

   The nvidia-35 vGPU type is supported by the pGPUs that are in the PCI addresses 0000:84:00.0 and 0000:85:00.0. The nvidia-36 vGPU type is supported only by the pGPUs that are in the PCI address 0000:86:00.0.

5. Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_vgpu_deploy.yaml on your workstation:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
      name: compute-vgpu

   Copy to Clipboard Toggle word wrap

6. In the compute_vgpu_deploy.yaml CR, specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite. That OpenStackDataPlaneNodeSet CR defines the nodes that you want to want to use for vGPU.

   Warning

   If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps:

   1. Check the services list of the node set and find the name of the DataPlaneService that points to nova.

   2. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova.

      If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config, then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap. If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute-vgpu
   spec:
     nodeSets:
       - openstack-edpm
       - compute-vgpu
       - ...
       - <nodeSet_name>

   Copy to Clipboard Toggle word wrap
   • Replace <nodeSet_name> with the names of the `OpenStackDataPlaneNodeSet`CRs that you want to include in your data plane deployment.
7. Save the compute_vgpu_deploy.yaml deployment file.

8. Deploy the data plane:

   $ oc create -f compute_vgpu_deploy.yaml

   Copy to Clipboard Toggle word wrap

9. Verify that the data plane is deployed:

   $ oc get openstackdataplanenodeset
   
   NAME                    STATUS    MESSAGE
   compute-vgpu   True            Deployed

   Copy to Clipboard Toggle word wrap
   Tip

   Append the -w option to the end of the get command to track deployment progress.

10. Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane:

    $ oc rsh -n openstack openstackclient
    
    $ openstack hypervisor list

    Copy to Clipboard Toggle word wrap
11. Optional: Enable SR-IOV VFs of the GPUs. For more information, see Preparing virtual function for SRIOV vGPU on the NVIDIA DOCS HUB.

Procedure

1. To define the maximum number of vGPUs your SR-IOV NVIDIA GPU can create for a specific vGPU type, create or update the ConfigMap CR named nova-extra-config.yaml. You must set the value of the enabled_mdev_types parameter and max_instances parameter under the specific mdev section for the vGPU type. This example configuration is for the A40-2Q NVIDIA GPU type which can create up to 24 vGPUs:

   apiVersion: v1
   kind: ConfigMap
   metadata:
      name: nova-extra-config
      namespace: openstack
   data:
      36-nova-max-instances.conf: |
         [devices]
         enabled_mdev_types = nvidia-558
   
         [mdev_nvidia-558]
         max_instances = 24

   Copy to Clipboard Toggle word wrap

   For more information about creating ConfigMap objects, see Creating and using config maps in Nodes.

2. Save the nova-extra-config.yaml file.

3. Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_vgpus_max_instance_deploy.yaml on your workstation:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute_ vgpus_max_instance

   Copy to Clipboard Toggle word wrap

4. In the compute_vgpus_max_instance_deploy.yaml, specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite.

   Warning

   If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps:

   1. Check the services list of the node set and find the name of the DataPlaneService that points to nova. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova.
   2. If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config, then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap. If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets and use that custom service in the required node sets instead.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute-vgpus_max_instance
   spec:
     nodeSets:
       - openstack-edpm
       - compute_vgpus_max_instance
       - ...
       - <nodeSet_name>

   Copy to Clipboard Toggle word wrap
   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
5. Save the compute_vgpus_max_instance_deploy.yaml deployment file.

6. Deploy the data plane:

   $ oc create -f compute_vgpus_max_instance_deploy.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the data plane is deployed:

   $ oc get openstackdataplanenodeset
   NAME           STATUS MESSAGE
   compute_vgpus_max_instance True   Deployed

   Copy to Clipboard Toggle word wrap

8. Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane:

   $ oc rsh -n openstack openstackclient
   $ openstack hypervisor list

   Copy to Clipboard Toggle word wrap

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh openstackclient

   Copy to Clipboard Toggle word wrap

2. Create an instance and install the NVIDIA device driver:

   $ openstack server create --flavor <flavor> \
     --image <image> --network <network> \
     --wait myInstanceFromImage

   Copy to Clipboard Toggle word wrap
   • Replace <flavor> with the name or ID of the flavor.
   • Replace <image> with the name or ID of the image.

   • Replace <network> with the name or ID of the network. You can use the --network option more than once to connect your instance to several networks, as required.

     For more information about creating an instance, see Creating an instance in Creating and managing instances.

     1. Create the file /etc/modprobe.d/blacklist-nouveau.conf.

     2. Disable the nouveau device driver by adding the following configuration to blacklist-nouveau.conf:

        $ blacklist nouveau
        $ options nvidia modeset=0

        Copy to Clipboard Toggle word wrap

     3. Regenerate the initramfs:

        $ dracut --force
        $ grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

        Copy to Clipboard Toggle word wrap
     4. Download and install the NVIDIA device driver from the product portal. For more information, see NVIDIA DOCS HUB.

     5. Reboot the node:

        $ sudo reboot

        Copy to Clipboard Toggle word wrap
3. Repeat this procedure for all instances that you want to allocate for GPU passthrough instances.

Verification

1. To verify that the GPU is correctly configured for PCI passthrough, see Creating a nodeset for PCI passthrough.

Procedure

1. Create a new trait:

    $ openstack --os-placement-api-version 1.6 trait \
    create CUSTOM_<TRAIT_NAME>

   Copy to Clipboard Toggle word wrap
   • Replace <TRAIT_NAME> with the name of the trait. The name can contain only the letters A through Z, the numbers 0 through 9 and the underscore "_" character.

2. Collect the existing resource provider traits of each host:

   $ existing_traits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')

   Copy to Clipboard Toggle word wrap

3. Check the existing resource provider traits for the traits you require a host or host aggregate to have:

    $ echo $existing_traits

   Copy to Clipboard Toggle word wrap

4. If the traits you require are not already added to the resource provider, then add the existing traits and your required traits to the resource providers for each host:

    $ openstack --os-placement-api-version 1.6 \
    resource provider trait set $existing_traits \
    --trait CUSTOM_<TRAIT_NAME> \
    <host_uuid>

   Copy to Clipboard Toggle word wrap

   • Replace <TRAIT_NAME> with the name of the trait that you want to add to the resource provider. You can use the --trait option more than once to add additional traits, as required.

     Note

     This command performs a full replacement of the traits for the resource provider. Therefore, you must retrieve the list of existing resource provider traits on the host and set them again to prevent them from being removed.

Procedure

1. Create an instance with the hardware and software profile that your vGPU instances require:

   $ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   Copy to Clipboard Toggle word wrap
   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Creating RHEL KVM or RHOSP-compatible images in Creating and managing images.
2. Log in to the instance as a cloud user.
3. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

4. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Image configuration parameters.

5. Create an image snapshot of the instance:

   $ openstack server image create \
    --name vgpu_image temp_vgpu_instance

   Copy to Clipboard Toggle word wrap
6. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   $ openstack --os-compute-api=2.86 flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu

   Copy to Clipboard Toggle word wrap

2. Assign a vGPU resource to the flavor:

   $ openstack --os-compute-api=2.86 flavor set m1.small-gpu \
    --property "resources:VGPU=1"

   Copy to Clipboard Toggle word wrap
   Note

   You can assign only one vGPU for each instance.

3. Optional: To customize the flavor for a specific vGPU type, add a required trait to the flavor:

   $ openstack --os-compute-api=2.86 flavor set m1.small-gpu \
    --property trait:CUSTOM_NVIDIA_11=required

   Copy to Clipboard Toggle word wrap

   For information on how to create custom resource provider traits for each vGPU type, see Creating a custom vGPU resource provider trait.

Procedure

1. Create an instance using a GPU flavor and image, for example:

   $ openstack --os-compute-api=2.86 server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

   Copy to Clipboard Toggle word wrap
2. Log in to the instance as a cloud-user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>

   Copy to Clipboard Toggle word wrap