Chapter 13. Configuring virtual GPUs for instances

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh openstackclient

2. Identify a node that you want to use for vGPU:

   1. Retrieve the IP address of the Compute node that you want to use for vGPU:

      $ openstack hypervisor list

   2. Use SSH to connect to the data plane node:

      $ ssh <node_ipaddress>

   3. Create the file /etc/modprobe.d/blacklist-nouveau.conf.

   4. Disable the nouveau driver by adding the following configuration to blacklist-nouveau.conf:

      blacklist nouveau
      options nouveau modeset=0

   5. Regenerate the initramfs:

      $ dracut --force
      $ grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

   6. Download and install the NVIDIA driver from the NVIDIA portal. For more information, see NVIDIA DOCS HUB.

   7. Reboot the node:

      $ sudo reboot

3. Repeat this procedure for all nodes that you want to allocate for vGPU instances.

Procedure

1. Virtual GPUs are mediated devices. Retrieve the PCI address for each device that can create mediated devices on each Compute node:

   $ ls /sys/class/mdev_bus/

   Note

   The PCI address of the GPU, or the GPU SR-IOV virtual function (VF) that can create vGPUs, is used as the device driver directory name, for example, 0000:84:00.0. In this procedure, the vGPU-capable resource is called an mdev device.

   Note

   Recent generations of NVIDIA cards now support SR-IOV. Refer to the NVIDIA documentation to discover if your GPU is SR-IOV-capable.

2. Review the supported mdev types for each available pGPU device on each Compute node to discover the available vGPU types:

   $ ls /sys/class/mdev_bus/<mdev_device>/mdev_supported_types

   • Replace <mdev_device> with the PCI address for the mdev device, for example, 0000:84:00.0. For example, the following Compute node has 4 pGPUs, and each pGPU supports the same 11 vGPU types:

     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:84:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:85:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:86:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45
     [root@computegpu-0 ~]# ls /sys/class/mdev_bus/0000:87:00.0/mdev_supported_types:
     NVIDIA-35  NVIDIA-36  NVIDIA-37  NVIDIA-38  NVIDIA-39  NVIDIA-40  NVIDIA-41  NVIDIA-42  NVIDIA-43  NVIDIA-44  NVIDIA-45

3. Create or update the ConfigMap CR named nova-extra-config.yaml and set the values of the parameters under [devices]:

   apiVersion: v1
   kind: ConfigMap
   metadata:
      name: nova-extra-config
      namespace: openstack
   data:
      34-nova-vgpu.conf: |
         [devices]
         enabled_mdev_types = nvidia-35, nvidia-36

   For more information about creating ConfigMap objects, see Creating and using config maps.

4. Optional: To configure more than one vGPU type, map the supported vGPU types to the pGPUs:

   [devices]
   enabled_mdev_types = nvidia-35, nvidia-36
   [mdev_nvidia-35]
   device_addresses = 0000:84:00.0,0000:85:00.0
   [vgpu_nvidia-36]
   device_addresses = 0000:86:00.0

   • The nvidia-35 vGPU type is supported by the pGPUs that are in the PCI addresses 0000:84:00.0 and 0000:85:00.0.
   • The nvidia-36 vGPU type is supported only by the pGPUs that are in the PCI address 0000:86:00.0.

5. Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_vgpu_deploy.yaml on your workstation:

   apiVersion: core.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
      name: compute-vgpu

6. In the compute_vgpu_deploy.yaml CR, specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite. That OpenStackDataPlaneNodeSet CR defines the nodes that you want to want to use for vGPU.

   Warning

   If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps:

   1. Check the services list of the node set and find the name of the DataPlaneService that points to nova.

   2. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova.

      If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config, then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap. If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute-vgpu
   spec:
     nodeSets:
       - openstack-edpm
       - compute-vgpu
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the `OpenStackDataPlaneNodeSet`CRs that you want to include in your data plane deployment.
7. Save the compute_vgpu_deploy.yaml deployment file.

8. Deploy the data plane:

   $ oc create -f compute_vgpu_deploy.yaml

9. Verify that the data plane is deployed:

   $ oc get openstackdataplanenodeset
   
   NAME                    STATUS    MESSAGE
   compute-vgpu   True            Deployed

   Tip

   Append the -w option to the end of the get command to track deployment progress.

10. Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane:

    $ oc rsh -n openstack openstackclient
    
    $ openstack hypervisor list

11. Optional: Enable SR-IOV VFs of the GPUs. For more information, see Preparing virtual function for SRIOV vGPU on the NVIDIA DOCS HUB.

Procedure

1. Access the remote shell for the openstackclient:

   $ oc rsh openstackclient

2. Identify the Compute node that you want to use for MIG:

   1. Retrieve the IP address of the Compute node that you want to use for MIG:

      $ openstack hypervisor list

   2. Use SSH to connect to the data plane node:

      $ ssh <node_ipaddress>

3. Identify the PCI address of your NVIDIA GPU:

   $ sudo nvidia-smi -L

4. Enable MIG mode on your GPU:

   sudo nvidia-smi -i <gpu-ids> -mig 1

5. Optional: You can query the current MIG mode of your GPU:

   $ sudo nvidia-smi -i <gpu_id> --query-gpu=pci.bus_id,mig.mode.current --format=csv

   where:

   <gpu_id>

   Specifies the PCI ID of your NVIDIA card.

   Example output:

   pci.bus_id, mig.mode.current
   00000000:17:00.0, Enabled

6. Check the partitions that are available on your NVIDIA GPU:

   $ sudo nvidia-smi mig -lgip

   Example output:

   +-------------------------------------------------------------------------------+
   | GPU instance profiles:                                                        |
   | GPU   Name               ID    Instances   Memory     P2P    SM    DEC   ENC  |
   |                                Free/Total   GiB              CE    JPEG  OFA  |
   |===============================================================================|
   |   0  MIG 1g.6gb          14     4/4        5.81       No     14     1     0   |
   |                                                               1     0     0   |
   +-------------------------------------------------------------------------------+
   |   0  MIG 1g.6gb+me       21     1/1        5.81       No     14     1     0   |
   |                                                               1     1     1   |
   +-------------------------------------------------------------------------------+
   |   0  MIG 2g.12gb          5     2/2        11.75      No     28     2     0   |
   |                                                               2     0     0   |
   +-------------------------------------------------------------------------------+
   |   0  MIG 2g.12gb+me       6     1/1        11.75      No     28     2     0   |
   |                                                               2     1     1   |
   +-------------------------------------------------------------------------------+
   |   0  MIG 4g.24gb          0     1/1        23.62      No     56     4     0   |
   |                                                               4     1     1   |
   +-------------------------------------------------------------------------------+

7. Create MIG partitions based on available profile IDs:

   $ sudo nvidia-smi mig -cgi <ID>,<ID> -C

   where:

   <ID>, <ID>

   Specifies a comma separated list of profiles that you want to create.

   Example:

   $ sudo nvidia-smi mig -cgi 14,14,14,14 -C

   Example output:

   Successfully created GPU instance ID  3 on GPU  0 using profile MIG 1g.6gb (ID 14)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID  3 using profile MIG 1g.6gb (ID  0)
   Successfully created GPU instance ID  4 on GPU  0 using profile MIG 1g.6gb (ID 14)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID  4 using profile MIG 1g.6gb (ID  0)
   Successfully created GPU instance ID  5 on GPU  0 using profile MIG 1g.6gb (ID 14)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID  5 using profile MIG 1g.6gb (ID  0)
   Successfully created GPU instance ID  6 on GPU  0 using profile MIG 1g.6gb (ID 14)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID  6 using profile MIG 1g.6gb (ID  0)q

8. Create a systemd unit file. Changes made directly to the Compute node are non-persistent, so a unit file is required to persistently recreate the partitions on reboot.

   1. Create a script called nvidia_mig_setup.sh in the /usr/local/bin directory that enables MIG and the partitions required for your environment:

      For example:

      #!/bin/bash
      udevadm settle
      sleep 5
      /usr/lib/nvidia/sriov-manage -e <gpu_id>
      nvidia-smi -i 0 -mig 1
      nvidia-smi mig -dci && nvidia-smi mig -dgi
      nvidia-smi mig -cgi 14,14,14,14 -C

      where:

      <gpu_id>

      Specifies the PCI ID of your NVIDIA card.

   2. Make the script executable:

      $ chmod +x /usr/local/bin/nvidia_mig_setup.sh

   3. Create a systemd unit file called nvidia-mig.service in /etc/systemd/system and add the following service configuration:

      [Unit]
      Description=NVIDIA MIG Partitioning Setup
      Before=edpm_libvirt.target
      After=systemd-udevd.service
      
      [Service]
      Type=oneshot
      ExecStart=/usr/local/bin/nvidia_mig_setup.sh
      RemainAfterExit=yes
      
      [Install]
      WantedBy=multi-user.target

   4. Reload systemd to recognize the new service:

      $ sudo systemctl daemon-reload

   5. Enable the service so that it starts on boot:

      $ sudo systemctl enable nvidia-mig.service

      Note

      To avoid warnings such as persistence mode is disabled, run the command sudo nvidia-smi -i <gpu_id> -pm 1.

9. Note the virtual GPU type name for your instance profile.

   Expand

   Virtual GPU type	Corresponding GPU instance Profile
   A30-4-24C	MIG 4g.24gb
   A30-2-12C	MIG 2g.12gb
   A30-1-6C	MIG 1g.6gb
   A30-1-6CME	MIG 1g.6gb+me

   Show more

   For more information see, see NVIDIA A30 and NVIDIA A30X Virtual GPU Types.

10. Use the virtual GPU type name and PCI address of your GPU that you identified in previous steps to list the MDEV types:

    VGPU_TYPE=<vgpu_type>
    PF=<gpu_id>
    for v in `ls -d /sys/bus/pci/devices/$PF/virtfn*`; do
        VF=$(readlink -f $v | xargs basename)
        for m in $(grep -El "${VGPU_TYPE}\$" /sys/bus/pci/devices/$VF/mdev_supported_types/nvidia-*/name); do
            d=$(dirname $m)
            if (( $(cat $d/available_instances) > 0 )); then echo $d; fi
        done
    done

    where:

    <vgpu_type>

    Specifies a human readable virtual GPU type name, for example A30-1-6C.

    <gpu_id>

    Specifies the PCI ID of your NVIDIA card.

11. Update the Nova ConfigMap CR file named nova-extra-config.yaml and add mdev_types:

    For example:

    [devices]
    enabled_mdev_types = <mdev_type>
    
    [mdev_nvidia_689]
    device_addresses = <sriov_ids>

    where:

    <mdev_type>

    Specifies a comma delimited list of the MDEV types that match your virtual GPU type that you want to enable. For example, nvidia-689.

    <sriov_ids>

    Specifies a comma delimited list of SR-IOV device addresses obtained.

Procedure

1. To define the maximum number of vGPUs your SR-IOV NVIDIA GPU can create for a specific vGPU type, create or update the ConfigMap CR named nova-extra-config.yaml. You must set the value of the enabled_mdev_types parameter and max_instances parameter under the specific mdev section for the vGPU type. This example configuration is for the A40-2Q NVIDIA GPU type which can create up to 24 vGPUs:

   apiVersion: v1
   kind: ConfigMap
   metadata:
      name: nova-extra-config
      namespace: openstack
   data:
      36-nova-max-instances.conf: |
         [devices]
         enabled_mdev_types = nvidia-558
   
         [mdev_nvidia-558]
         max_instances = 24

   For more information about creating ConfigMap objects, see Creating and using config maps in Nodes.

2. Save the nova-extra-config.yaml file.

3. Create a new OpenStackDataPlaneDeployment CR to configure the services on the data plane nodes and deploy the data plane, and save it to a file named compute_vgpus_max_instance_deploy.yaml on your workstation:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute_ vgpus_max_instance

4. In the compute_vgpus_max_instance_deploy.yaml, specify nodeSets to include all the OpenStackDataPlaneNodeSet CRs you want to deploy. Ensure that you include the OpenStackDataPlaneNodeSet CR that you selected as a prerequisite.

   Warning

   If your deployment has more than one node set, changes to the nova-extra-config.yaml ConfigMap might directly affect more than one node set, depending on how the node sets and the DataPlaneServices are configured. To check if a node set uses the nova-extra-config ConfigMap and therefore will be affected by the reconfiguration, complete the following steps:

   1. Check the services list of the node set and find the name of the DataPlaneService that points to nova. Ensure that the value of the edpmServiceType field of the DataPlaneService is set to nova.
   2. If the dataSources list of the DataPlaneService contains a configMapRef named nova-extra-config, then this node set uses this ConfigMap and therefore will be affected by the configuration changes in this ConfigMap. If some of the node sets that are affected should not be reconfigured, you must create a new DataPlaneService pointing to a separate ConfigMap for these node sets and use that custom service in the required node sets instead.

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute-vgpus_max_instance
   spec:
     nodeSets:
       - openstack-edpm
       - compute_vgpus_max_instance
       - ...
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
5. Save the compute_vgpus_max_instance_deploy.yaml deployment file.

6. Deploy the data plane:

   $ oc create -f compute_vgpus_max_instance_deploy.yaml

7. Verify that the data plane is deployed:

   $ oc get openstackdataplanenodeset
   NAME           STATUS MESSAGE
   compute_vgpus_max_instance True   Deployed

8. Access the remote shell for openstackclient and verify that the deployed Compute nodes are visible on the control plane:

   $ oc rsh -n openstack openstackclient
   $ openstack hypervisor list

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh openstackclient

2. Create an instance and install the NVIDIA device driver:

   $ openstack server create --flavor <flavor> \
     --image <image> --network <network> \
     --wait myInstanceFromImage

   • Replace <flavor> with the name or ID of the flavor.
   • Replace <image> with the name or ID of the image.

   • Replace <network> with the name or ID of the network. You can use the --network option more than once to connect your instance to several networks, as required.

     For more information about creating an instance, see Creating an instance in Creating and managing instances.

     1. Create the file /etc/modprobe.d/blacklist-nouveau.conf.

     2. Disable the nouveau device driver by adding the following configuration to blacklist-nouveau.conf:

        $ blacklist nouveau
        $ options nvidia modeset=0

     3. Regenerate the initramfs:

        $ dracut --force
        $ grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

     4. Download and install the NVIDIA device driver from the product portal. For more information, see NVIDIA DOCS HUB.

     5. Reboot the node:

        $ sudo reboot

3. Repeat this procedure for all instances that you want to allocate for GPU passthrough instances.

Verification

1. To verify that the GPU is correctly configured for PCI passthrough, see Creating a nodeset for PCI passthrough.

Procedure

1. Create a new trait:

    $ openstack --os-placement-api-version 1.6 trait \
    create CUSTOM_<TRAIT_NAME>

   • Replace <TRAIT_NAME> with the name of the trait. The name can contain only the letters A through Z, the numbers 0 through 9 and the underscore "_" character.

2. Collect the existing resource provider traits of each host:

   $ existing_traits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value <host_uuid> | sed 's/^/--trait /')

3. Check the existing resource provider traits for the traits you require a host or host aggregate to have:

    $ echo $existing_traits

4. If the traits you require are not already added to the resource provider, then add the existing traits and your required traits to the resource providers for each host:

    $ openstack --os-placement-api-version 1.6 \
    resource provider trait set $existing_traits \
    --trait CUSTOM_<TRAIT_NAME> \
    <host_uuid>

   • Replace <TRAIT_NAME> with the name of the trait that you want to add to the resource provider. You can use the --trait option more than once to add additional traits, as required.

     Note

     This command performs a full replacement of the traits for the resource provider. Therefore, you must retrieve the list of existing resource provider traits on the host and set them again to prevent them from being removed.

Procedure

1. Create an instance with the hardware and software profile that your vGPU instances require:

   $ openstack server create --flavor <flavor> \
    --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information about downloading RHEL cloud images, see Creating operating system images for instances in Performing storage operations.
2. Log in to the instance as a cloud user.
3. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

4. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances.

5. Create an image snapshot of the instance:

   $ openstack server image create \
    --name vgpu_image temp_vgpu_instance

6. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor, for example:

   $ openstack --os-compute-api=2.86 flavor create --vcpus 6 \
    --ram 8192 --disk 100 m1.small-gpu

2. Assign a vGPU resource to the flavor:

   $ openstack --os-compute-api=2.86 flavor set m1.small-gpu \
    --property "resources:VGPU=1"

   Note

   You can assign only one vGPU for each instance.

3. Optional: To customize the flavor for a specific vGPU type, add a required trait to the flavor:

   $ openstack --os-compute-api=2.86 flavor set m1.small-gpu \
    --property trait:CUSTOM_NVIDIA_11=required

   For information on how to create custom resource provider traits for each vGPU type, see Creating a custom vGPU resource provider trait.

Procedure

1. Create an instance using a GPU flavor and image, for example:

   $ openstack --os-compute-api=2.86 server create --flavor m1.small-gpu \
    --image vgpu_image --security-group web --nic net-id=internal0 \
    --key-name lambda vgpu-instance

2. Log in to the instance as a cloud user.

3. To verify that the GPU is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <gpu_name>