Chapter 8. Configuring PCI passthrough

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Generate a new roles data file named roles_data_pci_passthrough.yaml that includes the Controller, Compute, and ComputePCI roles, along with any other roles that you need for the overcloud:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_pci_passthrough.yaml \
    Compute:ComputePCI Compute Controller

   Copy to Clipboard Toggle word wrap

4. Open roles_data_pci_passthrough.yaml and edit or add the following parameters and sections:

   Expand

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputePCI
   Role name	name: Compute	name: ComputePCI
   description	Basic Compute Node role	PCI Passthrough Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepci-%index%
   deprecated_nic_config_name	compute.yaml	compute-pci-passthrough.yaml

   Show more
5. Register the PCI passthrough Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   Copy to Clipboard Toggle word wrap

   For more information, see Creating an inventory of the bare-metal node hardware in the Installing and managing Red Hat OpenStack Platform with director guide.

7. Tag each bare metal node that you want to designate for PCI passthrough with a custom PCI passthrough resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.PCI-PASSTHROUGH <node>

   Copy to Clipboard Toggle word wrap

   Replace <node> with the ID of the bare metal node.

8. Add the ComputePCI role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
   - name: Compute
     count: 3
   - name: ComputePCI
     count: 1
     defaults:
       resource_class: baremetal.PCI-PASSTHROUGH
       network_config:
         template: /home/stack/templates/nic-config/myRoleTopology.j2 

   1

   Copy to Clipboard Toggle word wrap

   1 1

   You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

   For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes. For an example node definition file, see Example node definition file.

9. Run the provisioning command to provision the new nodes for your role:

   (undercloud)$ openstack overcloud node provision \
   --stack <stack> \
   [--network-config \]
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

   Copy to Clipboard Toggle word wrap
   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

    Copy to Clipboard Toggle word wrap

11. If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
       ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
       ComputePCINetworkConfigTemplate: /home/stack/templates/nic-configs/<pci_passthrough_net_top>.j2
       ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Copy to Clipboard Toggle word wrap

    Replace <pci_passthrough_net_top> with the name of the file that contains the network topology of the ComputePCI role, for example, compute.yaml to use the default network topology.

Procedure

1. Create an environment file to configure the Controller node on the overcloud for PCI passthrough, for example, pci_passthrough_controller.yaml.

2. Add PciPassthroughFilter to the NovaSchedulerEnabledFilters parameter in pci_passthrough_controller.yaml:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   Copy to Clipboard Toggle word wrap

3. To specify the PCI alias for the devices on the Controller node, add the following configuration to pci_passthrough_controller.yaml:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   Copy to Clipboard Toggle word wrap

   For more information about configuring the device_type field, see PCI passthrough device type field.

   Note

   If the nova-api service is running in a role different from the Controller role, replace ControllerExtraConfig with the user role in the format <Role>ExtraConfig.

4. Optional: To set a default NUMA affinity policy for PCI passthrough devices, add numa_policy to the nova::pci::aliases: configuration from step 3:

   parameter_defaults:
     ...
     ControllerExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"
           numa_policy: "preferred"

   Copy to Clipboard Toggle word wrap
5. To configure the Compute node on the overcloud for PCI passthrough, create an environment file, for example, pci_passthrough_compute.yaml.

6. To specify the available PCIs for the devices on the Compute node, use the vendor_id and product_id options to add all matching PCI devices to the pool of PCI devices available for passthrough to instances. For example, to add all Intel® Ethernet Controller X710 devices to the pool of PCI devices available for passthrough to instances, add the following configuration to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIParameters:
       NovaPCIPassthrough:
         - vendor_id: "8086"
           product_id: "1572"

   Copy to Clipboard Toggle word wrap

   For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

7. You must create a copy of the PCI alias on the Compute node for instance migration and resize operations. To specify the PCI alias for the devices on the PCI passthrough Compute node, add the following to pci_passthrough_compute.yaml:

   parameter_defaults:
     ...
     ComputePCIExtraConfig:
       nova::pci::aliases:
         - name: "a1"
           product_id: "1572"
           vendor_id: "8086"
           device_type: "type-PF"

   Copy to Clipboard Toggle word wrap
   Note

   The Compute node aliases must be identical to the aliases on the Controller node. Therefore, if you added numa_affinity to nova::pci::aliases in pci_passthrough_controller.yaml, then you must also add it to nova::pci::aliases in pci_passthrough_compute.yaml.

8. To enable IOMMU in the server BIOS of the Compute nodes to support PCI passthrough, add the KernelArgs parameter to pci_passthrough_compute.yaml:

   • Use the following KernalArgs settings to enable an Intel IOMMU:

     parameter_defaults:
       ...
       ComputePCIParameters:
         KernelArgs: "intel_iommu=on iommu=pt vfio-pci.ids=<pci_device_id> rd.driver.pre=vfio-pci"

     Copy to Clipboard Toggle word wrap
     • Replace <pci_device_id> with the PCI device ID, for example, 10de:1eb8.

   • Use the following KernalArgs settings to enable an AMD IOMMU:

     parameter_defaults:
       ...
       ComputePCIParameters:
          KernelArgs: "amd_iommu=on iommu=pt"

     Copy to Clipboard Toggle word wrap
     Note

     When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each deployment. For more information, see Configuring manual node reboot to define KernelArgs.

9. Add your custom environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -r /home/stack/templates/roles_data_pci_passthrough.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/pci_passthrough_controller.yaml \
     -e /home/stack/templates/pci_passthrough_compute.yaml \
     -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
     -e /home/stack/templates/node-info.yaml

   Copy to Clipboard Toggle word wrap

10. Create and configure the flavors that your cloud users can use to request the PCI devices. The following example requests two devices, each with a vendor ID of 8086 and a product ID of 1572, using the alias defined in step 7:

    (overcloud)$ openstack flavor set \
     --property "pci_passthrough:alias"="a1:2" device_passthrough

    Copy to Clipboard Toggle word wrap

11. Optional: To override the default NUMA affinity policy for PCI passthrough devices, you can add the NUMA affinity policy property key to the flavor or the image:

    • To override the default NUMA affinity policy by using the flavor, add the hw:pci_numa_affinity_policy property key:

      (overcloud)$ openstack flavor set \
       --property "hw:pci_numa_affinity_policy"="required" \
       device_passthrough

      Copy to Clipboard Toggle word wrap

      For more information about the valid values for hw:pci_numa_affinity_policy, see Flavor metadata.

    • To override the default NUMA affinity policy by using the image, add the hw_pci_numa_affinity_policy property key:

      (overcloud)$ openstack image set \
       --property hw_pci_numa_affinity_policy=required \
       device_passthrough_image

      Copy to Clipboard Toggle word wrap
      Note

      If you set the NUMA affinity policy on both the image and the flavor then the property values must match. The flavor setting takes precedence over the image and default settings. Therefore, the configuration of the NUMA affinity policy on the image only takes effect if the property is not set on the flavor.

Verification

1. Create an instance with a PCI passthrough device:

   $ openstack server create --flavor device_passthrough \
    --image <image> --wait test-pci

   Copy to Clipboard Toggle word wrap
2. Log in to the instance as a cloud user. For more information, see Connecting to an instance.

3. To verify that the PCI device is accessible from the instance, enter the following command from the instance:

   $ lspci -nn | grep <device_name>

   Copy to Clipboard Toggle word wrap