Chapter 6. Deploying SR-IOV technologies

1. Generate the built-in ComputeSriov to define nodes in the OpenStack cluster that will run NeutronSriovAgent, NeutronSriovHostConfig and default compute services.

   # openstack overcloud roles generate \
   -o /home/stack/templates/roles_data.yaml \
   Controller ComputeSriov

2. Include the neutron-sriov.yaml and roles_data.yaml files when generating overcloud_images.yaml so that SR-IOV containers are prepared.

   SERVICES=\
   /usr/share/openstack-tripleo-heat-templates/environments/services
   
   openstack overcloud container image prepare \
   --namespace=registry.redhat.io/rhosp13 \
   --push-destination=192.168.24.1:8787 \
   --prefix=openstack- \
   --tag-from-label {version}-{release} \
   -e ${SERVICES}/neutron-sriov.yaml \
   --roles-file /home/stack/templates/roles_data.yaml \
   --output-env-file=/home/stack/templates/overcloud_images.yaml \
   --output-images-file=/home/stack/local_registry_images.yaml

   Note

   The push-destination IP address is the address that you previously set with the local_ip parameter in the undercloud.conf configuration file.

   For more information on container image preparation, see Director Installation and Usage.

3. To apply the KernelAgs and TunedProfile parameters, include the host-config-and-reboot.yaml file from /usr/share/openstack-tripleo-heat-templates/environments to your deployment script.

   openstack overcloud deploy --templates \
   … \
   -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml \
   ...

4. Configure the parameters for the SR-IOV nodes under parameter_defaults in accordance with the needs of your cluster, and the configuration of your hardware. These settings are typically added to the network-environment.yaml file.

     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

5. In the same file, configure role specific parameters for SR-IOV compute nodes.

   Note

   The NeutronSriovNumVFs parameter will soon be deprecated in favor of the numvfs attribute in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter, nor the numvfs parameter, after deployment. Changing either parameter within a running environment is known to cause a permanent outage for all running instances which have an SR-IOV port on that PF. Unless you hard reboot these instances, the SR-IOV PCI device will not be visible to the instance.

     ComputeSriovParameters:
       IsolCpusList: "1-19,21-39"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-19,21-39"
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysicalDevMappings:
         - tenant:p7p1
         - tenant:p7p2
       NeutronSriovNumVFs:
         - p7p1:5
         - p7p2:5
       NovaPCIPassthrough:
         - vendor_id: "8086"
           product_id: "1528"
           address: "0000:06:00.0"
           physical_network: "tenant"
         - vendor_id: "8086"
           product_id: "1528"
           address: "0000:06:00.1"
           physical_network: "tenant"
       NovaVcpuPinSet: '1-19,21-39'
       NovaReservedHostMemory: 4096

   Note

   Do not use the devname parameter when configuring PCI passthrough, as the device name of a NIC can change. Instead, use vendor_id and product_id because they are more stable, or use the address of the NIC. For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

6. Configure the SR-IOV enabled interfaces in the compute.yaml network configuration template. Ensure the interfaces are configured as standalone NICs for the purposes of creating SR-IOV virtual functions (VFs):

                - type: interface
                   name: p7p3
                   mtu: 9000
                   use_dhcp: false
                   defroute: false
                   nm_controlled: true
                   hotplug: true
   
                 - type: interface
                   name: p7p4
                   mtu: 9000
                   use_dhcp: false
                   defroute: false
                   nm_controlled: true
                   hotplug: true

7. Ensure that the list of default filters includes the value AggregateInstanceExtraSpecsFilter.

   NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','AggregateInstanceExtraSpecsFilter']

8. Deploy the overcloud.

Procedure

1. Generate the ComputeSriov role:

   openstack overcloud roles generate -o roles_data.yaml Controller ComputeSriov

2. Add the OvsHwOffload parameter under role-specific parameters with a value of true.
3. To configure neutron to use the iptables/hybrid firewall driver implementation, include the line: NeutronOVSFirewallDriver: iptables_hybrid. For more information about NeutronOVSFirewallDriver, see Using the Open vSwitch Firewall in the Advanced Overcloud Customization Guide.

4. Configure the physical_network parameter to match your environment.

   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Example:

     parameter_defaults:
       NeutronOVSFirewallDriver: iptables_hybrid
       ComputeSriovParameters:
         IsolCpusList: 2-9,21-29,11-19,31-39
         KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
         OvsHwOffload: true
         TunedProfileName: "cpu-partitioning"
         NeutronBridgeMappings:
           - tenant:br-tenant
         NovaPCIPassthrough:
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "tenant"
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "null"
         NovaReservedHostMemory: 4096
         NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39

   • Replace <vendor-id> with the vendor ID of the physical NIC.
   • Replace <product-id> with the product ID of the NIC VF.

   • Replace <address> with the address of the physical NIC.

     For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

5. Ensure that the list of default filters includes NUMATopologyFilter:

     NovaSchedulerDefaultFilters: [\'RetryFilter',\'AvailabilityZoneFilter',\'ComputeFilter',\'ComputeCapabilitiesFilter',\'ImagePropertiesFilter',\'ServerGroupAntiAffinityFilter',\'ServerGroupAffinityFilter',\'PciPassthroughFilter',\'NUMATopologyFilter']

6. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Note
   • Do not use the NeutronSriovNumVFs parameter when configuring Open vSwitch hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as a nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.

7. Include the ovs-hw-offload.yaml file in the overcloud deploy command:

   TEMPLATES_HOME=”/usr/share/openstack-tripleo-heat-templates”
   CUSTOM_TEMPLATES=”/home/stack/templates”
   
   openstack overcloud deploy --templates \
     -r ${CUSTOM_TEMPLATES}/roles_data.yaml \
     -e ${TEMPLATES_HOME}/environments/ovs-hw-offload.yaml \
     -e ${CUSTOM_TEMPLATES}/network-environment.yaml \
     -e ${CUSTOM_TEMPLATES}/neutron-ovs.yaml

1. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

2. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

3. Create the port.

   • Use vnic-type direct to create an SR-IOV virtual function (VF) port.

     # openstack port create --network net1 --vnic-type direct sriov_port

   • Use the following to create a virtual function with hardware offload.

     # openstack port create --network net1 --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]} sriov_hwoffload_port

   • Use vnic-type direct-physical to create an SR-IOV PF port.

     # openstack port create --network net1 --vnic-type direct-physical sriov_port

4. Deploy an instance

   # openstack server create --flavor <flavor> --image <image> --nic port-id=<id> <instance name>

1. You can configure the AggregateInstanceExtraSpecsFilter value, and other necessary filters, through the heat parameter NovaSchedulerDefaultFilters under parameter_defaults in the nova.conf configuration file before deployment.

     parameter_defaults:
       NovaSchedulerDefaultFilters: ['AggregateInstanceExtraSpecsFilter', 'RetryFilter','AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

   Note

   To add the AggregateInstanceExtraSpecsFilter configuration to an exiting cluster, you can add this parameter to the heat templates, and run the original deployment script again.

2. Create an aggregate group for single root I/O virtualization (SR-IOV), and add relevant hosts. Define metadata, for example, sriov=true, that matches defined flavor metadata.

   # openstack aggregate create sriov_group
   # openstack aggregate add host sriov_group compute-sriov-0.localdomain
   # openstack aggregate set --property sriov=true sriov_group

3. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

4. Set additional flavor properties. Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   openstack flavor set --property sriov=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB <flavor>