Chapter 7. Deploying SR-IOV technologies

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Generate a new roles data file named roles_data_compute_sriov.yaml that includes the Controller and ComputeSriov roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_compute_sriov.yaml \
    Controller ComputeSriov

   Copy to Clipboard Toggle word wrap

   ComputeSriov is a custom role provided with your RHOSP installation that includes the NeutronSriovAgent and NeutronSriovHostConfig services, in addition to the default compute services.

4. To prepare the SR-IOV containers, include the neutron-sriov.yaml and roles_data_compute_sriov.yaml files when you generate the overcloud_images.yaml file.

   $ sudo openstack tripleo container image prepare \
     --roles-file ~/templates/roles_data_compute_sriov.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml \
     -e ~/containers-prepare-parameter.yaml \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   Copy to Clipboard Toggle word wrap

   For more information on container image preparation, see Preparing container images in the Director Installation and Usage guide.

5. Create a copy of the /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml file in your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml /home/stack/templates/network-environment-sriov.yaml

   Copy to Clipboard Toggle word wrap

6. Add the following parameters under parameter_defaults in your network-environment-sriov.yaml file to configure the SR-IOV nodes for your cluster and your hardware configuration:

     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

   Copy to Clipboard Toggle word wrap

7. To determine the vendor_id and product_id for each PCI device type, use one of the following commands on the physical server that has the PCI cards:

   • To return the vendor_id and product_id from a deployed overcloud, use the following command:

     # lspci -nn -s  <pci_device_address>
     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

     Copy to Clipboard Toggle word wrap

   • To return the vendor_id and product_id of a physical function (PF) if you have not yet deployed the overcloud, use the following command:

     (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

     Copy to Clipboard Toggle word wrap

8. Configure role specific parameters for SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       IsolCpusList: "1-19,21-39"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-19,21-39"
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysicalDevMappings:
         - tenant:p7p1
       NovaComputeCpuDedicatedSet: '1-19,21-39'
       NovaReservedHostMemory: 4096

   Copy to Clipboard Toggle word wrap
   Note

   The NovaVcpuPinSet parameter is now deprecated, and is replaced by NovaComputeCpuDedicatedSet for dedicated, pinned workloads.

9. Configure the PCI passthrough devices for the SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       ...
       NovaPCIPassthrough:
         - vendor_id: "<vendor_id>"
           product_id: "<product_id>"
           address: <NIC_address>
           physical_network: "<physical_network>"
       ...

   Copy to Clipboard Toggle word wrap
   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device. For information about how to configure the address parameter, see Guidelines for configuring NovaPCIPassthrough in the Configuring the Compute Service for Instance Creation guide.

   • Replace <physical_network> with the name of the physical network the PCI device is located on.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts. For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough in the Configuring the Compute Service for Instance Creation guide.

10. Configure the SR-IOV enabled interfaces in the compute.yaml network configuration template. To create SR-IOV VFs, configure the interfaces as standalone NICs:

                 - type: sriov_pf
                    name: p7p3
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false
    
                  - type: sriov_pf
                    name: p7p4
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false

    Copy to Clipboard Toggle word wrap
    Note

    The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

11. Ensure that the list of default filters includes the value AggregateInstanceExtraSpecsFilter:

    NovaSchedulerDefaultFilters:
    ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','Serve
    rGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','AggregateInstanceExt
    raSpecsFilter']

    Copy to Clipboard Toggle word wrap
12. Run the overcloud_deploy.sh script.

Procedure

1. Open the NIC config file for your chosen role.

2. Add an entry for the interface type sriov_pf to configure a physical function that the host can use:

           - type: sriov_pf
               name: <interface_name>
               use_dhcp: false
               numvfs: <number_of_vfs>
               promisc: <true/false>

   Copy to Clipboard Toggle word wrap
   • Replace <interface_name> with the name of the interface.
   • Replace <number_of_vfs> with the number of VFs.
   • Optional: Replace <true/false> with true to set promiscuous mode, or false to disable promiscuous mode. The default value is true.
   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that physical function (PF). In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

3. Add an entry for the interface type sriov_vf to configure virtual functions that the host can use:

    - type: <bond_type>
      name: internal_bond
      bonding_options: mode=<bonding_option>
      use_dhcp: false
      members:
      - type: sriov_vf
          device: <pf_device_name>
          vfid: <vf_id>
      - type: sriov_vf
          device:  <pf_device_name>
          vfid: <vf_id>
   
    - type: vlan
      vlan_id:
        get_param: InternalApiNetworkVlanID
      spoofcheck: false
      device: internal_bond
      addresses:
      - ip_netmask:
          get_param: InternalApiIpSubnet
      routes:
        list_concat_unique:
        - get_param: InternalApiInterfaceRoutes

   Copy to Clipboard Toggle word wrap
   • Replace <bond_type> with the required bond type, for example, linux_bond. You can apply VLAN tags on the bond for other bonds, such as ovs_bond.

   • Replace <bonding_option> with one of the following supported bond modes:

     • active-backup

     • Balance-slb

       Note

       LACP bonds are not supported.

   • Specify the sriov_vf as the interface type to bond in the members section.

     Note

     If you are using an OVS bridge as the interface type, you can configure only one OVS bridge on the sriov_vf of a sriov_pf device. More than one OVS bridge on a single sriov_pf device can result in packet duplication across VFs, and decreased performance.

   • Replace <pf_device_name> with the name of the PF device.
   • If you use a linux_bond, you must assign VLAN tags. If you set a VLAN tag, ensure that you set a unique tag for each VF associated with a single sriov_pf device. You cannot have two VFs from the same PF on the same VLAN.
   • Replace <vf_id> with the ID of the VF. The applicable VF ID range starts at zero, and ends at the maximum number of VFs minus one.
   • Disable spoof checking.
   • Apply VLAN tags on the sriov_vf for linux_bond over VFs.

4. To reserve VFs for instances, include the NovaPCIPassthrough parameter in an environment file, for example:

   NovaPCIPassthrough:
    - address: "0000:19:0e.3"
      trusted: "true"
      physical_network: "sriov1"
    - address: "0000:19:0e.0"
      trusted: "true"
      physical_network: "sriov2"

   Copy to Clipboard Toggle word wrap

   Director identifies the host VFs, and derives the PCI addresses of the VFs that are available to the instance.

5. Enable IOMMU on all nodes that require NIC partitioning. For example, if you want NIC Partitioning for Compute nodes, enable IOMMU using the KernelArgs parameter for that role:

   parameter_defaults:
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   Copy to Clipboard Toggle word wrap
   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment.

   For more information, see Configuring manual node reboot to define KernelArgs in the Configuring the Compute Service for Instance Creation guide.

6. Add your role file and environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -r os-net-config.yaml
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

   Copy to Clipboard Toggle word wrap

Validation

1. Log in to the overcloud Compute node as heat-admin and check the number of VFs:

   [heat-admin@overcloud-compute-0 heat-admin]$ sudo cat /sys/class/net/p4p1/device/sriov_numvfs
   10
   [heat-admin@overcloud-compute-0 heat-admin]$ sudo cat /sys/class/net/p4p2/device/sriov_numvfs
   10

   Copy to Clipboard Toggle word wrap

2. Show OVS connections:

   [heat-admin@overcloud-compute-0]$ sudo ovs-vsctl show
   b6567fa8-c9ec-4247-9a08-cbf34f04c85f
       Manager "ptcp:6640:127.0.0.1"
           is_connected: true
       Bridge br-sriov2
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port phy-br-sriov2
               Interface phy-br-sriov2
                   type: patch
                   options: {peer=int-br-sriov2}
           Port br-sriov2
               Interface br-sriov2
                   type: internal
       Bridge br-sriov1
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port phy-br-sriov1
               Interface phy-br-sriov1
                   type: patch
                   options: {peer=int-br-sriov1}
           Port br-sriov1
               Interface br-sriov1
                   type: internal
       Bridge br-ex
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port br-ex
               Interface br-ex
                   type: internal
           Port phy-br-ex
               Interface phy-br-ex
                   type: patch
                   options: {peer=int-br-ex}
       Bridge br-tenant
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port br-tenant
               tag: 305
               Interface br-tenant
                   type: internal
           Port phy-br-tenant
               Interface phy-br-tenant
                   type: patch
                   options: {peer=int-br-tenant}
           Port dpdkbond0
               Interface dpdk0
                   type: dpdk
                   options: {dpdk-devargs="0000:18:0e.0"}
               Interface dpdk1
                   type: dpdk
                   options: {dpdk-devargs="0000:18:0a.0"}
       Bridge br-tun
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port vxlan-98140025
               Interface vxlan-98140025
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.37"}
           Port br-tun
               Interface br-tun
                   type: internal
           Port patch-int
               Interface patch-int
                   type: patch
                   options: {peer=patch-tun}
           Port vxlan-98140015
               Interface vxlan-98140015
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.21"}
           Port vxlan-9814009f
               Interface vxlan-9814009f
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.159"}
           Port vxlan-981400cc
               Interface vxlan-981400cc
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.204"}
       Bridge br-int
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port int-br-tenant
               Interface int-br-tenant
                   type: patch
                   options: {peer=phy-br-tenant}
           Port int-br-ex
               Interface int-br-ex
                   type: patch
                   options: {peer=phy-br-ex}
           Port int-br-sriov1
               Interface int-br-sriov1
                   type: patch
                   options: {peer=phy-br-sriov1}
           Port patch-tun
               Interface patch-tun
                   type: patch
                   options: {peer=patch-int}
           Port br-int
               Interface br-int
                   type: internal
           Port int-br-sriov2
               Interface int-br-sriov2
                   type: patch
                   options: {peer=phy-br-sriov2}
           Port vhu4142a221-93
               tag: 1
               Interface vhu4142a221-93
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu4142a221-93"}
       ovs_version: "2.13.2"

   Copy to Clipboard Toggle word wrap

3. Log in to your OVS-DPDK SR-IOV Compute node as heat-admin and check Linux bonds:

   [heat-admin@overcloud-computeovsdpdksriov-1 ~]$ cat /proc/net/bonding/<bond_name>
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: eno3v1
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   Peer Notification Delay (ms): 0
   
   Slave Interface: eno3v1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 4e:77:94:bd:38:d2
   Slave queue ID: 0
   
   Slave Interface: eno4v1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 4a:74:52:a7:aa:7c
   Slave queue ID: 0

   Copy to Clipboard Toggle word wrap

4. List OVS bonds:

   [heat-admin@overcloud-computeovsdpdksriov-1 ~]$ sudo ovs-appctl bond/show
   ---- dpdkbond0 ----
   bond_mode: balance-slb
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   next rebalance: 9491 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: ce:ee:c7:58:8e:b2(dpdk1)
   
   slave dpdk0: enabled
     may_enable: true
   
   slave dpdk1: enabled
     active slave
     may_enable: true

   Copy to Clipboard Toggle word wrap

Procedure

1. Generate an overcloud role for OVS hardware offload that is based on the Compute role:

   openstack overcloud roles generate -o roles_data.yaml \
   Controller Compute:ComputeOvsHwOffload

   Copy to Clipboard Toggle word wrap
2. Optional: Change the HostnameFormatDefault: '%stackname%-compute-%index%' name for the ComputeOvsHwOffload role.
3. Add the OvsHwOffload parameter under role-specific parameters with a value of true.
4. To configure neutron to use the iptables/hybrid firewall driver implementation, include the line: NeutronOVSFirewallDriver: iptables_hybrid. For more information about NeutronOVSFirewallDriver, see Using the Open vSwitch Firewall in the Advanced Overcloud Customization Guide.

5. Configure the physical_network parameter to match your environment.

   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Example:

     parameter_defaults:
       NeutronOVSFirewallDriver: iptables_hybrid
       ComputeSriovParameters:
         IsolCpusList: 2-9,21-29,11-19,31-39
         KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
         OvsHwOffload: true
         TunedProfileName: "cpu-partitioning"
         NeutronBridgeMappings:
           - tenant:br-tenant
         NovaPCIPassthrough:
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "tenant"
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "null"
         NovaReservedHostMemory: 4096
         NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39

     Copy to Clipboard Toggle word wrap
   • Replace <vendor-id> with the vendor ID of the physical NIC.
   • Replace <product-id> with the product ID of the NIC VF.

   • Replace <address> with the address of the physical NIC.

     For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough in the Configuring the Compute Service for Instance Creation guide.

6. Ensure that the list of default filters includes NUMATopologyFilter:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   Copy to Clipboard Toggle word wrap
   Note

   Optional: For details on how to troubleshoot and configure OVS Hardware Offload issues in RHOSP 16.2 with Mellanox ConnectX5 NICs, see Troubleshooting Hardware Offload.

7. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Copy to Clipboard Toggle word wrap
   Note
   • Do not use the NeutronSriovNumVFs parameter when configuring Open vSwitch hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as a nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.

8. Include the ovs-hw-offload.yaml file in the overcloud deploy command:

   TEMPLATES_HOME=”/usr/share/openstack-tripleo-heat-templates”
   CUSTOM_TEMPLATES=”/home/stack/templates”
   
   openstack overcloud deploy --templates \
     -r ${CUSTOM_TEMPLATES}/roles_data.yaml \
     -e ${TEMPLATES_HOME}/environments/ovs-hw-offload.yaml \
     -e ${CUSTOM_TEMPLATES}/network-environment.yaml \
     -e ${CUSTOM_TEMPLATES}/neutron-ovs.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Confirm that a PCI device is in switchdev mode:

   # devlink dev eswitch show pci/0000:03:00.0
   pci/0000:03:00.0: mode switchdev inline-mode none encap enable

   Copy to Clipboard Toggle word wrap

2. Verify if offload is enabled in OVS:

   # ovs-vsctl get Open_vSwitch . other_config:hw-offload
   “true”

   Copy to Clipboard Toggle word wrap

Procedure

1. To allocate a port from a switchdev-enabled NIC, log in as an admin user, create a neutron port with a binding-profile value of capabilities, and disable port security:

   $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 --disable-port-security

   Copy to Clipboard Toggle word wrap

2. Pass this port information when you create the instance.

   You associate the representor port with the instance VF interface and connect the representor port to OVS bridge br-int for one-time OVS data path processing. A VF port representor functions like a software version of a physical “patch panel” front-end.

   For more information about new instance creation, see Deploying an instance for SR-IOV.

Procedure

1. Use director to apply the following configuration on OVS:

   $ sudo ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

   Copy to Clipboard Toggle word wrap
2. Restart to enable HW Offload.

Procedure

1. Apply the following configuration. This is the default option if you do not explicitly configure tc-policy:

   $ sudo ovs-vsctl set Open_vSwitch . other_config:tc-policy=none

   Copy to Clipboard Toggle word wrap
2. Restart OVS.

Procedure

1. During deployment, use the host network configuration tool os-net-config to enable hw-tc-offload.
2. Enable hw-tc-offload on the sriov_config service any time you reboot the Compute node.

3. Set the hw-tc-offload parameter to on for the NICs that are attached to the bond:.

   [root@overcloud-computesriov-0 ~]# ethtool -k ens1f0 | grep tc-offload
   hw-tc-offload: on

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the eswitch mode to switchdev for the interfaces you use for HW offload.
2. Use the host network configuration tool os-net-config to enable eswitch during deployment.

3. Enable eswitch on the sriov_config service any time you reboot the Compute node.

   [root@overcloud-computesriov-0 ~]# devlink dev eswitch show pci/$(ethtool -i ens1f0 | grep bus-info | cut -d ':' -f 2,3,4 | awk '{$1=$1};1')

   Copy to Clipboard Toggle word wrap

Procedure

1. To get Broadcom chip table content, use the bcmcmd command.

   root@dni-7448-26:~# cl-bcmcmd l2 show
   
   mac=00:02:00:00:00:08 vlan=2000 GPORT=0x2 modid=0 port=2/xe1
   mac=00:02:00:00:00:09 vlan=2000 GPORT=0x2 modid=0 port=2/xe1 Hit

   Copy to Clipboard Toggle word wrap

2. Inspect the Traffic Control (TC) Layer.

   # tc -s filter show dev p5p1_1 ingress
   …
   filter block 94 protocol ip pref 3 flower chain 5
   filter block 94 protocol ip pref 3 flower chain 5 handle 0x2
     eth_type ipv4
     src_ip 172.0.0.1
     ip_flags nofrag
     in_hw in_hw_count 1
           action order 1: mirred (Egress Redirect to device eth4) stolen
           index 3 ref 1 bind 1 installed 364 sec used 0 sec
           Action statistics:
           Sent 253991716224 bytes 169534118 pkt (dropped 0, overlimits 0 requeues 0)
           Sent software 43711874200 bytes 30161170 pkt
           Sent hardware 210279842024 bytes 139372948 pkt
           backlog 0b 0p requeues 0
           cookie 8beddad9a0430f0457e7e78db6e0af48
           no_percpu

   Copy to Clipboard Toggle word wrap
3. Examine the in_hw flags and the statistics in this output. The word hardware indicates that the hardware processes the network traffic. If you use tc-policy=none, you can check this output or a tcpdump to investigate when hardware or software handles the packets. You can see a corresponding log message in dmesg or in ovs-vswitch.log when the driver is unable to offload packets.

4. For Mellanox, as an example, the log entries resemble syndrome messages in dmesg.

   [13232.860484] mlx5_core 0000:3b:00.0: mlx5_cmd_check:756:(pid 131368): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x6b1266)

   Copy to Clipboard Toggle word wrap

   In this example, the error code (0x6b1266) represents the following behavior:

   0x6B1266 |  set_flow_table_entry: pop vlan and forward to uplink is not allowed

   Copy to Clipboard Toggle word wrap

Procedure

1. Ensure SR-IOV and VT-d are enabled on the system.
2. Enable IOMMU in Linux by adding intel_iommu=on to kernel parameters, for example, using GRUB.

Procedure

1. To enable logging on the offload modules and to get additional log information for this failure, use the following commands on the Compute node:

   ovs-appctl vlog/set dpif_netlink:file:dbg
   # Module name changed recently (check based on the version used
   ovs-appctl vlog/set netdev_tc_offloads:file:dbg [OR] ovs-appctl vlog/set netdev_offload_tc:file:dbg
   ovs-appctl vlog/set tc:file:dbg

   Copy to Clipboard Toggle word wrap

2. Inspect the ovs-vswitchd logs again to see additional details about the issue.

   In the following example logs, the offload failed because of an unsupported attribute mark.

    2020-01-31T06:22:11.218Z|00471|dpif_netlink(handler402)|DBG|system@ovs-system: put[create] ufid:61bd016e-eb89-44fc-a17e-958bc8e45fda recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(7),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d2:f5:f3,dst=fa:16:3e:c4:a3:eb),eth_type(0x0800),ipv4(src=10.1.1.8/0.0.0.0,dst=10.1.1.31/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no),icmp(type=0/0,code=0/0), actions:set(tunnel(tun_id=0x3d,src=10.10.141.107,dst=10.10.141.124,ttl=64,tp_dst=4789,flags(df|key))),6
   
   2020-01-31T06:22:11.253Z|00472|netdev_tc_offloads(handler402)|DBG|offloading attribute pkt_mark isn't supported
   
   2020-01-31T06:22:11.257Z|00473|dpif_netlink(handler402)|ERR|failed to offload flow: Operation not supported: p6p1_5

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a flavor.

   $ openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

   Copy to Clipboard Toggle word wrap
   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in the Configuring the Compute Service for Instance Creation guide.

2. Create the network.

   $ openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   $ openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

   Copy to Clipboard Toggle word wrap

3. Create the port.

   • Use vnic-type direct to create an SR-IOV virtual function (VF) port.

     $ openstack port create --network net1 --vnic-type direct sriov_port

     Copy to Clipboard Toggle word wrap

   • Use the following command to create a virtual function with hardware offload. You must be an admin user to set --binding-profile.

     $ openstack port create --network net1 --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]} sriov_hwoffload_port

     Copy to Clipboard Toggle word wrap

   • Use vnic-type direct-physical to create an SR-IOV physical function (PF) port that is dedicated to a single instance. This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     $ openstack port create --network net1 --vnic-type direct-physical sriov_port

     Copy to Clipboard Toggle word wrap

4. Deploy an instance.

   $ openstack server create --flavor <flavor> --image <image> --nic port-id=<id> <instance name>

   Copy to Clipboard Toggle word wrap

Procedure

1. You can configure the AggregateInstanceExtraSpecsFilter value, and other necessary filters, through the heat parameter NovaSchedulerEnabledFilters under parameter_defaults in your deployment templates.

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AggregateInstanceExtraSpecsFilter
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   Copy to Clipboard Toggle word wrap
   Note

   To add this parameter to the configuration of an existing cluster, you can add it to the heat templates, and run the original deployment script again.

2. Create an aggregate group for SR-IOV, and add relevant hosts. Define metadata, for example, sriov=true, that matches defined flavor metadata.

   # openstack aggregate create sriov_group
   # openstack aggregate add host sriov_group compute-sriov-0.localdomain
   # openstack aggregate set --property sriov=true sriov_group

   Copy to Clipboard Toggle word wrap

3. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

   Copy to Clipboard Toggle word wrap

4. Set additional flavor properties. Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   # openstack flavor set --property sriov=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB <flavor>

   Copy to Clipboard Toggle word wrap