Network Functions Virtualization Planning and Configuration Guide

Procedure

1. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted.

   [stack@director ~]$ sudo subscription-manager register

2. Determine the entitlement pool ID for Red Hat OpenStack Platform director, for example {Pool ID} from the following command and output:

   [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
                        Red Hat Virtualization
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             {Pool-ID}-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

3. Include the Pool ID value in the following command to attach the Red Hat OpenStack Platform 16.1 entitlement.

   [stack@director ~]$ sudo subscription-manager attach --pool={Pool-ID}-123456

4. Disable the default repositories.

   subscription-manager repos --disable=*

5. Enable the required repositories for Red Hat OpenStack Platform with NFV.

   $ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms \
   --enable=rhel-8-for-x86_64-appstream-eus-rpms \
   --enable=rhel-8-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.9-for-rhel-8-x86_64-rpms \
   --enable=openstack-16.1-for-rhel-8-x86_64-rpms \
   --enable=rhel-8-for-x86_64-nfv-rpms \
   --enable=advanced-virt-for-rhel-8-x86_64-rpms \
   --enable=fast-datapath-for-rhel-8-x86_64-rpms

6. Update your system so you have the latest base system packages.

   [stack@director ~]$ sudo dnf update -y
   [stack@director ~]$ sudo reboot

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_compute_sriov.yaml that includes the Controller and ComputeSriov roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_compute_sriov.yaml \
    Controller ComputeSriov

   ComputeSriov is a custom role provided with your RHOSP installation that includes the NeutronSriovAgent, NeutronSriovHostConfig services, in addition to the default compute services.

4. To prepare the SR-IOV containers, include the neutron-sriov.yaml and roles_data_compute_sriov.yaml files when you generate the overcloud_images.yaml file.

   $ sudo openstack tripleo container image prepare \
     --roles-file ~/templates/roles_data_compute_sriov.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml \
     -e ~/containers-prepare-parameter.yaml \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   For more information on container image preparation, see Preparing container images in the Director Installation and Usage guide.

5. Create a copy of the /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml file in your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml /home/stack/templates/network-environment-sriov.yaml

6. Add the following parameters under parameter_defaults in your network-environment-sriov.yaml file to configure the SR-IOV nodes for your cluster and your hardware configuration:

     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

7. To determine the vendor_id and product_id for each PCI device type, use one of the following commands on the physical server that has the PCI cards:

   • To return the vendor_id and product_id from a deployed overcloud, use the following command:

     # lspci -nn -s  <pci_device_address>
     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

   • To return the vendor_id and product_id of a physical function (PF) if you have not yet deployed the overcloud, use the following command:

     (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

8. Configure role specific parameters for SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       IsolCpusList: "1-19,21-39"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-19,21-39"
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysicalDevMappings:
         - tenant:p7p1
       NovaComputeCpuDedicatedSet: '1-19,21-39'
       NovaReservedHostMemory: 4096

   Note

   The NovaVcpuPinSet parameter is now deprecated, and is replaced by NovaComputeCpuDedicatedSet for dedicated, pinned workloads.

9. Configure the PCI passthrough devices for the SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       ...
       NovaPCIPassthrough:
         - vendor_id: "<vendor_id>"
           product_id: "<product_id>"
           address: <NIC_address>
           physical_network: "<physical_network>"
       ...

   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device. For information about how to configure the address parameter, see Guidelines for configuring NovaPCIPassthrough in the Configuring the Compute Service for Instance Creation guide.

   • Replace <physical_network> with the name of the physical network the PCI device is located on.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts. For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough in the Configuring the Compute Service for Instance Creation guide.

10. Configure the SR-IOV enabled interfaces in the compute.yaml network configuration template. To create SR-IOV VFs, configure the interfaces as standalone NICs:

                 - type: sriov_pf
                    name: p7p3
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false
    
                  - type: sriov_pf
                    name: p7p4
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false

    Note

    The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

11. Ensure that the list of default filters includes the value AggregateInstanceExtraSpecsFilter:

    NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','AggregateInstanceExtraSpecsFilter']

12. Run the overcloud_deploy.sh script.

Procedure

1. Open the NIC config file for your chosen role.

2. Add an entry for the interface type sriov_pf to configure a physical function that the host can use:

           - type: sriov_pf
               name: <interface name>
               use_dhcp: false
               numvfs: <number of vfs>
               promisc: <true/false> #optional (Defaults to true)

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that physical function (PF). In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

3. Add an entry for the interface type sriov_vf to configure virtual functions that the host can use:

    - type: <bond_type>
      name: internal_bond
      bonding_options: mode=<bonding_option>
      use_dhcp: false
      members:
      - type: sriov_vf
          device: <pf_device_name>
          vfid: <vf_id>
      - type: sriov_vf
          device:  <pf_device_name>
          vfid: <vf_id>
   
    - type: vlan
      vlan_id:
        get_param: InternalApiNetworkVlanID
      spoofcheck: false
      device: internal_bond
      addresses:
      - ip_netmask:
          get_param: InternalApiIpSubnet
      routes:
        list_concat_unique:
        - get_param: InternalApiInterfaceRoutes

   • Replace <bond_type> with the required bond type, for example, linux_bond. You can apply VLAN tags on the bond for other bonds, such as ovs_bond.

   • Replace <bonding_option> with one of the following supported bond modes:

     • active-backup

     • Balance-slb

       Note

       LACP bonds are not supported.

   • Specify the sriov_vf as the interface type to bond in the members section.

     Note

     If you are using an OVS bridge as the interface type, you can configure only one OVS bridge on the sriov_vf of a sriov_pf device. More than one OVS bridge on a single sriov_pf device can result in packet duplication across VFs, and decreased performance.

   • Replace <pf_device_name> with the name of the PF device.
   • If you use a linux_bond, you must assign VLAN tags.
   • Replace <vf_id> with the ID of the VF. The applicable VF ID range starts at zero, and ends at the maximum number of VFs minus one.
4. Disable spoof checking, and apply VLAN tags on the sriov_vf for linux_bond over VFs.

5. To reserve VFs for instances, include the NovaPCIPassthrough parameter in an environment file, for example:

   NovaPCIPassthrough:
    - address: "0000:19:0e.3"
      trusted: "true"
      physical_network: "sriov1"
    - address: "0000:19:0e.0"
      trusted: "true"
      physical_network: "sriov2"

   Director identifies the host VFs, and derives the PCI addresses of the VFs that are available to the instance.

6. Enable IOMMU on all nodes that require NIC partitioning. For example, if you want NIC Partitioning for Compute nodes, enable IOMMU using the KernelArgs parameter for that role.

   parameter_defaults:
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

7. Add your role file and environment files to the stack with your other environment files and deploy the overcloud:
   

   (undercloud)$ openstack overcloud deploy --templates \
     -r os-net-config.yaml
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Validation

1. Check the number of VFs.

   [root@overcloud-compute-0 heat-admin]# cat /sys/class/net/p4p1/device/sriov_numvfs
   10
   [root@overcloud-compute-0 heat-admin]# cat /sys/class/net/p4p2/device/sriov_numvfs
   10

2. Check Linux bonds.

   [root@overcloud-compute-0 heat-admin]# cat /proc/net/bonding/intapi_bond
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: p4p1_1
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   
   Slave Interface: p4p1_1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 16:b4:4c:aa:f0:a8
   Slave queue ID: 0
   
   Slave Interface: p4p2_1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: b6:be:82:ac:51:98
   Slave queue ID: 0
   [root@overcloud-compute-0 heat-admin]# cat /proc/net/bonding/st_bond
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: p4p1_3
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   
   Slave Interface: p4p1_3
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 9a:86:b7:cc:17:e4
   Slave queue ID: 0
   
   Slave Interface: p4p2_3
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: d6:07:f8:78:dd:5b
   Slave queue ID: 0

3. List OVS bonds.

   [root@overcloud-compute-0 heat-admin]# ovs-appctl bond/show
   ---- bond_prov ----
   bond_mode: active-backup
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: f2:ad:c7:00:f5:c7(dpdk2)
   
   slave dpdk2: enabled
     active slave
     may_enable: true
   
   slave dpdk3: enabled
     may_enable: true
   
   ---- bond_tnt ----
   bond_mode: active-backup
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: b2:7e:b8:75:72:e8(dpdk0)
   
   slave dpdk0: enabled
     active slave
     may_enable: true
   
   slave dpdk1: enabled
     may_enable: true

4. Show OVS connections.

   [root@overcloud-compute-0 heat-admin]# ovs-vsctl show
   cec12069-9d4c-4fa8-bfe4-decfdf258f49
       Manager "ptcp:6640:127.0.0.1"
           is_connected: true
       Bridge br-tenant
           fail_mode: standalone
           Port br-tenant
               Interface br-tenant
                   type: internal
           Port bond_tnt
               Interface "dpdk0"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:02.2"}
               Interface "dpdk1"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:04.2"}
       Bridge "sriov2"
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "phy-sriov2"
               Interface "phy-sriov2"
                   type: patch
                   options: {peer="int-sriov2"}
           Port "sriov2"
               Interface "sriov2"
                   type: internal
       Bridge br-int
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "int-sriov2"
               Interface "int-sriov2"
                   type: patch
                   options: {peer="phy-sriov2"}
           Port br-int
               Interface br-int
                   type: internal
           Port "vhu93164679-22"
               tag: 4
               Interface "vhu93164679-22"
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu93164679-22"}
           Port "vhu5d6b9f5a-0d"
               tag: 3
               Interface "vhu5d6b9f5a-0d"
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu5d6b9f5a-0d"}
           Port patch-tun
               Interface patch-tun
                   type: patch
                   options: {peer=patch-int}
           Port "int-sriov1"
               Interface "int-sriov1"
                   type: patch
                   options: {peer="phy-sriov1"}
           Port int-br-vfs
               Interface int-br-vfs
                   type: patch
                   options: {peer=phy-br-vfs}
       Bridge br-vfs
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port phy-br-vfs
               Interface phy-br-vfs
                   type: patch
                   options: {peer=int-br-vfs}
           Port bond_prov
               Interface "dpdk3"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:04.5"}
               Interface "dpdk2"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:02.5"}
           Port br-vfs
               Interface br-vfs
                   type: internal
       Bridge "sriov1"
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "sriov1"
               Interface "sriov1"
                   type: internal
           Port "phy-sriov1"
               Interface "phy-sriov1"
                   type: patch
                   options: {peer="int-sriov1"}
       Bridge br-tun
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port br-tun
               Interface br-tun
                   type: internal
           Port patch-int
               Interface patch-int
                   type: patch
                   options: {peer=patch-tun}
           Port "vxlan-0a0a7315"
               Interface "vxlan-0a0a7315"
                   type: vxlan
                   options: {df_default="true", in_key=flow, local_ip="10.10.115.10", out_key=flow, remote_ip="10.10.115.21"}
       ovs_version: "2.10.0"

Procedure

1. Generate an overcloud role for OVS hardware offload that is based on the Compute role:

   openstack overcloud roles generate -o roles_data.yaml Controller Compute:ComputeOvsHwOffload

2. Optional: Change the HostnameFormatDefault: '%stackname%-compute-%index%' name for the ComputeOvsHwOffload role.
3. Add the OvsHwOffload parameter under role-specific parameters with a value of true.
4. To configure neutron to use the iptables/hybrid firewall driver implementation, include the line: NeutronOVSFirewallDriver: iptables_hybrid. For more information about NeutronOVSFirewallDriver, see Using the Open vSwitch Firewall in the Advanced Overcloud Customization Guide.

5. Configure the physical_network parameter to match your environment.

   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Example:

     parameter_defaults:
       NeutronOVSFirewallDriver: iptables_hybrid
       ComputeSriovParameters:
         IsolCpusList: 2-9,21-29,11-19,31-39
         KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
         OvsHwOffload: true
         TunedProfileName: "cpu-partitioning"
         NeutronBridgeMappings:
           - tenant:br-tenant
         NovaPCIPassthrough:
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "tenant"
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "null"
         NovaReservedHostMemory: 4096
         NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39

   • Replace <vendor-id> with the vendor ID of the physical NIC.
   • Replace <product-id> with the product ID of the NIC VF.

   • Replace <address> with the address of the physical NIC.

     For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough.

6. Ensure that the list of default filters includes NUMATopologyFilter:

     NovaSchedulerDefaultFilters: [\'AvailabilityZoneFilter',\'ComputeFilter',\'ComputeCapabilitiesFilter',\'ImagePropertiesFilter',\'ServerGroupAntiAffinityFilter',\'ServerGroupAffinityFilter',\'PciPassthroughFilter',\'NUMATopologyFilter']

7. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Note
   • Do not use the NeutronSriovNumVFs parameter when configuring Open vSwitch hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as a nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.

8. Include the ovs-hw-offload.yaml file in the overcloud deploy command:

   TEMPLATES_HOME=”/usr/share/openstack-tripleo-heat-templates”
   CUSTOM_TEMPLATES=”/home/stack/templates”
   
   openstack overcloud deploy --templates \
     -r ${CUSTOM_TEMPLATES}/roles_data.yaml \
     -e ${TEMPLATES_HOME}/environments/ovs-hw-offload.yaml \
     -e ${CUSTOM_TEMPLATES}/network-environment.yaml \
     -e ${CUSTOM_TEMPLATES}/neutron-ovs.yaml

Procedure

1. To allocate a port from a switchdev-enabled NIC, create a neutron port with a binding-profile value of capabilities, and disable port security:

   $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 --disable-port-security

Procedure

1. Use director to apply the following configuration on OVS:

   $ sudo ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

2. Restart to enable HW Offload.

Procedure

1. Apply the following configuration. This is the default option if you do not explicitly configure tc-policy:

   $ sudo ovs-vsctl set Open_vSwitch . other_config:tc-policy=none

2. Restart OVS.

Procedure

1. During deployment, use the host network configuration tool os-net-config to enable hw-tc-offload.
2. Enable hw-tc-offload on the sriov_config service any time you reboot the Compute node.

3. Set the hw-tc-offload parameter to on for the NICs that are attached to the bond:.

   [root@overcloud-computesriov-0 ~]# ethtool -k ens1f0 | grep tc-offload
   hw-tc-offload: on

Procedure

1. Set the eswitch mode to switchdev for the interfaces you use for HW offload.
2. Use the host network configuration tool os-net-config to enable eswitch during deployment.

3. Enable eswitch on the sriov_config service any time you reboot the Compute node.

   [root@overcloud-computesriov-0 ~]# devlink dev eswitch show pci/$(ethtool -i ens1f0 | grep bus-info | cut -d ':' -f 2,3,4 | awk '{$1=$1};1')

Procedure

1. To get Broadcom chip table content, use the bcmcmd command.

   root@dni-7448-26:~# cl-bcmcmd l2 show
   
   mac=00:02:00:00:00:08 vlan=2000 GPORT=0x2 modid=0 port=2/xe1
   mac=00:02:00:00:00:09 vlan=2000 GPORT=0x2 modid=0 port=2/xe1 Hit

2. Inspect the Traffic Control (TC) Layer.

   # tc -s filter show dev p5p1_1 ingress
   …
   filter block 94 protocol ip pref 3 flower chain 5
   filter block 94 protocol ip pref 3 flower chain 5 handle 0x2
     eth_type ipv4
     src_ip 172.0.0.1
     ip_flags nofrag
     in_hw in_hw_count 1
           action order 1: mirred (Egress Redirect to device eth4) stolen
           index 3 ref 1 bind 1 installed 364 sec used 0 sec
           Action statistics:
           Sent 253991716224 bytes 169534118 pkt (dropped 0, overlimits 0 requeues 0)
           Sent software 43711874200 bytes 30161170 pkt
           Sent hardware 210279842024 bytes 139372948 pkt
           backlog 0b 0p requeues 0
           cookie 8beddad9a0430f0457e7e78db6e0af48
           no_percpu

3. Examine the in_hw flags and the statistics in this output. The word hardware indicates that the hardware processes the network traffic. If you use tc-policy=none, you can check this output or a tcpdump to investigate when hardware or software handles the packets. You can see a corresponding log message in dmesg or in ovs-vswitch.log when the driver is unable to offload packets.

4. For Mellanox, as an example, the log entries resemble syndrome messages in dmesg.

   [13232.860484] mlx5_core 0000:3b:00.0: mlx5_cmd_check:756:(pid 131368): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x6b1266)

   In this example, the error code (0x6b1266) represents the following behavior:

   0x6B1266 |  set_flow_table_entry: pop vlan and forward to uplink is not allowed

Procedure

1. Ensure SR-IOV and VT-d are enabled on the system.
2. Enable IOMMU in Linux by adding intel_iommu=on to kernel parameters, for example, using GRUB.

Procedure

1. To enable logging on the offload modules and to get additional log information for this failure, use the following commands on the Compute node:

   ovs-appctl vlog/set dpif_netlink:file:dbg
   # Module name changed recently (check based on the version used
   ovs-appctl vlog/set netdev_tc_offloads:file:dbg [OR] ovs-appctl vlog/set netdev_offload_tc:file:dbg
   ovs-appctl vlog/set tc:file:dbg

2. Inspect the ovs-vswitchd logs again to see additional details about the issue.

   In the following example logs, the offload failed because of an unsupported attribute mark.

    2020-01-31T06:22:11.218Z|00471|dpif_netlink(handler402)|DBG|system@ovs-system: put[create] ufid:61bd016e-eb89-44fc-a17e-958bc8e45fda recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(7),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d2:f5:f3,dst=fa:16:3e:c4:a3:eb),eth_type(0x0800),ipv4(src=10.1.1.8/0.0.0.0,dst=10.1.1.31/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no),icmp(type=0/0,code=0/0), actions:set(tunnel(tun_id=0x3d,src=10.10.141.107,dst=10.10.141.124,ttl=64,tp_dst=4789,flags(df|key))),6
   
   2020-01-31T06:22:11.253Z|00472|netdev_tc_offloads(handler402)|DBG|offloading attribute pkt_mark isn't supported
   
   2020-01-31T06:22:11.257Z|00473|dpif_netlink(handler402)|ERR|failed to offload flow: Operation not supported: p6p1_5

1. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in the Configuring the Compute Service for Instance Creation guide.

2. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

3. Create the port.

   • Use vnic-type direct to create an SR-IOV virtual function (VF) port.

     # openstack port create --network net1 --vnic-type direct sriov_port

   • Use the following command to create a virtual function with hardware offload.

     # openstack port create --network net1 --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]} sriov_hwoffload_port

   • Use vnic-type direct-physical to create an SR-IOV physical function (PF) port that is dedicated to a single instance. This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     # openstack port create --network net1 --vnic-type direct-physical sriov_port

4. Deploy an instance.

   # openstack server create --flavor <flavor> --image <image> --nic port-id=<id> <instance name>

1. You can configure the AggregateInstanceExtraSpecsFilter value, and other necessary filters, through the heat parameter NovaSchedulerDefaultFilters under parameter_defaults in your deployment templates.

     parameter_defaults:
       NovaSchedulerDefaultFilters: ['AggregateInstanceExtraSpecsFilter','AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

   Note

   To add this parameter to the configuration of an exiting cluster, you can add it to the heat templates, and run the original deployment script again.

2. Create an aggregate group for SR-IOV, and add relevant hosts. Define metadata, for example, sriov=true, that matches defined flavor metadata.

   # openstack aggregate create sriov_group
   # openstack aggregate add host sriov_group compute-sriov-0.localdomain
   # openstack aggregate set --property sriov=true sriov_group

3. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

4. Set additional flavor properties. Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   # openstack flavor set --property sriov=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB <flavor>

1. Copy the usr/share/openstack-tripleo-heat-templates/plan-samples/plan-environment-derived-params.yaml file to a local directory and set the input parameters to suit your environment.

     workflow_parameters:
       tripleo.derive_params.v1.derive_parameters:
         # DPDK Parameters #
         # Specifies the minimum number of CPU physical cores to be allocated for DPDK
         # PMD threads. The actual allocation will be based on network config, if
         # the a DPDK port is associated with a numa node, then this configuration
         # will be used, else 1.
         num_phy_cores_per_numa_node_for_pmd: 1
         # Amount of memory to be configured as huge pages in percentage. Ouf the
         # total available memory (excluding the NovaReservedHostMemory), the
         # specified percentage of the remaining is configured as huge pages.
         huge_page_allocation_percentage: 50

2. Run the openstack overcloud deploy command and include the following information:

   • The update-plan-only option
   • The role file and all environment files specific to your environment

   • The plan-environment-derived-parms.yaml file with the --plan-environment-file optional argument

     $ openstack overcloud deploy --templates --update-plan-only \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/<environment-file> \
     ... _#repeat as necessary_ ...
     **-p /home/stack/plan-environment-derived-params.yaml**

1. Copy the derived parameters from the deploy command output to the network-environment.yaml file.

     # DPDK compute node.
     ComputeOvsDpdkParameters:
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "1,2,3,4,5,6,7,9,10,17,18,19,20,21,22,23,11,12,13,14,15,25,26,27,28,29,30,31"
       NovaComputeCpuDedicatedSet: ['2,3,4,5,6,7,18,19,20,21,22,23,10,11,12,13,14,15,26,27,28,29,30,31']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "1024,1024"
       OvsDpdkMemoryChannels: "4"
       OvsPmdCoreList: "1,17,9,25"

   Note

   These parameters apply to the specific role, ComputeOvsDpdk. You can apply these parameters globally, but role-specific parameters overwrite any global parameters.

2. Deploy the overcloud using the role file and all environment files specific to your environment.

1. Set the NeutronGlobalPhysnetMtu parameter in the network-environment.yaml file.

   parameter_defaults:
     # MTU global configuration
     NeutronGlobalPhysnetMtu: 9000

   Note

   Ensure that the OvsDpdkSocketMemory value in the network-environment.yaml file is large enough to support jumbo frames. For details, see Section 7.4.2, “Memory parameters” .

2. Set the MTU value on the bridge to the Compute node in the controller.yaml file.

     -
       type: ovs_bridge
       name: br-link0
       use_dhcp: false
       members:
         -
           type: interface
           name: nic3
           mtu: 9000

3. Set the MTU values for an OVS-DPDK bond in the compute.yaml file:

   - type: ovs_user_bridge
     name: br-link0
     use_dhcp: false
     members:
       - type: ovs_dpdk_bond
         name: dpdkbond0
         mtu: 9000
         rx_queue: 2
         members:
           - type: ovs_dpdk_port
             name: dpdk0
             mtu: 9000
             members:
               - type: interface
                 name: nic4
           - type: ovs_dpdk_port
             name: dpdk1
             mtu: 9000
             members:
               - type: interface
                 name: nic5

1. Create an aggregate group, and add relevant hosts for OVS-DPDK. Define metadata, for example dpdk=true, that matches defined flavor metadata.

    # openstack aggregate create dpdk_group
    # openstack aggregate add host dpdk_group [compute-host]
    # openstack aggregate set --property dpdk=true dpdk_group

   Note

   Pinned CPU instances can be located on the same Compute node as unpinned instances. For more information, see Configuring CPU pinning on the Compute node in the Configuring the Compute Service for Instance Creation guide.

2. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

3. Set flavor properties. Note that the defined metadata, dpdk=true, matches the defined metadata in the DPDK aggregate.

   # openstack flavor set <flavor> --property dpdk=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB --property hw:emulator_threads_policy=isolate

   For details about the emulator threads policy for performance improvements, see Configuring emulator threads.

4. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

5. Optional: If you use multiqueue with OVS-DPDK, set the hw_vif_multiqueue_enabled property on the image that you want to use to create a instance:

   # openstack image set --property hw_vif_multiqueue_enabled=true <image>

6. Deploy an instance.

   # openstack server create --flavor <flavor> --image <glance image> --nic net-id=<network ID> <server_name>

1. Review the bridge configuration, and confirm that the bridge has datapath_type=netdev.

   # ovs-vsctl list bridge br0
   _uuid               : bdce0825-e263-4d15-b256-f01222df96f3
   auto_attach         : []
   controller          : []
   datapath_id         : "00002608cebd154d"
   datapath_type       : netdev
   datapath_version    : "<built-in>"
   external_ids        : {}
   fail_mode           : []
   flood_vlans         : []
   flow_tables         : {}
   ipfix               : []
   mcast_snooping_enable: false
   mirrors             : []
   name                : "br0"
   netflow             : []
   other_config        : {}
   ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
   protocols           : []
   rstp_enable         : false
   rstp_status         : {}
   sflow               : []
   status              : {}
   stp_enable          : false

2. Optionally, you can view logs for errors, such as if the container fails to start.

   # less /var/log/containers/neutron/openvswitch-agent.log

3. Confirm that the Poll Mode Driver CPU mask of the ovs-dpdk is pinned to the CPUs. In case of hyper threading, use sibling CPUs.

   For example, to check the sibling of CPU4, run the following command:

   # cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
   4,20

   The sibling of CPU4 is CPU20, therefore proceed with the following command:

   # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

   Display the status:

   # tuna -t ovs-vswitchd -CP
   thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
   3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
   3219   OTHER 	0    	6     	1        	0   	handler24
   3220   OTHER 	0    	6     	1        	0   	handler21
   3221   OTHER 	0    	6     	1        	0   	handler22
   3222   OTHER 	0    	6     	1        	0   	handler23
   3223   OTHER 	0    	6     	1        	0   	handler25
   3224   OTHER 	0    	6     	1        	0   	handler26
   3225   OTHER 	0    	6     	1        	0   	handler27
   3226   OTHER 	0    	6     	1        	0   	handler28
   3227   OTHER 	0    	6     	2        	0   	handler31
   3228   OTHER 	0    	6     	2        	4   	handler30
   3229   OTHER 	0    	6     	2        	5   	handler32
   3230   OTHER 	0    	6	953538      	431   revalidator29
   3231   OTHER 	0    	6   1424258      	976   revalidator33
   3232   OTHER 	0    	6   1424693      	836   revalidator34
   3233   OTHER 	0    	6	951678      	503   revalidator36
   3234   OTHER 	0    	6   1425128      	498   revalidator35
   *3235   OTHER 	0    	4	151123       	51       	pmd37*
   *3236   OTHER 	0   	20	298967       	48       	pmd38*
   3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
   3165   OTHER 	0    	6	237634        	0   vhost_thread1
   3166   OTHER 	0    	6  	3665        	0       	urcu2

1. Install ceph-ansible on the undercloud.

   $ sudo yum install ceph-ansible -y

2. Generate the roles_data.yaml file for the ComputeHCI role.

   $ openstack overcloud roles generate -o ~/<templates>/roles_data.yaml Controller \
    ComputeHCIOvsDpdk

3. Create and configure a new flavor with the openstack flavor create and openstack flavor set commands. For more information about creating a flavor, see Creating a new role in the Advanced Overcloud Customization Guide.

4. Deploy the overcloud with the custom roles_data.yaml file that you generated.

   # time openstack overcloud deploy --templates \
    --timeout 360 \
    -r ~/<templates>/roles_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/neutron-ovs-dpdk.yaml \
    -e ~/<templates>/<custom environment file>

1. Add the custom resources for OVS-DPDK under resource_registry:

     resource_registry:
       # Specify the relative/absolute path to the config files you want to use for override the default.
       OS::TripleO::ComputeOvsDpdkSriov::Net::SoftwareConfig: nic-configs/computeovsdpdksriov.yaml
       OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml

2. Under parameter_defaults, set the tunnel type to vxlan, and the network type to vxlan,vlan:

   NeutronTunnelTypes: 'vxlan'
   NeutronNetworkType: 'vxlan,vlan'

3. Under parameters_defaults, set the bridge mapping:

   # The OVS logical->physical bridge mappings to use.
   NeutronBridgeMappings:
     - dpdk-mgmt:br-link0

4. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role:

     ##########################
     # OVS DPDK configuration #
     ##########################
     ComputeOvsDpdkSriovParameters:
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2-19,22-39"
       NovaComputeCpuDedicatedSet: ['4-19,24-39']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "3072,1024"
       OvsDpdkMemoryChannels: "4"
       OvsPmdCoreList: "2,22,3,23"
       NovaComputeCpuSharedSet: [0,20,1,21]
       NovaLibvirtRxQueueSize: 1024
       NovaLibvirtTxQueueSize: 1024

   Note

   To prevent failures during guest creation, assign at least one CPU with sibling thread on each NUMA node. In the example, the values for the OvsPmdCoreList parameter denote cores 2 and 22 from NUMA 0, and cores 3 and 23 from NUMA 1.

   Note

   These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the virtual machines is the boot parameter minus the OvsDpdkSocketMemory.

   You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

   Note

   OvsDpdkMemoryChannels is a required setting for this procedure. For optimum operation, ensure you deploy DPDK with appropriate parameters and values.

5. Configure the role-specific parameters for SR-IOV:

     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.0"
         trusted: "true"
         physical_network: "sriov-1"
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.1"
         trusted: "true"
         physical_network: "sriov-2"

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic2
         primary: true

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageMgmtNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageMgmtIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: ExternalNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: ExternalIpSubnet
       routes:
       - default: true
         next_hop:
           get_param: ExternalInterfaceDefaultRoute

3. Create the OVS bridge to access neutron-dhcp-agent and neutron-metadata-agent services.

     - type: ovs_bridge
       name: br-link0
       use_dhcp: false
       mtu: 9000
       members:
       - type: interface
         name: nic3
         mtu: 9000
       - type: vlan
         vlan_id:
           get_param: TenantNetworkVlanID
         mtu: 9000
         addresses:
         - ip_netmask:
             get_param: TenantIpSubnet

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic3
         primary: true
       - type: interface
         name: nic4

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet

3. Set a bridge with a DPDK port to link to the controller.

     - type: ovs_user_bridge
       name: br-link0
       use_dhcp: false
       ovs_extra:
         - str_replace:
             template: set port br-link0 tag=_VLAN_TAG_
             params:
               _VLAN_TAG_:
                  get_param: TenantNetworkVlanID
       addresses:
         - ip_netmask:
             get_param: TenantIpSubnet
       members:
         - type: ovs_dpdk_bond
           name: dpdkbond0
           mtu: 9000
           rx_queue: 2
           members:
             - type: ovs_dpdk_port
               name: dpdk0
               members:
                 - type: interface
                   name: nic7
             - type: ovs_dpdk_port
               name: dpdk1
               members:
                 - type: interface
                   name: nic8

   Note

   To include multiple DPDK devices, repeat the type code section for each DPDK device that you want to add.

   Note

   When using OVS-DPDK, all bridges on the same Compute node must be of type ovs_user_bridge. Red Hat OpenStack Platform does not support both ovs_bridge and ovs_user_bridge located on the same node.

1. Run the overcloud_deploy.sh script: