Network Functions Virtualization Planning and Configuration Guide

Procedure

1. To retrieve the information listed above, substitute <UUID> with the UUID of the bare-metal node to complete the following command:

Procedure

1. Register your system with the Content Delivery Network, and enter your Customer Portal user name and password when prompted.

   [stack@director ~]$ sudo subscription-manager register

2. Determine the entitlement Pool ID for Red Hat OpenStack Platform director. For example:

   [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
                        Red Hat Virtualization
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             {Pool-ID}-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

3. Include the Pool ID value in the following command to attach the Red Hat OpenStack Platform 15 entitlement.

   [stack@director ~]$ sudo subscription-manager attach --pool={Pool-ID}-123456

4. Disable the default repositories.

   subscription-manager repos --disable=*

5. Enable the required repositories for Red Hat OpenStack Platform with NFV.

   $ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms \
   --enable=rhel-8-for-x86_64-appstream-rpms \
   --enable=rhel-8-for-x86_64-highavailability-rpms \
   --enable=ansible-2.8-for-rhel-8-x86_64-rpms \
   --enable=openstack-15-for-rhel-8-x86_64-rpms \
   --enable=rhel-8-for-x86_64-nfv-rpms \
   --enable=advanced-virt-for-rhel-8-x86_64-rpms \
   --enable=fast-datapath-for-rhel-8-x86_64-rpms

6. Perform an update on your system for the latest base system packages.

   [stack@director ~]$ sudo dnf update -y
   [stack@director ~]$ sudo reboot

Procedure

1. Set the neutron_driver parameter to null in the containers-prepare-parameter.yaml file.

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
        neutron_driver: openvswitch
        ...

2. Include the neutron-ovs.yaml environment file in the /usr/share/openstack-tripleo-heat-templates/environments/services directory with your deployment script.

   TEMPLATES=/usr/share/openstack-tripleo-heat-templates
   
   openstack overcloud deploy --templates \
   -e ${TEMPLATES}/environments/network-environment.yaml \
   -e ${TEMPLATES}/environments/network-isolation.yaml \
   -e ${TEMPLATES}/environments/host-config-and-reboot.yaml \
   -e ${TEMPLATES}/environments/services/neutron-ovs.yaml \
   -e ${TEMPLATES}/environments/services/neutron-ovs-dpdk.yaml \
   -e ${TEMPLATES}/environments/services/neutron-sriov.yaml \
   -e /home/stack/containers-prepare-parameter.yaml

1. Generate the built-in ComputeSriov to define nodes in the OpenStack cluster that run NeutronSriovAgent, NeutronSriovHostConfig and default compute services.

   # openstack overcloud roles generate \
   -o /home/stack/templates/roles_data.yaml \
   Controller ComputeSriov

2. To prepare the SR-IOV containers, include the neutron-sriov.yaml and roles_data.yaml files when you generate the overcloud_images.yaml file.

   SERVICES=\
   /usr/share/openstack-tripleo-heat-templates/environments/services
   
   openstack tripleo container image prepare \
   --namespace=registry.redhat.io/rhosp15-rhel8 \
   --push-destination=192.168.24.1:8787 \
   --prefix=openstack- \
   --tag-from-label {version}-{release} \
   -e ${SERVICES}/neutron-sriov.yaml \
   --roles-file /home/stack/templates/roles_data.yaml \
   --output-env-file=/home/stack/templates/overcloud_images.yaml \
   --output-images-file=/home/stack/local_registry_images.yaml

   Note

   The push-destination IP address is the address that you previously set with the local_ip parameter in the undercloud.conf configuration file.

   For more information on container image preparation, see Director Installation and Usage.

3. To apply the KernelAgs and TunedProfile parameters, include the host-config-and-reboot.yaml file from /usr/share/openstack-tripleo-heat-templates/environments with your deployment script.

   openstack overcloud deploy --templates \
   … \
   -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml \
   ...

4. Configure the parameters for the SR-IOV nodes under parameter_defaults appropriately for your cluster, and your hardware configuration. These settings typically belong in the network-environment.yaml file.

     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

5. In the same file, configure role specific parameters for SR-IOV compute nodes.

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that physical function (PF). In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

     ComputeSriovParameters:
       IsolCpusList: "1-19,21-39"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-19,21-39"
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysicalDevMappings:
         - tenant:p7p1
         - tenant:p7p2
       NeutronSriovNumVFs:
         - p7p1:5
         - p7p2:5
       NovaPCIPassthrough:
         - devname: "p7p1"
           physical_network: "tenant"
         - devname: "p7p2"
           physical_network: "tenant"
       NovaVcpuPinSet: '1-19,21-39'
       NovaReservedHostMemory: 4096

6. Configure the SR-IOV-enabled interfaces in the compute.yaml network configuration template. To create SR-IOV virtual functions (VFs), configure the interfaces as standalone NICs:

                - type: interface
                   name: p7p3
                   mtu: 9000
                   use_dhcp: false
                   defroute: false
                   nm_controlled: true
                   hotplug: true
   
                 - type: interface
                   name: p7p4
                   mtu: 9000
                   use_dhcp: false
                   defroute: false
                   nm_controlled: true
                   hotplug: true

7. Ensure that the list of default filters includes the value AggregateInstanceExtraSpecsFilter.

   NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','AggregateInstanceExtraSpecsFilter']

8. Deploy the overcloud.

1. Use the interface type sriov_pf in an os-net-config role file to configure a physical function (PF) that the host can use.

           - type: sriov_pf
             name: <interface name>
             use_dhcp: false
             numvfs: <number of vfs>
             promisc: <true/false> #optional (Defaults to true)

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

1. Use the interface type sriov_pf to configure VFs in a bond that the host can use.

                  - type: linux_bond
                    name: internal_bond
                    bonding_options: mode=active-backup
                    use_dhcp: false
                    members:
                    - type: sriov_vf
                      device: nic7
                      vfid: 1
                    - type: sriov_vf
                      device: nic8
                      vfid: 1
   
                  - type: vlan
                    vlan_id:
                      get_param: InternalApiNetworkVlanID
                    device: internal_bond
                    addresses:
                    - ip_netmask:
                        get_param: InternalApiIpSubnet

   • The VLAN tag must be unique across all VFs that belong to a common PF device. You must assign VLAN tags to an interface type:

     • linux_bond
     • ovs_bridge
     • ovs_dpdk_port
   • The applicable VF ID range starts at zero, and ends at the total number of VFs minus one.

2. To reserve VFs for VMs, use the NovaPCIPassthrough parameter. You must assign a regex value to the address parameter to identify the VFs that you want to pass through to Nova, to be used by the virtual instances and not the host.

   You can obtain these values from lspci. You might need to pre-emptively boot a compute node into a Linux environment to obtain this information.

   The lspci command returns the address of each device in the format <bus>:<device>:<slot>. Enter these address values in the NovaPCIPassthrough parameter in the following format:

     NovaPCIPassthrough:
       - physical_network: "sriovnet2"
         address: {"domain": ".*", "bus": "06", "slot": "11", "function": "[5-7]"}
       - physical_network: "sriovnet2"
         address: {"domain": ".*", "bus": "06", "slot": "10", "function": "[5]"}

3. Ensure that IOMMU is enabled on all nodes that require NIC partitioning. For example, if you want NIC Partitioning for compute nodes, enable IOMMU using the KernelArgs parameter for that role:

   parameter_defaults:
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

Validation

1. Check the number of VFs.

   [root@overcloud-compute-0 heat-admin]# cat /sys/class/net/p4p1/device/sriov_numvfs
   10
   [root@overcloud-compute-0 heat-admin]# cat /sys/class/net/p4p2/device/sriov_numvfs
   10

2. Check Linux bonds.

   [root@overcloud-compute-0 heat-admin]# cat /proc/net/bonding/intapi_bond
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: p4p1_1
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   
   Slave Interface: p4p1_1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 16:b4:4c:aa:f0:a8
   Slave queue ID: 0
   
   Slave Interface: p4p2_1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: b6:be:82:ac:51:98
   Slave queue ID: 0
   [root@overcloud-compute-0 heat-admin]# cat /proc/net/bonding/st_bond
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: p4p1_3
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   
   Slave Interface: p4p1_3
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 9a:86:b7:cc:17:e4
   Slave queue ID: 0
   
   Slave Interface: p4p2_3
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: d6:07:f8:78:dd:5b
   Slave queue ID: 0

3. List OVS bonds

   [root@overcloud-compute-0 heat-admin]# ovs-appctl bond/show
   ---- bond_prov ----
   bond_mode: active-backup
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: f2:ad:c7:00:f5:c7(dpdk2)
   
   slave dpdk2: enabled
     active slave
     may_enable: true
   
   slave dpdk3: enabled
     may_enable: true
   
   ---- bond_tnt ----
   bond_mode: active-backup
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: b2:7e:b8:75:72:e8(dpdk0)
   
   slave dpdk0: enabled
     active slave
     may_enable: true
   
   slave dpdk1: enabled
     may_enable: true

4. Show OVS connections.

   [root@overcloud-compute-0 heat-admin]# ovs-vsctl show
   cec12069-9d4c-4fa8-bfe4-decfdf258f49
       Manager "ptcp:6640:127.0.0.1"
           is_connected: true
       Bridge br-tenant
           fail_mode: standalone
           Port br-tenant
               Interface br-tenant
                   type: internal
           Port bond_tnt
               Interface "dpdk0"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:02.2"}
               Interface "dpdk1"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:04.2"}
       Bridge "sriov2"
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "phy-sriov2"
               Interface "phy-sriov2"
                   type: patch
                   options: {peer="int-sriov2"}
           Port "sriov2"
               Interface "sriov2"
                   type: internal
       Bridge br-int
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "int-sriov2"
               Interface "int-sriov2"
                   type: patch
                   options: {peer="phy-sriov2"}
           Port br-int
               Interface br-int
                   type: internal
           Port "vhu93164679-22"
               tag: 4
               Interface "vhu93164679-22"
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu93164679-22"}
           Port "vhu5d6b9f5a-0d"
               tag: 3
               Interface "vhu5d6b9f5a-0d"
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu5d6b9f5a-0d"}
           Port patch-tun
               Interface patch-tun
                   type: patch
                   options: {peer=patch-int}
           Port "int-sriov1"
               Interface "int-sriov1"
                   type: patch
                   options: {peer="phy-sriov1"}
           Port int-br-vfs
               Interface int-br-vfs
                   type: patch
                   options: {peer=phy-br-vfs}
       Bridge br-vfs
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port phy-br-vfs
               Interface phy-br-vfs
                   type: patch
                   options: {peer=int-br-vfs}
           Port bond_prov
               Interface "dpdk3"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:04.5"}
               Interface "dpdk2"
                   type: dpdk
                   options: {dpdk-devargs="0000:82:02.5"}
           Port br-vfs
               Interface br-vfs
                   type: internal
       Bridge "sriov1"
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port "sriov1"
               Interface "sriov1"
                   type: internal
           Port "phy-sriov1"
               Interface "phy-sriov1"
                   type: patch
                   options: {peer="int-sriov1"}
       Bridge br-tun
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           Port br-tun
               Interface br-tun
                   type: internal
           Port patch-int
               Interface patch-int
                   type: patch
                   options: {peer=patch-tun}
           Port "vxlan-0a0a7315"
               Interface "vxlan-0a0a7315"
                   type: vxlan
                   options: {df_default="true", in_key=flow, local_ip="10.10.115.10", out_key=flow, remote_ip="10.10.115.21"}
       ovs_version: "2.10.0"

1. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

2. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

3. Create the port.

   • Use vnic-type direct to create an SR-IOV virtual function (VF) port.

     # openstack port create --network net1 --vnic-type direct sriov_port

   • Use the following to create a virtual function with hardware offload.

     # openstack port create --network net1 --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]} sriov_hwoffload_port

   • Use vnic-type direct-physical to create an SR-IOV PF port.

     # openstack port create --network net1 --vnic-type direct-physical sriov_port

4. Deploy an instance.

   # openstack server create --flavor <flavor> --image <image> --nic port-id=<id> <instance name>

1. Ensure that the AggregateInstanceExtraSpecsFilter value is included in the scheduler_default_filters parameter in the nova.conf configuration file. You can set this configuration through the heat parameter NovaSchedulerDefaultFilters under role-specific parameters before deployment.

     ComputeOvsDpdkSriovParameters:
       NovaSchedulerDefaultFilters: ['AggregateInstanceExtraSpecsFilter', 'RetryFilter','AvailabilityZoneFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter','NUMATopologyFilter']

   Note

   This parameter can be added to heat templates and the original deployment script re-run to add this to the configuration of an exiting cluster.

2. Create an aggregate group for single root I/O virtualization (SR-IOV) and add relevant hosts. Define metadata, for example, sriov=true, that matches defined flavor metadata.

   # openstack aggregate create sriov_group
   # openstack aggregate add host sriov_group compute-sriov-0.localdomain
   # openstack aggregate set sriov_group sriov=true

3. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

4. Set additional flavor properties. Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   openstack flavor set --property sriov=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB <flavor>

1. Copy the usr/share/openstack-tripleo-heat-templates/plan-samples/plan-environment-derived-params.yaml file to a local directory and set the input parameters to suit your environment.

     workflow_parameters:
       tripleo.derive_params.v1.derive_parameters:
         # DPDK Parameters #
         # Specifies the minimum number of CPU physical cores to be allocated for DPDK
         # PMD threads. The actual allocation will be based on network config, if
         # the a DPDK port is associated with a numa node, then this configuration
         # will be used, else 1.
         num_phy_cores_per_numa_node_for_pmd: 1
         # Amount of memory to be configured as huge pages in percentage. Ouf the
         # total available memory (excluding the NovaReservedHostMemory), the
         # specified percentage of the remaining is configured as huge pages.
         huge_page_allocation_percentage: 50

2. Run the openstack overcloud deploy command with the following options:

   • The update-plan-only option
   • The role file and all environment files specific to your environment

   • The plan-environment-derived-parms.yaml file with the --plan-environment-file optional argument

     $ openstack overcloud deploy --templates --update-plan-only \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/<environment-file> \
     ... #repeat as necessary ...
     -p /home/stack/plan-environment-derived-params.yaml

1. Copy the derived parameters from the deploy command output to the network-environment.yaml file.

     # DPDK compute node.
     ComputeOvsDpdkParameters:
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "1,2,3,4,5,6,7,9,10,17,18,19,20,21,22,23,11,12,13,14,15,25,26,27,28,29,30,31"
       NovaVcpuPinSet: ['2,3,4,5,6,7,18,19,20,21,22,23,10,11,12,13,14,15,26,27,28,29,30,31']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "1024,1024"
       OvsDpdkMemoryChannels: "4"
       OvsDpdkCoreList: "0,16,8,24"
       OvsPmdCoreList: "1,17,9,25"

   Note

   These parameters apply to the specific role, ComputeOvsDpdk. You can apply these parameters globally, but role-specific parameters overwrite global parameters.

2. Deploy the overcloud using the role file and all environment files specific to your environment. For more information, see Deploying the Overcloud.

1. If you use composable roles, copy and modify the roles_data.yaml file to add the custom role for OVS-DPDK.
2. Update the appropriate network-environment.yaml file to include parameters for kernel arguments and DPDK arguments.
3. Update the compute.yaml file to include the bridge for DPDK interface parameters.
4. Update the controller.yaml file to include the same bridge details for DPDK interface parameters.
5. Run the overcloud_deploy.sh script to deploy the overcloud with the DPDK parameters.

1. Set the NeutronGlobalPhysnetMtu parameter in the network-environment.yaml file.

   parameter_defaults:
     # MTU global configuration
     NeutronGlobalPhysnetMtu: 9000

   Note

   Ensure that the NeutronDpdkSocketMemory value in the network-environment.yaml file is large enough to support jumbo frames. See Section 7.4.2, “Memory parameters” for details.

2. Set the MTU value on the bridge to the Compute node in the controller.yaml file.

     -
       type: ovs_bridge
       name: br-link0
       use_dhcp: false
       members:
         -
           type: interface
           name: nic3
           mtu: 9000

3. Set the MTU values for an OVS-DPDK bond in the compute.yaml file:

   - type: ovs_user_bridge
     name: br-link0
     use_dhcp: false
     members:
       - type: ovs_dpdk_bond
         name: dpdkbond0
         mtu: 9000
         rx_queue: 2
         members:
           - type: ovs_dpdk_port
             name: dpdk0
             mtu: 9000
             members:
               - type: interface
                 name: nic4
           - type: ovs_dpdk_port
             name: dpdk1
             mtu: 9000
             members:
               - type: interface
                 name: nic5

1. Ensure parameters for your DPDK compute role populate in network-environment.yaml. If necessary, copy these parameters from: derived OVS-DPDK.

    # DPDK compute node.
     ComputeOvsDpdkParameters:
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-7,17-23,9-15,25-31
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "1-7,17-23,9-15,25-31"
       NovaVcpuPinSet: ['2-7,18-23,10-15,26-31']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "1024,1024"
       OvsDpdkMemoryChannels: "4"
       OvsDpdkCoreList: "0,16,8,24"
       OvsPmdCoreList: "1,17,9,25"

2. Deploy the overcloud using the openstack overcloud deploy command.

   • Include the role file and all environment files specific to your environment.

   • Apply the KernelArgs and TunedProfile parameters by including the host-config-and-reboot.yaml file from /usr/share/openstack-tripleo-heat-templates/environments to your deployment script:

     TEMPLATES_HOME=”/usr/share/openstack-tripleo-heat-templates”
     CUSTOM_TEMPLATES=”/home/stack/templates”
     
     openstack overcloud deploy --templates \
     -r ${CUSTOM_TEMPLATES}/roles_data.yaml \
     -e ${TEMPLATES_HOME}/environments/host-config-and-reboot.yaml \
     -e ${CUSTOM_TEMPLATES}/network-environment.yaml \
     -e ${CUSTOM_TEMPLATES}/controller.yaml
     -e ${CUSTOM_TEMPLATES}/computeovsdpdk.yaml \
     ...

1. Create an aggregate group and add relevant hosts for OVS-DPDK. Define metadata, for example dpdk=true, that matches defined flavor metadata.

    # openstack aggregate create dpdk_group
    # openstack aggregate add host dpdk_group [compute-host]
    # openstack aggregate set --property dpdk=true dpdk_group

   Note

   Use host aggregates to separate CPU-pinned instances from unpinned instances. Instances without CPU pinning have different resourcing requirements of instances with CPU pinning.

2. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

3. Set additional flavor properties. Note that the defined metadata, dpdk=true, matches the defined metadata in the DPDK aggregate.

   # openstack flavor set <flavor> --property dpdk=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB --property hw:emulator_threads_policy=isolate

   In this example, m1.medium_huge_4cpu is the flavor name and the remaining parameters set the other properties for the flavor.

   For details on the emulator threads policy for performance improvements, see: Configure Emulator Threads to run on a Dedicated Physical CPU.

4. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

5. Deploy an instance.

   # openstack server create --flavor <flavor> --image <glance image> --nic net-id=<network ID>

1. Set the image properties.

   # openstack image set --property hw_vif_multiqueue_enabled=true <image>

2. Set additional flavor properties.

   # openstack flavor set --property hw:vif_multiqueue_enabled=true <flavor>

1. Review the bridge configuration and confirm that the bridge was created with the datapath_type=netdev.

   # ovs-vsctl list bridge br0
   _uuid               : bdce0825-e263-4d15-b256-f01222df96f3
   auto_attach         : []
   controller          : []
   datapath_id         : "00002608cebd154d"
   datapath_type       : netdev
   datapath_version    : "<built-in>"
   external_ids        : {}
   fail_mode           : []
   flood_vlans         : []
   flow_tables         : {}
   ipfix               : []
   mcast_snooping_enable: false
   mirrors             : []
   name                : "br0"
   netflow             : []
   other_config        : {}
   ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
   protocols           : []
   rstp_enable         : false
   rstp_status         : {}
   sflow               : []
   status              : {}
   stp_enable          : false

2. Confirm that the docker container neutron_ovs_agent is configured to start automatically.

   # docker inspect neutron_ovs_agent | grep -A1 RestartPolicy
               "RestartPolicy": {
                   "Name": "always",

3. If the container has issues with starting, you can view any related messages.

   # less /var/log/containers/neutron/openvswitch-agent.log

4. Confirm that the PMD CPU mask of the ovs-dpdk are pinned to the CPUs. In case of HT, use sibling CPUs.

   For example, observe CPU 4:

   # cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
   4,20

   Use CPU 4 and 20:

   # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

   Display their status:

   # tuna -t ovs-vswitchd -CP
   thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
   3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
   3219   OTHER 	0    	6     	1        	0   	handler24
   3220   OTHER 	0    	6     	1        	0   	handler21
   3221   OTHER 	0    	6     	1        	0   	handler22
   3222   OTHER 	0    	6     	1        	0   	handler23
   3223   OTHER 	0    	6     	1        	0   	handler25
   3224   OTHER 	0    	6     	1        	0   	handler26
   3225   OTHER 	0    	6     	1        	0   	handler27
   3226   OTHER 	0    	6     	1        	0   	handler28
   3227   OTHER 	0    	6     	2        	0   	handler31
   3228   OTHER 	0    	6     	2        	4   	handler30
   3229   OTHER 	0    	6     	2        	5   	handler32
   3230   OTHER 	0    	6	953538      	431   revalidator29
   3231   OTHER 	0    	6   1424258      	976   revalidator33
   3232   OTHER 	0    	6   1424693      	836   revalidator34
   3233   OTHER 	0    	6	951678      	503   revalidator36
   3234   OTHER 	0    	6   1425128      	498   revalidator35
   *3235   OTHER 	0    	4	151123       	51       	pmd37*
   *3236   OTHER 	0   	20	298967       	48       	pmd38*
   3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
   3165   OTHER 	0    	6	237634        	0   vhost_thread1
   3166   OTHER 	0    	6  	3665        	0       	urcu2

1. To increase the RX and TX queue size, include the following lines to the parameter_defaults: section of a relevant director role. Here is an example with ComputeOvsDpdk role:

   parameter_defaults:
     ComputeOvsDpdkParameters:
       -NovaLibvirtRxQueueSize: 1024
       -NovaLibvirtTxQueueSize: 1024

1. Set a new NeutronPhysnetNUMANodesMapping parameter to map the physical network to the NUMA node that you associate with the physical network.

2. If you use tunnels, such as VxLAN or GRE, you must also set the NeutronTunnelNUMANodes parameter.

   parameter_defaults:
     NeutronPhysnetNUMANodesMapping: {<physnet_name>: [<NUMA_NODE>]}
     NeutronTunnelNUMANodes: <NUMA_NODE>,<NUMA_NODE>

1. Add the custom resources for OVS-DPDK under resource_registry:

     resource_registry:
       # Specify the relative/absolute path to the config files you want to use for override the default.
       OS::TripleO::ComputeOvsDpdkSriov::Net::SoftwareConfig: nic-configs/computeovsdpdksriov.yaml
       OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml

2. Under parameter_defaults, set the tunnel type to vxlan and the network type to vxlan,vlan:

   NeutronTunnelTypes: 'vxlan'
   NeutronNetworkType: 'vxlan,vlan'

3. Under parameters_defaults, set the bridge mapping:

   # The OVS logical->physical bridge mappings to use.
   NeutronBridgeMappings:
     - dpdk-mgmt:br-link0

4. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role:

     ##########################
     # OVS DPDK configuration #
     ##########################
     ComputeOvsDpdkSriovParameters:
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2-19,22-39"
       NovaVcpuPinSet: ['4-19,24-39']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "3072,1024"
       OvsDpdkMemoryChannels: "4"
       OvsDpdkCoreList: "0,20,1,21"
       OvsPmdCoreList: "2,22,3,23"
       NovaComputeCpuSharedSet: [0,20,1,21]
       NovaLibvirtRxQueueSize: 1024
       NovaLibvirtTxQueueSize: 1024

   Note

   You must assign at least one CPU with sibling thread on each NUMA node with or without DPDK NICs present for DPDK PMD to avoid failures in creating guest instances. This is expressed in the example through the OvsPmdCoreList parameter, having cores 2 and 22 from NUMA 1, and having cores 3 and 23 from NUMA 2.

   Note

   These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the VMs is the boot parameter minus the OvsDpdkSocketMemory.

   You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

   Note

   OvsDPDKCoreList and OvsDpdkMemoryChannels are the required settings for this procedure. If you deploy DPDK without appropriate values, the deployment might fail or be unstable.

5. Configure the role-specific parameters for SR-IOV:

     NovaPCIPassthrough:
       - devname: "p7p3"
         trusted: "true"
         physical_network: "sriov-1"
       - devname: "p7p4"
         trusted: "true"
         physical_network: "sriov-2"

1. Create the control plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic2
         primary: true

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageMgmtNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageMgmtIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: ExternalNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: ExternalIpSubnet
       routes:
       - default: true
         next_hop:
           get_param: ExternalInterfaceDefaultRoute

3. Create the OVS bridge for access to neutron-dhcp-agent and neutron-metadata-agent services.

     - type: ovs_bridge
       name: br-link0
       use_dhcp: false
       mtu: 9000
       members:
       - type: interface
         name: nic3
         mtu: 9000
       - type: vlan
         vlan_id:
           get_param: TenantNetworkVlanID
         mtu: 9000
         addresses:
         - ip_netmask:
             get_param: TenantIpSubnet

1. Create the control plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic3
         primary: true
       - type: interface
         name: nic4

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet

3. Set a bridge with a DPDK port to link to the controller.

     - type: ovs_user_bridge
       name: br-link0
       use_dhcp: false
       ovs_extra:
         - str_replace:
             template: set port br-link0 tag=_VLAN_TAG_
             params:
               _VLAN_TAG_:
                  get_param: TenantNetworkVlanID
       addresses:
         - ip_netmask:
             get_param: TenantIpSubnet
       members:
         - type: ovs_dpdk_bond
           name: dpdkbond0
           mtu: 9000
           rx_queue: 2
           members:
             - type: ovs_dpdk_port
               name: dpdk0
               members:
                 - type: interface
                   name: nic7
             - type: ovs_dpdk_port
               name: dpdk1
               members:
                 - type: interface
                   name: nic8

   Note

   To include multiple DPDK devices, repeat the type code section for each DPDK device that you want to add.

   Note

   When using OVS-DPDK, all bridges on the same Compute node must be of type ovs_user_bridge. The director might accept the configuration, but Red Hat OpenStack Platform does not support mixing ovs_bridge and ovs_user_bridge on the same node.

1. Run the overcloud_deploy.sh script to deploy the overcloud.