Search

Chapter 4. Configure DPDK Accelerated Open vSwitch (OVS) for Networking

download PDF

This chapter covers DPDK with Open vSwitch installation and tuning within the Red Hat OpenStack Platform environment.

See Planning Your OVS-DPDK Deployment to understand the parameters used to configure OVS-DPDK.

Note

This guide provides examples for CPU assignments, memory allocation, and NIC configurations that may vary from your topology and use case. See the Network Functions Virtualization Product Guide and the Network Functions Virtualization Planning Guide to understand the hardware and configuration options.

Note

Do not edit or change isolated_cores or other values in etc/tuned/cpu-partitioning-variables.conf that are modified by these director heat templates.

In the following procedures, you need to:

  • Update the appropriate network-environment.yaml file to include parameters for kernel arguments and DPDK arguments.
  • Update the compute.yaml file to include the bridge for DPDK interface parameters.
  • Update the controller.yaml file to include the same bridge details for DPDK interface parameters.
  • Run the overcloud_deploy.sh script to deploy the overcloud with the DPDK parameters.
Note

For deployments that use hugepages, you also need to configure reserved_huge_pages. See How to set reserved_huge_pages in /etc/nova/nova.conf in Red Hat OpenStack Platform 10 for details.

OpenStack NFV Config Guide Topology 450694 0617 ECE OVS DPDK

Before you begin the procedure, ensure that you have the following:

  • Red Hat OpenStack Platform 10 with Red Hat Enterprise Linux 7.5
  • OVS-DPDK 2.9
  • Tested NIC. For a list of tested NICs for NFV, see Tested NICs.
Note

Red Hat OpenStack Platform 10 with OVS 2.9 operates in OVS client mode for OVS-DPDK deployments.

4.1. Naming Conventions

We recommend that you follow a consistent naming convention when you use custom roles in your OpenStack deployment, especially with multiple nodes. This naming convention can assist you when creating the following files and configurations:

  • instackenv.json - To differentiate between nodes with different hardware or NIC capabilities.

    "name":"computeovsdpdk-0"
  • roles_data.yaml - To differentiate between compute-based roles that support DPDK.

    `ComputeOvsDpdk`
  • network-environment.yaml - To ensure that you match the custom role to the correct flavor name.

     `OvercloudComputeOvsDpdkFlavor: computeovsdpdk`
  • nic-config file names - To differentiate NIC yaml files for compute nodes that support DPDK interfaces.
  • Flavor creation - To help you match a flavor and capabilities:profile value to the appropriate bare metal node and custom role.

    # openstack flavor create --id auto --ram 4096 --disk 40 --vcpus 4 computeovsdpdk
    # openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="computeovsdpdk" computeovsdpdk
  • Bare metal node - To ensure that you match the bare metal node with the appropriate hardware and capability:profile value.

     # openstack baremetal node update computeovsdpdk-0 add properties/capabilities='profile:computeovsdpdk,boot_option:local'
Note

The flavor name does not have to match the capabilities:profile value for the flavor, but the flavor capabilities:profile value must match the bare metal node properties/capabilities='profile value. All three use computeovsdpdk in this example.

Note

Ensure that all your nodes used for a custom role and profile have the same CPU, RAM, and PCI hardware topology.

4.2. Configure Two-Port OVS-DPDK Data Plane Bonding with VLAN Tunnelling

This section covers the procedures to configure and deploy OVS-DPDK with two data plane ports in an OVS-DPDK bond, with control plane Linux bonding for your OpenStack environment.

4.3. Configure Single-Port OVS-DPDK with VXLAN Tunnelling

This section covers the procedures to configure single-port OVS-DPDK with control plane Linux bonding and VXLAN tunnelling for your OpenStack environment.

4.4. Set the MTU Value for OVS-DPDK Interfaces

Red Hat OpenStack Platform supports jumbo frames for OVS-DPDK. To set the MTU value for jumbo frames you must:

  • Set the global MTU value for networking in the network-environment.yaml file.
  • Set the physical DPDK port MTU value in the compute.yaml file. This value is also used by the vhost user interface.
  • Set the MTU value within any guest instances on the Compute node to ensure that you have a comparable MTU value from end to end in your configuration.
Note

VXLAN packets include an extra 50 bytes in the header. Calculate your MTU requirements based on these additional header bytes. For example, an MTU value of 9000 means the VXLAN tunnel MTU value is 8950 to account for these extra bytes.

Note

You do not need any special configuration for the physical NIC since the NIC is controlled by the DPDK PMD and has the same MTU value set by the compute.yaml file. You cannot set an MTU value larger than the maximum value supported by the physical NIC.

To set the MTU value for OVS-DPDK interfaces:

  1. Set the NeutronGlobalPhysnetMtu parameter in the network-environment.yaml file.

    parameter_defaults:
      # Global MTU configuration on Neutron
      NeutronGlobalPhysnetMtu: 9000
    Note

    Ensure that the NeutronDpdkSocketMemory value in the network-environment.yaml file is large enough to support jumbo frames. See Memory Parameters for details.

  2. Set the MTU value on the bridge to the Compute node in the controller.yaml file.

      -
        type: ovs_bridge
        name: br-link0
        use_dhcp: false
        mtu: 9000
        members:
          -
            type: ovs_bond
            name: bond0
            use_dhcp: true
            members:
              -
                type: interface
                name: nic7
                mtu: 9000
              -
                type: interface
                name: nic8
                mtu: 9000
          -
            type: vlan
            vlan_id: {get_param: TenantNetworkVlanID}
            device: bond0
            mtu: 9000
            addresses:
              -
                ip_netmask: {get_param: TenantIpSubnet}

To set the MTU values for the OVS-DPDK interfaces and bonds in the compute.yaml file:

  -
    type: ovs_user_bridge
    name: br-link0
    use_dhcp: false
    ovs_extra:
      -
        str_replace:
          template: set port br-link0 tag=VLAN_TAG
          params:
            VLAN_TAG: {get_param: TenantNetworkVlanID}
    addresses:
      -
        ip_netmask: {get_param: TenantIpSubnet}
    members:
      -
        type: ovs_dpdk_bond
        name: dpdkbond0
        mtu: 9000
        ovs_extra:
          - set interface dpdk0 mtu_request=$MTU
          - set interface dpdk1 mtu_request=$MTU
          - set interface dpdk0 options:n_rxq=2
          - set interface dpdk1 options:n_rxq=2
        members:
          -
            type: ovs_dpdk_port
            name: dpdk0
            members:
              -
                type: interface
                name: nic7
          -
            type: ovs_dpdk_port
            name: dpdk1
            members:
              -
                type: interface
                name: nic8

4.5. Set Multiqueue for OVS-DPDK Interfaces

To set the number of queues for an OVS-DPDK port on the Compute node, modify the compute.yaml file as follows:

  -
    type: ovs_user_bridge
    name: br-link0
    use_dhcp: false
    ovs_extra:
      -
        str_replace:
          template: set port br-link0 tag=VLAN_TAG
          params:
            VLAN_TAG: {get_param: TenantNetworkVlanID}
    addresses:
      -
        ip_netmask: {get_param: TenantIpSubnet}
    members:
      -
        type: ovs_dpdk_bond
        name: dpdkbond0
        mtu: 9000
        ovs_extra:
          - set interface dpdk0 mtu_request=$MTU
          - set interface dpdk1 mtu_request=$MTU
          - set interface dpdk0 options:n_rxq=2
          - set interface dpdk1 options:n_rxq=2
        members:
          -
            type: ovs_dpdk_port
            name: dpdk0
            members:
              -
                type: interface
                name: nic7
          -
            type: ovs_dpdk_port
            name: dpdk1
            members:
              -
                type: interface
                name: nic8

4.6. Known Limitations

There are certain limitations when configuring OVS-DPDK with Red Hat OpenStack Platform 10 for the NFV use case:

  • Use Linux bonds for control plane networks. Ensure both PCI devices used in the bond are on the same NUMA node for optimum performance. Neutron Linux bridge configuration is not supported by Red Hat.
  • Huge pages are required for every instance running on the hosts with OVS-DPDK. If huge pages are not present in the guest, the interface will appear but not function.
  • There is a performance degradation of services that use tap devices, because these devices do not support DPDK. For example, services such as DVR, FWaaS, and LBaaS use tap devices.

    • With OVS-DPDK, you can enable DVR with netdev datapath, but this has poor performance and is not suitable for a production environment. DVR uses kernel namespace and tap devices to perform the routing.
    • To ensure the DVR routing performs well with OVS-DPDK, you need to use a controller such as ODL which implements routing as OpenFlow rules. With OVS-DPDK, OpenFlow routing removes the bottleneck introduced by the Linux kernel interfaces so that the full performance of datapath is maintained.
  • When using OVS-DPDK, all bridges should be of type ovs_user_bridge on the Compute node. The director may accept the configuration, but Red Hat OpenStack Platform does not support mixing ovs_bridge and ovs_user_bridge.

4.7. Create a Flavor and Deploy an Instance for OVS-DPDK

After you have completed configuring OVS-DPDK for your Red Hat OpenStack Platform deployment with NFV, you can create a flavor and deploy an instance with the following steps:

  1. Create an aggregate group and add a host to it for OVS-DPDK. Define metadata, for example, "aggregate_instance_extra_specs:dpdk"="true", that matches flavor metadata.

     # openstack aggregate create dpdk_group
     # openstack aggregate set --property \
     "aggregate_instance_extra_specs:dpdk"="true" dpdk_group
     # openstack aggregate add host dpdk compute-ovs-dpdk-0.localdomain
  2. Create a flavor.

    # openstack flavor create <flavor --ram <MB> --disk <GB> --vcpus <#>
  3. Set additional flavor properties. Note that the defined metadata, "aggregate_instance_extra_specs:dpdk"=true", matches the defined metadata on the DPDK aggregate.

    # openstack flavor set --property "aggregate_instance_extra_specs:dpdk"="true" \
    --property hw:cpu_policy=dedicated  \
    --property hw:mem_page_size=large <flavor>
  4. Create the network.

    # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
  5. Create the subnet.

    # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp
  6. Deploy an instance.

    # openstack server create --flavor <flavor> --image <glance_image> --nic net-id=net1 <name>

You have now deployed an instance for the OVS-DPDK with NFV use case.

4.7.1. Optimizing Performance with Emulator Thread Pinning

To improve performance, you can pin the Qemu emulator thread to an alternate core.

  1. Determine which cores are used as vCPUs for your instance.

    # virsh dumpxml dpdk_vm | grep cpuset
        <vcpupin vcpu='0' cpuset='2'/>
        <vcpupin vcpu='1' cpuset='18'/>
        <vcpupin vcpu='2' cpuset='1'/>
        <vcpupin vcpu='3' cpuset='17'/>
        <emulatorpin cpuset='1-2,17-18'/>
  2. Select the core you want to pin the emulator thread to. Ensure the selected core is from the NovaVcpuPinSet.

    #virsh emulatorpin <vm-name> --cpulist 2
    Note

    The pCPU associated with the emulator pin thread consumes one vCPU (two threads if hyperthreading is enabled) from the NovaVcpuPinSet.

4.8. Troubleshooting the Configuration

This section describes the steps to troubleshoot the DPDK-OVS configuration.

  1. Review the bridge configuration and confirm that the bridge was created with the datapath_type=netdev. For example:

    # ovs-vsctl list bridge br0
    _uuid               : bdce0825-e263-4d15-b256-f01222df96f3
    auto_attach         : []
    controller          : []
    datapath_id         : "00002608cebd154d"
    datapath_type       : netdev
    datapath_version    : "<built-in>"
    external_ids        : {}
    fail_mode           : []
    flood_vlans         : []
    flow_tables         : {}
    ipfix               : []
    mcast_snooping_enable: false
    mirrors             : []
    name                : "br0"
    netflow             : []
    other_config        : {}
    ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
    protocols           : []
    rstp_enable         : false
    rstp_status         : {}
    sflow               : []
    status              : {}
    stp_enable          : false
  2. Review the OVS service by confirming that the neutron-ovs-agent is configured to start automatically.

    # systemctl status neutron-openvswitch-agent.service
    neutron-openvswitch-agent.service - OpenStack Neutron Open vSwitch Agent
    Loaded: loaded (/usr/lib/systemd/system/neutron-openvswitch-agent.service; enabled; vendor preset: disabled)
    Active: active (running) since Mon 2015-11-23 14:49:31 AEST; 25min ago

    If the service is having trouble starting, you can view any related messages.

    # journalctl -t neutron-openvswitch-agent.service
  3. Confirm that the PMD CPU mask of the ovs-dpdk are pinned to the CPUs. In case of HT, use sibling CPUs.

    For example, take CPU4:

    # cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
    4,20

    So, using CPU 4 and 20:

    # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

    Display the status.

    # tuna -t ovs-vswitchd -CP
    thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
    3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
    3219   OTHER 	0    	6     	1        	0   	handler24
    3220   OTHER 	0    	6     	1        	0   	handler21
    3221   OTHER 	0    	6     	1        	0   	handler22
    3222   OTHER 	0    	6     	1        	0   	handler23
    3223   OTHER 	0    	6     	1        	0   	handler25
    3224   OTHER 	0    	6     	1        	0   	handler26
    3225   OTHER 	0    	6     	1        	0   	handler27
    3226   OTHER 	0    	6     	1        	0   	handler28
    3227   OTHER 	0    	6     	2        	0   	handler31
    3228   OTHER 	0    	6     	2        	4   	handler30
    3229   OTHER 	0    	6     	2        	5   	handler32
    3230   OTHER 	0    	6	953538      	431   revalidator29
    3231   OTHER 	0    	6   1424258      	976   revalidator33
    3232   OTHER 	0    	6   1424693      	836   revalidator34
    3233   OTHER 	0    	6	951678      	503   revalidator36
    3234   OTHER 	0    	6   1425128      	498   revalidator35
    *3235   OTHER 	0    	4	151123       	51       	pmd37*
    *3236   OTHER 	0   	20	298967       	48       	pmd38*
    3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
    3165   OTHER 	0    	6	237634        	0   vhost_thread1
    3166   OTHER 	0    	6  	3665        	0       	urcu2
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.