Configuring network functions virtualization

Procedure

1. Log in to the Compute nodes in your RHOSP deployment that have Mellanox NICs that you want to configure.

2. Use the mstflint utility to query the ESWITCH_IPV4_TTL_MODIFY_ENABLE Mellanox firmware parameter .

   [root@compute-1 ~]# yum install -y mstflint
   [root@compute-1 ~]# mstconfig -d <PF PCI BDF> q ESWITCH_IPV4_TTL_MODIFY_ENABLE

3. If the ESWITCH_IPV4_TTL_MODIFY_ENABLE parameter is enabled and set to 1, then set the value to 0 to disable it.

   [root@compute-1 ~]# mstconfig -d <PF PCI BDF> s ESWITCH_IPV4_TTL_MODIFY_ENABLE=0`

4. Reboot the node.

Procedure

1. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted.

   [stack@director ~]$ sudo subscription-manager register

2. Determine the entitlement pool ID for Red Hat OpenStack Platform director, for example {Pool ID} from the following command and output:

   [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             {Pool-ID}-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

3. Include the Pool ID value in the following command to attach the Red Hat OpenStack Platform 17.1 entitlement.

   [stack@director ~]$ sudo subscription-manager attach --pool={Pool-ID}-123456

4. Disable the default repositories.

   subscription-manager repos --disable=*

5. Enable the required repositories for Red Hat OpenStack Platform with NFV.

   $ sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms \
   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.9-for-rhel-9-x86_64-rpms \
   --enable=openstack-17.1-for-rhel-9-x86_64-rpms \
   --enable=rhel-9-for-x86_64-nfv-rpms \
   --enable=fast-datapath-for-rhel-9-x86_64-rpms

6. Update your system so you have the latest base system packages.

   [stack@director ~]$ sudo dnf update -y
   [stack@director ~]$ sudo reboot

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Generate a new roles data file named roles_data_compute_sriov.yaml, that includes the Controller and ComputeSriov roles:

   $  openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_compute_sriov.yaml \
    Controller ComputeSriov

   Note

   If you are using multiple technologies in your RHOSP environment, OVS-DPDK, SR-IOV, and OVS hardware offload, you generate just one roles data file to include all the roles:

   $ openstack overcloud roles generate -o /home/stack/templates/\
   roles_data.yaml Controller ComputeOvsDpdk ComputeOvsDpdkSriov \
   Compute:ComputeOvsHwOffload

4. To generate an images file, you run the openstack tripleo container image prepare command. The following inputs are needed:

   • The roles data file that you generated in an earlier step, for example, roles_data_compute_sriov.yaml.

   • The SR-IOV environment file appropriate for your Networking service mechanism driver:

     • ML2/OVN environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml

     • ML2/OVS environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml

       Example

       In this example, the overcloud_images.yaml file is being generated for an ML2/OVN environment:

       $ sudo openstack tripleo container image prepare \
         --roles-file ~/templates/roles_data_compute_sriov.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml \
         -e ~/containers-prepare-parameter.yaml \
         --output-env-file=/home/stack/templates/overcloud_images.yaml

5. Note the path and file name of the roles data file and the images file that you have created. You use these files later when you deploy your overcloud.

Procedure

1. Use one of the following commands on the physical server that has the PCI cards:

   • If your overcloud is deployed:

     $ lspci -nn -s  <pci_device_address>

     Sample output

     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet
     Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

   • If your overcloud has not been deployed:

     $ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

2. Retain the vendor and product IDs for PCI passthrough devices on the SR-IOV compute nodes. You will need these IDs in a later step.
3. Log in to the undercloud as the stack user.

4. Source the stackrc file:

   $ source ~/stackrc

5. Create a custom environment YAML file, for example, sriov-overrides.yaml. Configure the PCI passthrough devices for the SR-IOV compute nodes by adding the following content to the file:

   parameter_defaults:
     ComputeSriovParameters:
       ...
       NovaPCIPassthrough:
         - vendor_id: "<vendor_id>"
           product_id: "<product_id>"
           address: <NIC_address>
           physical_network: "<physical_network>"
       ...

   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device.

   • Replace <physical_network> with the name of the physical network the PCI device is located on.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts.

6. Also in the custom environment file, ensure that PciPassthroughFilter and AggregateInstanceExtraSpecsFilter are in the list of filters for the NovaSchedulerEnabledFilters parameter, that the Compute service (nova) uses to filter a node:

   parameter_defaults:
     ComputeSriovParameters:
       ...
       NovaPCIPassthrough:
         - vendor_id: "<vendor_id>"
           product_id: "<product_id>"
           address: <NIC_address>
           physical_network: "<physical_network>"
       ...
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - AggregateInstanceExtraSpecsFilter

7. Note the path and file name of the custom environment file that you have created. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the custom environment YAML file that you created in Section 7.2, “Configuring PCI passthrough devices for SR-IOV”, or create a new one.

4. Add role-specific parameters for the SR-IOV Compute nodes to the custom environment file.

   Example

     ComputeSriovParameters:
       IsolCpusList: 9-63,73-127
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=9-63,73-127
       NovaReservedHostMemory: 4096
       NovaComputeCpuSharedSet: 0-8,64-72
       NovaComputeCpuDedicatedSet: 9-63,73-127

5. Review the configuration defaults that RHOSP director uses to configure SR-IOV. These defaults are provided in the file and they vary based on your mechanism driver:

   • ML2/OVN

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-ovn-sriov.yaml

   • ML2/OVS

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-sriov.yaml

6. If you need to override any of the configuration defaults, add your overrides to the custom environment file.

   This custom environment file, for example, is where you can add Nova PCI whitelist values or set the network type.

   Example

   In this example, the Networking service (neutron) network type is set to VLAN and ranges are added for the tenants:

   parameter_defaults:
     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

7. If you created a new custom environment file, note its path and file name. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as overcloud-baremetal-deploy.yaml, as instructed in Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. In the bare metal nodes definition file, add a declaration to the Ansible playbook, cli-overcloud-node-kernelargs.yaml.

   The playbook contains kernel arguments to use when you provision bare metal nodes.

   - name: ComputeSriov
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
   ...

5. If you want to set any extra Ansible variables when running the playbook, use the extra_vars property to set them.

   Note

   The variables that you add to extra_vars should be the same role-specific parameters for the SR-IOV Compute nodes that you added to the custom environment file earlier in Section 7.3, “Adding role-specific parameters and configuration overrides”.

   Example

   - name: ComputeSriov
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
         extra_vars:
           kernel_args: 'default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt isolcpus=9-63,73-127'
           tuned_isolated_cores: '9-63,73-127'
           tuned_profile: 'cpu-partitioning'
           reboot_wait_timeout: 1800

6. Note the path and file name of the bare metal nodes definition file that you have created. You use this file later when you configure your NICs and as the input file for the overcloud node provision command when you provision your nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Copy a sample network configuration template.

   Copy a NIC configuration Jinja2 template from the examples in the /usr/share/ansible/roles/tripleo_network_config/templates/ directory. Choose the one that most closely matches your NIC requirements. Modify it as needed.

4. In your NIC configuration template, for example, single_nic_vlans.j2, add your PF and VF interfaces. To create SR-IOV VFs, configure the interfaces as standalone NICs.

   Example

   ...
   - type: sriov_pf
     name: enp196s0f0np0
     mtu: 9000
     numvfs: 16
     use_dhcp: false
     defroute: false
     nm_controlled: true
     hotplug: true
     promisc: false
   ...

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, the modification might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

5. Add the custom network configuration template to the bare metal nodes definition file that you created in Section 7.4, “Creating a bare metal nodes definition file for SR-IOV”.

   Example

   - name: ComputeSriov
     count: 2
     hostname_format: compute-%index%
     defaults:
       networks:
         - network: internal_api
           subnet: internal_api_subnet
         - network: tenant
           subnet: tenant_subnet
         - network: storage
           subnet: storage_subnet
       network_config:
         template: /home/stack/templates/single_nic_vlans.j2
   ...

6. Note the path and file name of the NIC configuration template that you have created. You use this file later if you want to partition your NICs.

Next steps

1. If you want to partition your NICs, proceed to Section 7.6, “Configuring NIC partitioning”.

2. Otherwise, perform these steps:

   1. Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide
   2. Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
   3. Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
   4. Section 7.8, “Deploying an SR-IOV overcloud”

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the NIC configuration template, for example single_nic_vlans.j2, that you created earlier in Section 7.5, “Creating a NIC configuration template for SR-IOV”.

   Tip

   As you complete the steps in this section, you can refer to Section 7.7, “Example configurations for NIC partitions”.

4. Add an entry for the interface type sriov_pf to configure a physical function that the host can use:

      - type: sriov_pf
        name: <interface_name>
        use_dhcp: false
        numvfs: <number_of_vfs>
        promisc: <true/false>

   • Replace <interface_name> with the name of the interface.
   • Replace <number_of_vfs> with the number of VFs.
   • Optional: Replace <true/false> with true to set promiscuous mode, or false to disable promiscuous mode. The default value is true.
   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that physical function (PF). In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

5. Add an entry for the interface type sriov_vf to configure virtual functions that the host can use:

    - type: <bond_type>
      name: internal_bond
      bonding_options: mode=<bonding_option>
      use_dhcp: false
      members:
      - type: sriov_vf
          device: <pf_device_name>
          vfid: <vf_id>
      - type: sriov_vf
          device:  <pf_device_name>
          vfid: <vf_id>
   
    - type: vlan
      vlan_id:
        get_param: InternalApiNetworkVlanID
      spoofcheck: false
      device: internal_bond
      addresses:
      - ip_netmask:
          get_param: InternalApiIpSubnet
      routes:
        list_concat_unique:
        - get_param: InternalApiInterfaceRoutes

   • Replace <bond_type> with the required bond type, for example, linux_bond. You can apply VLAN tags on the bond for other bonds, such as ovs_bond.

   • Replace <bonding_option> with one of the following supported bond modes:

     • active-backup

     • Balance-slb

       Note

       LACP bonds are not supported.

   • Specify the sriov_vf as the interface type to bond in the members section.

     Note

     If you are using an OVS bridge as the interface type, you can configure only one OVS bridge on the sriov_vf of a sriov_pf device. More than one OVS bridge on a single sriov_pf device can result in packet duplication across VFs, and decreased performance.

   • Replace <pf_device_name> with the name of the PF device.
   • If you use a linux_bond, you must assign VLAN tags. If you set a VLAN tag, ensure that you set a unique tag for each VF associated with a single sriov_pf device. You cannot have two VFs from the same PF on the same VLAN.
   • Replace <vf_id> with the ID of the VF. The applicable VF ID range starts at zero, and ends at the maximum number of VFs minus one.
   • Disable spoof checking.
   • Apply VLAN tags on the sriov_vf for linux_bond over VFs.

6. To reserve VFs for instances, include the NovaPCIPassthrough parameter in an environment file.

   Example

   NovaPCIPassthrough:
    - address: "0000:19:0e.3"
      trusted: "true"
      physical_network: "sriov1"
    - address: "0000:19:0e.0"
      trusted: "true"
      physical_network: "sriov2"

   RHOSP director identifies the host VFs, and derives the PCI addresses of the VFs that are available to the instance.

7. Enable IOMMU on all nodes that require NIC partitioning.

   Example

   For example, if you want NIC partitioning for Compute nodes, enable IOMMU using the KernelArgs parameter for that role:

   parameter_defaults:
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment.

8. Ensure that you add this NIC configuration template, for example single_nic_vlans.j2, to the bare metal nodes definition file that you created in Section 7.4, “Creating a bare metal nodes definition file for SR-IOV”.

Next steps

1. Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide
2. Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
3. Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
4. Section 7.8, “Deploying an SR-IOV overcloud”

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Enter the openstack overcloud deploy command.

   It is important to list the inputs to the openstack overcloud deploy command in a particular order. The general rule is to specify the default heat template files first followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Add your inputs to the openstack overcloud deploy command in the following order:

   1. A custom network definition file that contains the specifications for your SR-IOV network on the overcloud, for example, network-data.yaml.

      For more information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   2. A roles file that contains the Controller and ComputeOvsHwOffload roles that RHOSP director uses to deploy your OVS hardware offload environment.

      Example: roles_data_compute_sriov.yaml

      For more information, see Section 7.1, “Generating roles and image files for SR-IOV”.

   3. An output file from provisioning your overcloud networks.

      Example: overcloud-networks-deployed.yaml

      For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   4. An output file from provisioning your overcloud VIPs.

      Example: overcloud-vip-deployed.yaml

      For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   5. An output file from provisioning bare-metal nodes.

      Example: overcloud-baremetal-deployed.yaml

      For more information, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   6. An images file that director uses to determine where to obtain container images and how to store them.

      Example: overcloud_images.yaml

      For more information, see Section 7.1, “Generating roles and image files for SR-IOV”.

   7. An environment file for the Networking service (neutron) mechanism driver and router scheme that your environment uses:

      • ML2/OVN

        • Distributed virtual routing (DVR): neutron-ovn-dvr-ha.yaml
        • Centralized virtual routing: neutron-ovn-ha.yaml

      • ML2/OVS

        • Distributed virtual routing (DVR): neutron-ovs-dvr.yaml
        • Centralized virtual routing: neutron-ovs.yaml

   8. An environment file for SR-IOV, depending on your mechanism driver:

      • ML2/OVN

        • neutron-ovn-sriov.yaml

      • ML2/OVS

        • neutron-sriov.yaml

          Note

          If you also have an OVS-DPDK environment, and want to locate OVS-DPDK and SR-IOV instances on the same node, include the following environment files in your deployment script:

          • ML2/OVN

            neutron-ovn-dpdk.yaml

          • ML2/OVS

            neutron-ovs-dpdk.yaml

   9. One or more custom environment files that contain your configuration for:

      • PCI passthrough devices for the SR-IOV nodes.
      • role-specific parameters for the SR-IOV nodes

      • overrides of default configuration values for the SR-IOV environment.

        Example: sriov-overrides.yaml

        For more information, see:

      • Section 7.2, “Configuring PCI passthrough devices for SR-IOV”.

      • Section 7.3, “Adding role-specific parameters and configuration overrides”.

        Example

        This excerpt from a sample openstack overcloud deploy command demonstrates the proper ordering of the command’s inputs for an SR-IOV, ML2/OVN environment that uses DVR:

        $ openstack overcloud deploy \
        --log-file overcloud_deployment.log \
        --templates /usr/share/openstack-tripleo-heat-templates/ \
        --stack overcloud \
        -n /home/stack/templates/network_data.yaml \
        -r /home/stack/templates/roles_data_compute_sriov.yaml \
        -e /home/stack/templates/overcloud-networks-deployed.yaml \
        -e /home/stack/templates/overcloud-vip-deployed.yaml \
        -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
        -e /home/stack/templates/overcloud-images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-dvr-ha.yaml
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-sriov.yaml \
        -e /home/stack/templates/sriov-overrides.yaml

4. Run the openstack overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.

Verification

1. Perform the steps in Validating your overcloud deployment in the Installing and managing Red Hat OpenStack Platform with director guide.

2. To verify that your NICs are partitioned properly, do the following:

   1. Log in to the overcloud Compute node as tripleo-admin and check the number of VFs:

      Example

      In this example, the number of VFs for both p4p1 and p4p2 is 10:

      $ sudo cat /sys/class/net/p4p1/device/sriov_numvfs
      
      10
      
      $ sudo cat /sys/class/net/p4p2/device/sriov_numvfs
      
      10

   2. Show the OVS connections:

      $ sudo ovs-vsctl show

      Sample output

      You should see output similar to the following:

      b6567fa8-c9ec-4247-9a08-cbf34f04c85f
          Manager "ptcp:6640:127.0.0.1"
              is_connected: true
          Bridge br-sriov2
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port phy-br-sriov2
                  Interface phy-br-sriov2
                      type: patch
                      options: {peer=int-br-sriov2}
              Port br-sriov2
                  Interface br-sriov2
                      type: internal
          Bridge br-sriov1
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port phy-br-sriov1
                  Interface phy-br-sriov1
                      type: patch
                      options: {peer=int-br-sriov1}
              Port br-sriov1
                  Interface br-sriov1
                      type: internal
          Bridge br-ex
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port br-ex
                  Interface br-ex
                      type: internal
              Port phy-br-ex
                  Interface phy-br-ex
                      type: patch
                      options: {peer=int-br-ex}
          Bridge br-tenant
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port br-tenant
                  tag: 305
                  Interface br-tenant
                      type: internal
              Port phy-br-tenant
                  Interface phy-br-tenant
                      type: patch
                      options: {peer=int-br-tenant}
              Port dpdkbond0
                  Interface dpdk0
                      type: dpdk
                      options: {dpdk-devargs="0000:18:0e.0"}
                  Interface dpdk1
                      type: dpdk
                      options: {dpdk-devargs="0000:18:0a.0"}
          Bridge br-tun
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port vxlan-98140025
                  Interface vxlan-98140025
                      type: vxlan
                      options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.37"}
              Port br-tun
                  Interface br-tun
                      type: internal
              Port patch-int
                  Interface patch-int
                      type: patch
                      options: {peer=patch-tun}
              Port vxlan-98140015
                  Interface vxlan-98140015
                      type: vxlan
                      options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.21"}
              Port vxlan-9814009f
                  Interface vxlan-9814009f
                      type: vxlan
                      options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.159"}
              Port vxlan-981400cc
                  Interface vxlan-981400cc
                      type: vxlan
                      options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.204"}
          Bridge br-int
              Controller "tcp:127.0.0.1:6633"
                  is_connected: true
              fail_mode: secure
              datapath_type: netdev
              Port int-br-tenant
                  Interface int-br-tenant
                      type: patch
                      options: {peer=phy-br-tenant}
              Port int-br-ex
                  Interface int-br-ex
                      type: patch
                      options: {peer=phy-br-ex}
              Port int-br-sriov1
                  Interface int-br-sriov1
                      type: patch
                      options: {peer=phy-br-sriov1}
              Port patch-tun
                  Interface patch-tun
                      type: patch
                      options: {peer=patch-int}
              Port br-int
                  Interface br-int
                      type: internal
              Port int-br-sriov2
                  Interface int-br-sriov2
                      type: patch
                      options: {peer=phy-br-sriov2}
              Port vhu4142a221-93
                  tag: 1
                  Interface vhu4142a221-93
                      type: dpdkvhostuserclient
                      options: {vhost-server-path="/var/lib/vhost_sockets/vhu4142a221-93"}
          ovs_version: "2.13.2"

   3. Log in to your SR-IOV Compute node as tripleo-admin and check the Linux bonds:

      $ cat /proc/net/bonding/<bond_name>

      Sample output

      You should see output similar to the following:

      Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
      
      Bonding Mode: fault-tolerance (active-backup)
      Primary Slave: None
      Currently Active Slave: eno3v1
      MII Status: up
      MII Polling Interval (ms): 0
      Up Delay (ms): 0
      Down Delay (ms): 0
      Peer Notification Delay (ms): 0
      
      Slave Interface: eno3v1
      MII Status: up
      Speed: 10000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: 4e:77:94:bd:38:d2
      Slave queue ID: 0
      
      Slave Interface: eno4v1
      MII Status: up
      Speed: 10000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: 4a:74:52:a7:aa:7c
      Slave queue ID: 0

3. List the OVS bonds:

   $ sudo ovs-appctl bond/show

   Sample output

   You should see output similar to the following:

   ---- dpdkbond0 ----
   bond_mode: balance-slb
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   next rebalance: 9491 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: ce:ee:c7:58:8e:b2(dpdk1)
   
   slave dpdk0: enabled
     may_enable: true
   
   slave dpdk1: enabled
     active slave
     may_enable: true

4. If you used NovaPCIPassthrough to pass VFs to instances, test by deploying an SR-IOV instance.

Procedure

1. Create an aggregate group, and add relevant hosts.

   Define metadata, for example, sriov=true, that matches defined flavor metadata.

   $ openstack aggregate create sriov_group
   $ openstack aggregate add host sriov_group compute-sriov-0.localdomain
   $ openstack aggregate set --property sriov=true sriov_group

2. Create a flavor.

   $ openstack flavor create <flavor> --ram <size_mb> --disk <size_gb> \
   --vcpus <number>

3. Set additional flavor properties.

   Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   $ openstack flavor set --property sriov=true \
   --property hw:cpu_policy=dedicated \
   --property hw:mem_page_size=1GB <flavor>

Procedure

1. Create a flavor.

   $ openstack flavor create <flavor_name> --ram <size_mb> \
   --disk <size_gb> --vcpus <number>

   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in Configuring the Compute service for instance creation.

2. Create the network and the subnet:

   $ openstack network create <network_name> \
   --provider-physical-network tenant \
   --provider-network-type vlan --provider-segment <vlan_id>
   
   $ openstack subnet create <name> --network <network_name> \
   --subnet-range <ip_address_cidr> --dhcp

3. Create a virtual function (VF) port or physical function (PF) port:

   • VF port:

     $ openstack port create --network <network_name> \
     --vnic-type direct <port_name>

   • PF port that is dedicated to a single instance:

     This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     $ openstack port create --network <network_name> \
     --vnic-type direct-physical <port_name>

4. Create an instance.

   $ openstack server create --flavor <flavor> --image <image_name> \
   --nic port-id=<id> <instance_name>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Generate an overcloud role for OVS hardware offload that is based on the Compute role:

   Example

   In this example, a role is created, ComputeOvsHwOffload, based on the Compute role. The roles file that the command generates is named, roles_data_compute_ovshwol.yaml:

   $ openstack overcloud roles generate -o \
   roles_data_compute_ovshwol.yaml Controller Compute:ComputeOvsHwOffload

   Note

   If your RHOSP environment includes a mix of OVS-DPDK, SR-IOV, and OVS TC-flower hardware offload technologies, you generate just one roles data file, such as roles_data.yaml to include all the roles:

   $ openstack overcloud roles generate -o /home/stack/templates/\
   roles_data.yaml Controller ComputeOvsDpdk ComputeOvsDpdkSriov \
   Compute:ComputeOvsHwOffload

4. (Optional) change the HostnameFormatDefault: '%stackname%-compute-%index%' name for the ComputeOvsHwOffload role.

5. To generate an images file, you run the openstack tripleo container image prepare command. The following inputs are needed:

   • The roles data file that you generated in an earlier step, for example, roles_data_compute_ovshwol.yaml.

   • The SR-IOV environment file appropriate for your Networking service mechanism driver:

     • ML2/OVN environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml

     • ML2/OVS environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml

       Example

       In this example, the overcloud_images.yaml file is being generated for an ML2/OVN environment:

       $ sudo openstack tripleo container image prepare \
         --roles-file ~/templates/roles_data_compute_ovshwol.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml \
         -e ~/containers-prepare-parameter.yaml \
         --output-env-file=/home/stack/templates/overcloud_images.yaml

6. Note the path and file name of the roles data file and the images file that you have created. You use these files later when you deploy your overcloud.

Procedure

1. Use one of the following commands on the physical server that contains the PCI cards:

   • If your overcloud is deployed:

     $ lspci -nn -s  <pci_device_address>

     Sample output

     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet
     Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

   • If your overcloud has not been deployed:

     $ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

2. Note the vendor and product IDs for PCI passthrough devices on the ComputeOvsHwOffload nodes. You will need these IDs in a later step.
3. Log in to the undercloud as the stack user.

4. Source the stackrc file:

   $ source ~/stackrc

5. Create a custom environment YAML file, for example, ovshwol-overrides.yaml. Configure the PCI passthrough devices for the compute nodes by adding the following content to the file:

   parameter_defaults:
     NeutronOVSFirewallDriver: iptables_hybrid
     ComputeOvsHwOffloadParameters:
       IsolCpusList: 2-9,21-29,11-19,31-39
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
       OvsHwOffload: true
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-tenant
       NovaPCIPassthrough:
         - vendor_id: <vendor-id>
           product_id: <product-id>
           address: <address>
           physical_network: "tenant"
         - vendor_id: <vendor-id>
           product_id: <product-id>
           address: <address>
           physical_network: "null"
       NovaReservedHostMemory: 4096
       NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39
       ...

   Note

   If you are using Mellanox smart NICs, add DerivePciWhitelistEnabled: true under the ComputeOvsHwOffloadParameters parameter. When using OVS hardware offload, the Compute service (nova) scheduler operates similarly to SR-IOV passthrough for instance spawning.

   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device.
   • Replace <physical_network> with the name of the physical network the PCI device is located on.
   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts.

6. In the custom environment file, ensure that PciPassthroughFilter and NUMATopologyFilter are in the list of filters for the NovaSchedulerEnabledFilters parameter. The Compute service (nova) uses this parameter to filter a node:

   parameter_defaults:
     ...
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter
       - AggregateInstanceExtraSpecsFilter

   Note

   Optional: For details on how to troubleshoot and configure OVS Hardware Offload issues in RHOSP 17.1 with Mellanox ConnectX5 NICs, see Troubleshooting Hardware Offload.

7. Note the path and file name of the custom environment file that you have created. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the custom environment YAML file that you created in Section 8.2, “Configuring PCI passthrough devices for OVS TC-flower hardware offload”, or create a new one.

4. Add role-specific parameters for the ComputeOvsHwOffload nodes to the custom environment file.

   Example

     ComputeOvsHwOffloadParameters:
       IsolCpusList: 9-63,73-127
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=9-63,73-127
       NovaReservedHostMemory: 4096
       NovaComputeCpuSharedSet: 0-8,64-72
       NovaComputeCpuDedicatedSet: 9-63,73-127
       TunedProfileName: "cpu-partitioning"

5. Add the OvsHwOffload parameter under role-specific parameters with a value of true.

     ComputeOvsHwOffloadParameters:
       IsolCpusList: 9-63,73-127
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=9-63,73-127
       NovaReservedHostMemory: 4096
       NovaComputeCpuSharedSet: 0-8,64-72
       NovaComputeCpuDedicatedSet: 9-63,73-127
       TunedProfileName: "cpu-partitioning"
       OvsHwOffload: true
     ...

6. Review the configuration defaults that RHOSP director uses to configure OVS hardware offload. These defaults are provided in the file, and they vary based on your mechanism driver:

   • ML2/OVN

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-ovn-sriov.yaml

   • ML2/OVS

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-sriov.yaml

7. If you need to override any of the configuration defaults, add your overrides to the custom environment file.

   This custom environment file, for example, is where you can add Nova PCI whitelist values or set the network type.

   Example

   In this example, the Networking service (neutron) network type is set to VLAN and ranges are added for the tenants:

   parameter_defaults:
     NeutronNetworkType: vlan
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

8. If you created a new custom environment file, note its path and file name. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as overcloud-baremetal-deploy.yaml, as instructed in Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. In the bare metal nodes definition file, add a declaration to the Ansible playbook, cli-overcloud-node-kernelargs.yaml.

   The playbook contains kernel arguments to use when you provision bare metal nodes.

   - name: ComputeOvsHwOffload
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
   ...

5. If you want to set any extra Ansible variables when running the playbook, use the extra_vars property to set them.

   Note

   The variables that you add to extra_vars should be the same role-specific parameters for the ComputeOvsHwOffload nodes that you added to the custom environment file earlier in Section 8.3, “Adding role-specific parameters and configuration overrides for OVS TC-flower hardware offload”.

   Example

   - name: ComputeOvsHwOffload
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
         extra_vars:
           kernel_args: 'default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt isolcpus=9-63,73-127'
           tuned_isolated_cores: '9-63,73-127'
           tuned_profile: 'cpu-partitioning'
           reboot_wait_timeout: 1800

6. Note the path and file name of the bare metal nodes definition file that you have created. You use this file later when you configure your NICs and as the input file for the overcloud node provision command when you provision your nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Copy a sample network configuration template.

   Copy a NIC configuration Jinja2 template from the examples in the /usr/share/ansible/roles/tripleo_network_config/templates/ directory. Choose the one that most closely matches your NIC requirements. Modify it as needed.

4. In your NIC configuration template, for example, single_nic_vlans.j2, add your PF and VF interfaces. To create VFs, configure the interfaces as standalone NICs.

   Example

   ...
   - type: sriov_pf
     name: enp196s0f0np0
     mtu: 9000
     numvfs: 16
     use_dhcp: false
     defroute: false
     nm_controlled: true
     hotplug: true
     promisc: false
     link_mode: switchdev
   ...

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, the modification might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

5. Add the custom network configuration template to the bare metal nodes definition file that you created in Section 8.4, “Creating a bare metal nodes definition file for OVS TC-flower hardware offload”.

   Example

   - name: ComputeOvsHwOffload
     count: 2
     hostname_format: compute-%index%
     defaults:
       networks:
         - network: internal_api
           subnet: internal_api_subnet
         - network: tenant
           subnet: tenant_subnet
         - network: storage
           subnet: storage_subnet
       network_config:
         template: /home/stack/templates/single_nic_vlans.j2
   ...

6. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Note
   • Do not use the NeutronSriovNumVFs parameter when configuring OVS hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.
7. Note the path and file name of the NIC configuration template that you have created. You use this file later if you want to partition your NICs.

Next steps

1. Provision your overcloud networks.

   For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide

2. Provision your overcloud VIPs.

   For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide

3. Provision your bare metal nodes.

   For more information, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide

4. Deploy your overcloud.

   For more information, see Section 8.6, “Deploying an OVS TC-flower hardware offload overcloud”.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Enter the openstack overcloud deploy command.

   It is important to list the inputs to the openstack overcloud deploy command in a particular order. The general rule is to specify the default heat template files first followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Add your inputs to the openstack overcloud deploy command in the following order:

   1. A custom network definition file that contains the specifications for your SR-IOV network on the overcloud, for example, network-data.yaml.

      For more information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   2. A roles file that contains the Controller and ComputeOvsHwOffload roles that RHOSP director uses to deploy your OVS hardware offload environment.

      Example: roles_data_compute_ovshwol.yaml

      For more information, see Section 8.1, “Generating roles and image files for OVS TC-flower hardware offload”.

   3. An output file from provisioning your overcloud networks.

      Example: overcloud-networks-deployed.yaml

      For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   4. An output file from provisioning your overcloud VIPs.

      Example: overcloud-vip-deployed.yaml

      For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   5. An output file from provisioning bare-metal nodes.

      Example: overcloud-baremetal-deployed.yaml

      For more information, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   6. An images file that director uses to determine where to obtain container images and how to store them.

      Example: overcloud_images.yaml

      For more information, see Section 8.1, “Generating roles and image files for OVS TC-flower hardware offload”.

   7. An environment file for the Networking service (neutron) mechanism driver and router scheme that your environment uses:

      • ML2/OVN

        • Distributed virtual routing (DVR): neutron-ovn-dvr-ha.yaml
        • Centralized virtual routing: neutron-ovn-ha.yaml

      • ML2/OVS

        • Distributed virtual routing (DVR): neutron-ovs-dvr.yaml
        • Centralized virtual routing: neutron-ovs.yaml

   8. An environment file for SR-IOV, depending on your mechanism driver:

      • ML2/OVN

        • neutron-ovn-sriov.yaml

      • ML2/OVS

        • neutron-sriov.yaml

          Note

          If you also have an OVS-DPDK environment, and want to locate OVS-DPDK and SR-IOV instances on the same node, include the following environment files in your deployment script:

          • ML2/OVN

            neutron-ovn-dpdk.yaml

          • ML2/OVS

            neutron-ovs-dpdk.yaml

   9. One or more custom environment files that contain your configuration for:

      • PCI passthrough devices for the ComputeOvsHwOffload nodes.
      • role-specific parameters for the ComputeOvsHwOffload nodes

      • overrides of default configuration values for the OVS hardware offload environment.

        Example: ovshwol-overrides.yaml

        For more information, see:

      • Section 8.2, “Configuring PCI passthrough devices for OVS TC-flower hardware offload”.

      • Section 8.3, “Adding role-specific parameters and configuration overrides for OVS TC-flower hardware offload”.

        Example

        This excerpt from a sample openstack overcloud deploy command demonstrates the proper ordering of the command’s inputs for an SR-IOV, ML2/OVN environment that uses DVR:

        $ openstack overcloud deploy \
        --log-file overcloud_deployment.log \
        --templates /usr/share/openstack-tripleo-heat-templates/ \
        --stack overcloud \
        -n /home/stack/templates/network_data.yaml \
        -r /home/stack/templates/roles_data_compute_ovshwol.yaml \
        -e /home/stack/templates/overcloud-networks-deployed.yaml \
        -e /home/stack/templates/overcloud-vip-deployed.yaml \
        -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
        -e /home/stack/templates/overcloud-images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-dvr-ha.yaml
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-sriov.yaml \
        -e /home/stack/templates/ovshwol-overrides.yaml

4. Run the openstack overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.

Next steps

1. Ensure that the e-switch mode for the NICs is set to switchdev.

   The switchdev mode establishes representor ports on the NIC that are mapped to the VFs.

   Important

   You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

   1. Check the NIC by running this command:

      Example

      In this example, the NIC pci/0000:03:00.0 is queried:

      $ sudo devlink dev eswitch show pci/0000:03:00.0

      Sample output

      You should see output similar to the following:

      pci/0000:03:00.0: mode switchdev inline-mode none encap enable

   2. To set the NIC to switchdev mode, run this command:

      Example

      In this example, the e-switch mode for the NIC pci/0000:03:00.0 is set to switchdev:

      $ sudo devlink dev eswitch set pci/0000:03:00.0 mode switchdev

2. To allocate a port from a switchdev-enabled NIC, do the following:

   1. Log in as a RHOSP user with the admin role, and create a neutron port with a binding-profile value of capabilities, and disable port security:

      Important

      You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

      $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 --disable-port-security

   2. Pass this port information when you create the instance.

      You associate the representor port with the instance VF interface and connect the representor port to OVS bridge br-int for one-time OVS data path processing. A VF port representor functions like a software version of a physical “patch panel” front-end.

      For more information about new instance creation, see Section 8.8, “Creating an instance in an SR-IOV or an OVS TC-flower hardware offload environment”.

3. Apply the following configuration on the interfaces, and the representor ports, to ensure that TC Flower pushes the flow programming at the port level:

    $ sudo ethtool -K <device-name> hw-tc-offload on

4. Adjust the number of channels for each network interface to improve performance.

   A channel includes an interrupt request (IRQ) and the set of queues that trigger the IRQ. When you set the mlx5_core driver to switchdev mode, the mlx5_core driver defaults to one combined channel, which might not deliver optimal performance.

   On the physical function (PF) representors, enter the following command to adjust the number of CPUs available to the host.

   Example

   In this example, the number of multi-purpose channels is set to 3 on the network interface, eno3s0f0:

   $ sudo ethtool -L enp3s0f0 combined 3

Procedure

1. Create an aggregate group, and add relevant hosts.

   Define metadata, for example, sriov=true, that matches defined flavor metadata.

   $ openstack aggregate create sriov_group
   $ openstack aggregate add host sriov_group compute-sriov-0.localdomain
   $ openstack aggregate set --property sriov=true sriov_group

2. Create a flavor.

   $ openstack flavor create <flavor> --ram <size_mb> --disk <size_gb> \
   --vcpus <number>

3. Set additional flavor properties.

   Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   $ openstack flavor set --property sriov=true \
   --property hw:cpu_policy=dedicated \
   --property hw:mem_page_size=1GB <flavor>

Procedure

1. Create a flavor.

   $ openstack flavor create <flavor_name> --ram <size_mb> \
   --disk <size_gb> --vcpus <number>

   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in Configuring the Compute service for instance creation.

2. Create the network and the subnet:

   $ openstack network create <network_name> \
   --provider-physical-network tenant \
   --provider-network-type vlan --provider-segment <vlan_id>
   
   $ openstack subnet create <name> --network <network_name> \
   --subnet-range <ip_address_cidr> --dhcp

3. If you are not a RHOSP user with the admin role, your RHOSP administrator can provide you with the necessary VF or PF to create an instance. Proceed to step 5.

4. If you are a RHOSP user with the admin role, you can create VF or PF ports:

   • VF port:

     $ openstack port create --network <network_name> --vnic-type direct \
     --binding-profile '{"capabilities": ["switchdev"]} <port_name>

   • PF port that is dedicated to a single instance:

     This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     $ openstack port create --network <network_name> \
     --vnic-type direct-physical <port_name>

5. Create an instance.

   $ openstack server create --flavor <flavor> --image <image_name> \
   --nic port-id=<id> <instance_name>

Procedure

1. During deployment, use the host network configuration tool os-net-config to enable hw-tc-offload.
2. Enable hw-tc-offload on the sriov_config service any time you reboot the Compute node.

3. Set the hw-tc-offload parameter to on for the NICs that are attached to the bond:.

   Example

   $ ethtool -k ens1f0 | grep tc-offload
   
   hw-tc-offload: on

Procedure

1. Set the eswitch mode to switchdev for the interfaces you use for HW offload.
2. Use the host network configuration tool os-net-config to enable eswitch during deployment.

3. Enable eswitch on the sriov_config service any time you reboot the Compute node.

   Example

   $ devlink dev eswitch show pci/$(ethtool -i ens1f0 | grep bus-info \
   | cut -d ':' -f 2,3,4 | awk '{$1=$1};1')

Procedure

1. To get Broadcom chip table content, use the bcmcmd command.

   $ cl-bcmcmd l2 show

   Sample output

   mac=00:02:00:00:00:08 vlan=2000 GPORT=0x2 modid=0 port=2/xe1
   mac=00:02:00:00:00:09 vlan=2000 GPORT=0x2 modid=0 port=2/xe1 Hit

2. Inspect the Traffic Control (TC) Layer.

   $ tc -s filter show dev p5p1_1 ingress

   Sample output

   …
   filter block 94 protocol ip pref 3 flower chain 5
   filter block 94 protocol ip pref 3 flower chain 5 handle 0x2
     eth_type ipv4
     src_ip 172.0.0.1
     ip_flags nofrag
     in_hw in_hw_count 1
           action order 1: mirred (Egress Redirect to device eth4) stolen
           index 3 ref 1 bind 1 installed 364 sec used 0 sec
           Action statistics:
           Sent 253991716224 bytes 169534118 pkt (dropped 0, overlimits 0 requeues 0)
           Sent software 43711874200 bytes 30161170 pkt
           Sent hardware 210279842024 bytes 139372948 pkt
           backlog 0b 0p requeues 0
           cookie 8beddad9a0430f0457e7e78db6e0af48
           no_percpu

3. Examine the in_hw flags and the statistics in this output. The word hardware indicates that the hardware processes the network traffic. If you use tc-policy=none, you can check this output or a tcpdump to investigate when hardware or software handles the packets. You can see a corresponding log message in dmesg or in ovs-vswitch.log when the driver is unable to offload packets.

4. For Mellanox, as an example, the log entries resemble syndrome messages in dmesg.

   Sample output

   [13232.860484] mlx5_core 0000:3b:00.0: mlx5_cmd_check:756:(pid 131368): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x6b1266)

   In this example, the error code (0x6b1266) represents the following behavior:

   Sample output

   0x6B1266 |  set_flow_table_entry: pop vlan and forward to uplink is not allowed

Procedure

1. Ensure SR-IOV and VT-d are enabled on the system.
2. Enable IOMMU in Linux by adding intel_iommu=on to kernel parameters, for example, using GRUB.

Procedure

1. To enable logging on the offload modules and to get additional log information for this failure, use the following commands on the Compute node:

   ovs-appctl vlog/set dpif_netlink:file:dbg
   # Module name changed recently (check based on the version used
   ovs-appctl vlog/set netdev_tc_offloads:file:dbg [OR] ovs-appctl vlog/set netdev_offload_tc:file:dbg
   ovs-appctl vlog/set tc:file:dbg

2. Inspect the ovs-vswitchd logs again to see additional details about the issue.

   In the following example logs, the offload failed because of an unsupported attribute mark.

    2020-01-31T06:22:11.218Z|00471|dpif_netlink(handler402)|DBG|system@ovs-system: put[create] ufid:61bd016e-eb89-44fc-a17e-958bc8e45fda recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(7),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d2:f5:f3,dst=fa:16:3e:c4:a3:eb),eth_type(0x0800),ipv4(src=10.1.1.8/0.0.0.0,dst=10.1.1.31/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no),icmp(type=0/0,code=0/0), actions:set(tunnel(tun_id=0x3d,src=10.10.141.107,dst=10.10.141.124,ttl=64,tp_dst=4789,flags(df|key))),6
   
   2020-01-31T06:22:11.253Z|00472|netdev_tc_offloads(handler402)|DBG|offloading attribute pkt_mark isn't supported
   
   2020-01-31T06:22:11.257Z|00473|dpif_netlink(handler402)|ERR|failed to offload flow: Operation not supported: p6p1_5

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create an Ansible playbook YAML file, for example, /home/stack/cli-overcloud-tuned-maxpower-conf.yaml.

4. Add the following configuration to your cli-overcloud-tuned-maxpower-conf.yaml file:

   cat <<EOF > /home/stack/cli-overcloud-tuned-maxpower-conf.yaml
   {% raw %}
   ---
   #/home/stack/cli-overcloud-tuned-maxpower-conf.yaml
   - name: Overcloud Node set tuned power state
     hosts: compute-0 compute-1
     any_errors_fatal: true
     gather_facts: false
     pre_tasks:
       - name: Wait for provisioned nodes to boot
         wait_for_connection:
           timeout: 600
           delay: 10
         connection: local
     tasks:
       - name: Check the max power state for this system
         become: true
         block:
           - name: Get power states
             shell: "for s in /sys/devices/system/cpu/cpu2/cpuidle/*; do grep . $s/{name,latency}; done"
             register: _list_of_power_states
           - name: Print available power states
             debug:
               msg: "{{ _list_of_power_states.stdout.split('\n') }}"
           - name: Check for active tuned power-save profile
             stat:
               path: "/etc/tuned/active_profile"
             register: _active_profile
           - name: Check the profile
             slurp:
               path: "/etc/tuned/active_profile"
             when: _active_profile.stat.exists
             register: _active_profile_name
           - name: Print states
             debug:
               var: (_active_profile_name.content|b64decode|string)
           - name: Check the max power state for this system
             block:
               - name: Check if the cstate config is present in the conf file
                 lineinfile:
                   dest: /etc/tuned/cpu-partitioning-powersave-variables.conf
                   regexp: '^max_power_state'
                   line: 'max_power_state=cstate.name:C6'
                 register: _cstate_entry_check
   {% endraw %}
   EOF

5. Add the power save profile to your roles data file.

   For more information, see 10.2. Generating roles and image files.

6. Add the cli-overcloud-tuned-maxpower-conf.yaml playbook to your bare metal nodes definition file.

   For more information, see 10.5. Creating a bare metal nodes definition file.

7. Ensure that you have set queue size in your NIC configuration template.

   For more information, see 10.6. Creating a NIC configuration template.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Generate a new roles data file, for example, roles_data_compute_ovsdpdk.yaml, that includes the Controller and ComputeOvsDpdk roles:

   $ openstack overcloud roles generate \
   -o /home/stack/templates/roles_data_compute_ovsdpdk.yaml \
   Controller ComputeOvsDpdk

   Note

   If you are using multiple technologies in your RHOSP environment, OVS-DPDK, SR-IOV, and OVS hardware offload, you generate just one roles data file to include all the roles:

   $ openstack overcloud roles generate -o /home/stack/templates/\
   roles_data.yaml Controller ComputeOvsDpdk ComputeOvsDpdkSriov \
   Compute:ComputeOvsHwOffload

4. Optional: You can configure OVS-DPDK to enter sleep mode when no packets are being forwarded, by using the TuneD profile, cpu-partitioning-powersave.

   To configure cpu-partitioning-powersave, replace the default TuneD profile with the power saving TuneD profile, cpu-partitioning-powersave, in your generated roles data file:

   TunedProfileName: "cpu-partitioning-powersave"

   Example

   In this generated roles data file, /home/stack/templates/roles_data_compute_ovsdpdk.yaml, the default value of TunedProfileName is replaced with cpu-partitioning-powersave:

   $ sed -i \
   's/TunedProfileName:.*$/TunedProfileName: "cpu-partitioning-powersave"/' \
   /home/stack/templates/roles_data_compute_ovsdpdk.yaml

5. To generate an images file, you run the openstack tripleo container image prepare command. The following inputs are needed:

   • The roles data file that you generated in an earlier step, for example, roles_data_compute_ovsdpdk.yaml.

   • The DPDK environment file appropriate for your Networking service mechanism driver:

     • neutron-ovn-dpdk.yaml file for ML2/OVN environments.

     • neutron-ovs-dpdk.yaml file for ML2/OVS environments.

       Example

       In this example, the overcloud_images.yaml file is being generated for an ML2/OVN environment:

       $ sudo openstack tripleo container image prepare \
         --roles-file ~/templates/roles_data_compute_ovsdpdk.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dpdk.yaml \
         -e ~/containers-prepare-parameter.yaml \
         --output-env-file=/home/stack/templates/overcloud_images.yaml

6. Note the path and file name of the roles data file and the images file that you have created. You use these files later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a custom environment YAML file, for example, ovs-dpdk-overrides.yaml.

4. In the custom environment file, ensure that AggregateInstanceExtraSpecsFilter is in the list of filters for the NovaSchedulerEnabledFilters parameter that the Compute service (nova) uses to filter a node:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - AggregateInstanceExtraSpecsFilter

5. Add role-specific parameters for the OVS-DPDK Compute nodes to the custom environment file.

   Example

   parameter_defaults:
     ComputeOvsDpdkParameters:
       NeutronBridgeMappings: "dpdk:br-dpdk"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1GB hugepages=64 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39"
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "4096,4096"
       OvsDpdkMemoryChannels: "4"
       OvsDpdkCoreList: "0,20,1,21"
       NovaComputeCpuDedicatedSet: "4,6,8,10,12,14,16,18,24,26,28,30,32,34,36,38,5,7,9,11,13,15,17,19,27,29,31,33,35,37,39"
       NovaComputeCpuSharedSet: "0,20,1,21"
       OvsPmdCoreList: "2,22,3,23"

6. If you need to override any of the configuration defaults in those files, add your overrides to the custom environment file that you created in step 3.

   RHOSP director uses the following files to configure OVS-DPDK:

   • ML2/OVN deployments

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-ovn-dpdk.yaml

   • ML2/OVS deployments

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-ovs-dpdk.yaml

7. Note the path and file name of the custom environment file that you have created. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the custom environment YAML file that you created in Section 10.3, “Creating an environment file for your OVS-DPDK customizations”, or create a new one.

4. Under parameter_defaults, add the following key-value pair:

   parameter_defaults:
   ...
   
     NeutronOVSFirewallDriver: openvswitch

5. If you created a new custom environment file, note its path and file name. You use this file later when you deploy your overcloud.

6. After you deploy the overcloud, run the openstack port set command to disable the OVS firewall driver for data plane interfaces:

   $ openstack port set --no-security-group --disable-port-security ${PORT}

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as overcloud-baremetal-deploy.yaml, as instructed in Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. In the overcloud-baremetal-deploy.yaml file add a declaration to the Ansible playbook, cli-overcloud-node-kernelargs.yaml. The playbook contains kernel arguments to use when you are provisioning bare metal nodes.

   - name: ComputeOvsDpdk
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
   ...

5. If you want to set any extra Ansible variables when running the playbook, use the extra_vars property to set them.

   For more information, see Bare-metal node provisioning attributes in the Installing and managing Red Hat OpenStack Platform with director guide.

   Note

   The variables that you add to extra_vars should be the same role-specific parameters for the OVS-DPDK Compute nodes that you added to the custom environment file earlier in Create an environment file for your OVS-DPDK customizations.

   Example

   - name: ComputeOvsDpdk
     ...
     ansible_playbooks:
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
       extra_vars:
         kernel_args: 'default_hugepagesz=1GB hugepagesz=1GB hugepages=64 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39'
         tuned_isolated_cores: '2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39'
         tuned_profile: 'cpu-partitioning'
         reboot_wait_timeout: 1800
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
       extra_vars:
         pmd: '2,22,3,23'
         memory_channels: '4'
         socket_mem: '4096,4096'
         pmd_auto_lb: true
         pmd_load_threshold: "70"
         pmd_improvement_threshold: "25"
         pmd_rebal_interval: "2"

6. If you are using NIC partitioning on NVIDIA Mellanox cards, to avoid VF connectivity issues, set the Ansible variable, dpdk_extra: '-a 0000:00:00.0', which causes the allow list of PCI addresses to allow no addresses:

   Example

   - name: ComputeOvsDpdk
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
         extra_vars:
           reboot_wait_timeout: 600
           kernel_args: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-11,13-23
           tuned_profile: cpu-partitioning
           tuned_isolated_cores: 1-11,13-23
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
         extra_vars:
           memory_channels: 4
           lcore: 0,12
           pmd: 1,13,2,14,3,15
           socket_mem: 4096
           dpdk_extra: -a 0000:00:00.0
           disable_emc: false
           enable_tso: false
           revalidator: ' handler: '
           pmd_auto_lb: false
           pmd_load_threshold: ' pmd_improvement_threshold: '
           pmd_rebal_interval: ''
           nova_postcopy: true

7. Optional: You can configure OVS-DPDK to enter sleep mode when no packets are being forwarded, by using the TuneD profile, cpu-partitioning-powersave.

   To configure cpu-partitioning-powersave, add the following lines to your bare metal nodes definition file:

   ...
   
   tuned_profile: "cpu-partitioning-powersave"
   
   ...
   
     - playbook: /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/playbooks/cli-overcloud-tuned-maxpower-conf.yaml
     - playbook: /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/playbooks/overcloud-nm-config.yaml
       extra_vars:
         reboot_wait_timeout: 900
   
   ...
   
         pmd_sleep_max: "50"
   
   ...

   Example

   - name: ComputeOvsDpdk
     ...
     ansible_playbooks:
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
       extra_vars:
         kernel_args: default_hugepagesz=1GB hugepagesz=1GB hugepages=64 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
         tuned_isolated_cores: 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
         tuned_profile: cpu-partitioning
         reboot_wait_timeout: 1800
     - playbook: /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/playbooks/cli-overcloud-tuned-maxpower-conf.yaml
     - playbook: /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/playbooks/overcloud-nm-config.yaml
       extra_vars:
         reboot_wait_timeout: 900
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
       extra_vars:
         pmd: 2,22,3,23
         memory_channels: 4
         socket_mem: 4096,4096
         pmd_auto_lb: true
         pmd_load_threshold: "70"
         pmd_improvement_threshold: "25"
         pmd_rebal_interval: "2"
         pmd_sleep_max: "50"

8. Note the path and file name of the bare metal nodes definition file that you have created. You use this file later when you configure your NICs and as the input file for the overcloud node provision command when you provision your nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Copy a sample network configuration template.

   Copy a NIC configuration Jinja2 template from the examples in the /usr/share/ansible/roles/tripleo_network_config/templates/ directory. Choose the one that most closely matches your NIC requirements. Modify it as needed.

4. In your NIC configuration template, for example, single_nic_vlans.j2, add your DPDK interfaces.

   Note

   In the sample NIC configuration template, single_nic_vlans.j2, the nodes only use one single network interface as a trunk with VLANs. The native VLAN, the untagged traffic, is the control plane, and each VLAN corresponds to one of the RHOSP networks: internal API, storage, and so on.

   Example

   ...
     - type: ovs_dpdk_bond
       name: dpdkbond0
       mtu: 9000
       rx_queue: 1
       ovs_extra:
         - set Interface dpdk0 options:n_rxq_desc=4096
         - set Interface dpdk0 options:n_txq_desc=4096
         - set Interface dpdk1 options:n_rxq_desc=4096
         - set Interface dpdk1 options:n_txq_desc=4096
       members:
       - type: ovs_dpdk_port
         name: dpdk0
         driver: vfio-pci
         members:
         - type: interface
           name: nic5
       - type: ovs_dpdk_port
         name: dpdk1
         driver: vfio-pci
         members:
         - type: interface
           name: nic6
   ...

5. Add the custom network configuration template, for example, single_nic_vlans.j2, to the bare metal nodes definition file, for example, overcloud-baremetal-deploy.yaml that you created in Section 10.5, “Creating a bare metal nodes definition file”.

   Example

   - name: ComputeOvsDpdk
     count: 2
     hostname_format: compute-%index%
     defaults:
       networks:
         - network: internal_api
           subnet: internal_api_subnet
         - network: tenant
           subnet: tenant_subnet
         - network: storage
           subnet: storage_subnet
       network_config:
         template: /home/stack/templates/single_nic_vlans.j2
   ...

6. Optional: You can configure OVS-DPDK to enter sleep mode when no packets are being forwarded, by using the TuneD profile, cpu-partitioning-powersave.

   To configure cpu-partitioning-powersave, ensure that you have set the queue size in your NIC configuration template.

   Example

   ...
     - type: ovs_dpdk_bond
       name: dpdkbond0
       mtu: 9000
       rx_queue: 1
       ovs_extra:
         - set Interface dpdk0 options:n_rxq_desc=4096
         - set Interface dpdk0 options:n_txq_desc=4096
         - set Interface dpdk1 options:n_rxq_desc=4096
         - set Interface dpdk1 options:n_txq_desc=4096
       members:
       - type: ovs_dpdk_port
         name: dpdk0
         driver: vfio-pci
         members:
         - type: interface
           name: nic5
       - type: ovs_dpdk_port
         name: dpdk1
         driver: vfio-pci
         members:
         - type: interface
           name: nic6
   ...

7. Note the path and file name of the NIC configuration template that you have created. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the custom environment YAML file that you created in Section 10.3, “Creating an environment file for your OVS-DPDK customizations”, or create a new one.

4. Under parameter_defaults set the NeutronGlobalPhysnetMtu parameter.

   Example

   In this example, NeutronGlobalPhysnetMtu is set to 9000:

   parameter_defaults:
     # MTU global configuration
     NeutronGlobalPhysnetMtu: 9000

   Note

   Ensure that the OvsDpdkSocketMemory value in the network-environment.yaml file is large enough to support jumbo frames. For more information, see Memory parameters.

5. Open your NIC configuration template, for example, single_nic_vlans.j2, that you created in Section 10.6, “Creating a NIC configuration template”.

6. Set the MTU value on the bridge to the Compute node.

     -
       type: ovs_bridge
       name: br-link0
       use_dhcp: false
       members:
         -
           type: interface
           name: nic3
           mtu: 9000

7. Set the MTU values for an OVS-DPDK bond:

   - type: ovs_user_bridge
     name: br-link0
     use_dhcp: false
     members:
       - type: ovs_dpdk_bond
         name: dpdkbond0
         mtu: 9000
         rx_queue: 2
         members:
           - type: ovs_dpdk_port
             name: dpdk0
             mtu: 9000
             members:
               - type: interface
                 name: nic4
           - type: ovs_dpdk_port
             name: dpdk1
             mtu: 9000
             members:
               - type: interface
                 name: nic5

8. Note the paths and file names of your NIC configuration template and your custom environment file. You use these files later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the NIC configuration template, such as single_nic_vlans.j2, that you created in Section 10.6, “Creating a NIC configuration template”.

4. Set the number of queues for interfaces in OVS-DPDK on the Compute node:

   - type: ovs_user_bridge
     name: br-link0
     use_dhcp: false
     members:
       - type: ovs_dpdk_bond
         name: dpdkbond0
         mtu: 9000
         rx_queue: 2
         members:
           - type: ovs_dpdk_port
             name: dpdk0
             mtu: 9000
             members:
               - type: interface
                 name: nic4
           - type: ovs_dpdk_port
             name: dpdk1
             mtu: 9000
             members:
               - type: interface
                 name: nic5

5. Note the path and file name of the NIC configuration template. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Set parameters in the bare metal nodes definition file that you created in Section 10.5, “Creating a bare metal nodes definition file”, for example overcloud-baremetal-deploy.yaml:

   pmd_auto_lb

   Set to true to enable PMD automatic load balancing.

   pmd_load_threshold

   Percentage of processing cycles that one of the PMD threads must use consistently before triggering the PMD load balance. Integer, range 0-100.

   pmd_improvement_threshold

   Minimum percentage of evaluated improvement across the non-isolated PMD threads that triggers a PMD auto load balance. Integer, range 0-100.

   To calculate the estimated improvement, a dry run of the reassignment is done and the estimated load variance is compared with the current variance. The default is 25%.

   pmd_rebal_interval

   Minimum time in minutes between two consecutive PMD Auto Load Balance operations. Range 0-20,000 minutes.

   Configure this value to prevent triggering frequent reassignments where traffic patterns are changeable. For example, you might trigger a reassignment once every 10 minutes or once every few hours.

   Example

   ansible_playbooks:
   …
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
     extra_vars:
      …
      pmd_auto_lb: true
      pmd_load_threshold: "70"
      pmd_improvement_threshold: "25"
      pmd_rebal_interval: "2"

4. Open the custom environment YAML file that you created in Section 10.3, “Creating an environment file for your OVS-DPDK customizations”, or create a new one.

5. In the custom environment file, add the same bare metal node pre-provisioning values that you set in step 3. Use these equivalent parameters:

   OvsPmdAutoLb

   Heat equivalent of pmd_auto_lb.

   Set to true to enable PMD automatic load balancing.

   OvsPmdLoadThreshold

   Heat equivalent of pmd_load_threshold.

   Percentage of processing cycles that one of the PMD threads must use consistently before triggering the PMD load balance. Integer, range 0-100.

   OvsPmdImprovementThreshold

   Heat equivalent of pmd_improvement_threshold.

   Minimum percentage of evaluated improvement across the non-isolated PMD threads that triggers a PMD auto load balance. Integer, range 0-100.

   To calculate the estimated improvement, a dry run of the reassignment is done and the estimated load variance is compared with the current variance. The default is 25%.

   OvsPmdRebalInterval

   Heat equivalent of pmd_rebal_interval.

   Minimum time in minutes between two consecutive PMD Auto Load Balance operations. Range 0-20,000 minutes.

   Configure this value to prevent triggering frequent reassignments where traffic patterns are changeable. For example, you might trigger a reassignment once every 10 minutes or once every few hours.

   Example

   parameter_merge_strategies:
     ComputeOvsDpdkSriovParameters:merge
   …
   parameter_defaults:
     ComputeOvsDpdkSriovParameters:
      …
      OvsPmdAutoLb: true
      OvsPmdLoadThreshold: 70
      OvsPmdImprovementThreshold: 25
      OvsPmdRebalInterval: 2

6. Note the paths and file names of your NIC configuration template and your custom environment file. You use these files later when you provision your bare metal nodes and deploy your overcloud.

Next steps

1. Provision your networks and VIPs.

2. Provision your bare metal nodes.

   Ensure that you use your bare metal nodes definition file, such as overcloud-baremetal-deploy.yaml, as the input for running the provision command.

3. Proceed to Section 10.10, “Deploying an OVS-DPDK overcloud”.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Enter the openstack overcloud deploy command.

   It is important to list the inputs to the openstack overcloud deploy command in a particular order. The general rule is to specify the default heat template files first followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Add your inputs to the openstack overcloud deploy command in the following order:

   1. A custom network definition file that contains the specifications for your SR-IOV network on the overcloud, for example, network-data.yaml.

      For more information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   2. A roles file that contains the Controller and ComputeOvsDpdk roles that RHOSP director uses to deploy your SR-IOV environment.

      Example: roles_data_compute_ovsdpdk.yaml

      For more information, see Section 10.2, “Generating roles and image files”.

   3. The output file from provisioning your overcloud networks.

      Example: overcloud-networks-deployed.yaml

      For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   4. The output file from provisioning your overcloud VIPs.

      Example: overcloud-vip-deployed.yaml

      For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   5. The output file from provisioning bare-metal nodes.

      Example: overcloud-baremetal-deployed.yaml

      For more information, see:

      • Section 10.9, “Configuring DPDK parameters for node provisioning”.
      • Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   6. An images file that director uses to determine where to obtain container images and how to store them.

      Example: overcloud_images.yaml

      For more information, see Section 10.2, “Generating roles and image files”.

   7. An environment file for the Networking service (neutron) mechanism driver and router scheme that your environment uses:

      • ML2/OVN

        • Distributed virtual routing (DVR): neutron-ovn-dvr-ha.yaml
        • Centralized virtual routing: neutron-ovn-ha.yaml

      • ML2/OVS

        • Distributed virtual routing (DVR): neutron-ovs-dvr.yaml
        • Centralized virtual routing: neutron-ovs.yaml

   8. An environment file for OVS-DPDK, depending on your mechanism driver:

      • ML2/OVN

        • neutron-ovn-dpdk.yaml

      • ML2/OVS

        • neutron-ovs-dpdk.yaml

          Note

          If you also have an SR-IOV environment, and want to locate SR-IOV and OVS-DPDK instances on the same node, include the following environment files in your deployment script:

          • ML2/OVN

            neutron-ovn-sriov.yaml

          • ML2/OVS

            neutron-sriov.yaml

   9. One or more custom environment files that contain your configuration for:

      • overrides of default configuration values for the OVS-DPDK environment.
      • firewall as a virtual network function (VNF).

      • maximum transmission unit (MTU) value for jumbo frames.

        Example: ovs-dpdk-overrides.yaml

        For more information, see:

      • Section 10.3, “Creating an environment file for your OVS-DPDK customizations”.
      • Section 10.4, “Configuring a firewall for security groups”.

      • Section 10.7, “Setting the MTU value for OVS-DPDK interfaces”.

        Example

        This excerpt from a sample openstack overcloud deploy command demonstrates the proper ordering of the command’s inputs for an OVS-DPDK, ML2/OVN environment that uses DVR:

        $ openstack overcloud deploy \
        --log-file overcloud_deployment.log \
        --templates /usr/share/openstack-tripleo-heat-templates/ \
        --stack overcloud \
        -n /home/stack/templates/network_data.yaml \
        -r /home/stack/templates/roles_data_compute_ovsdpdk.yaml \
        -e /home/stack/templates/overcloud-networks-deployed.yaml \
        -e /home/stack/templates/overcloud-vip-deployed.yaml \
        -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
        -e /home/stack/templates/overcloud-images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-dvr-ha.yaml
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-dpdk.yaml \
        -e /home/stack/templates/ovs-dpdk-overrides.yaml

4. Run the openstack overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.

1. Create an aggregate group, and add relevant hosts for OVS-DPDK. Define metadata, for example dpdk=true, that matches defined flavor metadata.

    # openstack aggregate create dpdk_group
    # openstack aggregate add host dpdk_group [compute-host]
    # openstack aggregate set --property dpdk=true dpdk_group

   Note

   Pinned CPU instances can be located on the same Compute node as unpinned instances. For more information, see Configuring CPU pinning on Compute nodes in Configuring the Compute service for instance creation.

2. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

3. Set flavor properties. Note that the defined metadata, dpdk=true, matches the defined metadata in the DPDK aggregate.

   # openstack flavor set <flavor> --property dpdk=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB --property hw:emulator_threads_policy=isolate

   For details about the emulator threads policy for performance improvements, see Configuring emulator threads in Configuring the Compute service for instance creation.

4. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

5. Optional: If you use multiqueue with OVS-DPDK, set the hw_vif_multiqueue_enabled property on the image that you want to use to create a instance:

   # openstack image set --property hw_vif_multiqueue_enabled=true <image>

6. Deploy an instance.

   # openstack server create --flavor <flavor> --image <glance image> --nic net-id=<network ID> <server_name>

1. Review the bridge configuration, and confirm that the bridge has datapath_type=netdev.

   # ovs-vsctl list bridge br0
   _uuid               : bdce0825-e263-4d15-b256-f01222df96f3
   auto_attach         : []
   controller          : []
   datapath_id         : "00002608cebd154d"
   datapath_type       : netdev
   datapath_version    : "<built-in>"
   external_ids        : {}
   fail_mode           : []
   flood_vlans         : []
   flow_tables         : {}
   ipfix               : []
   mcast_snooping_enable: false
   mirrors             : []
   name                : "br0"
   netflow             : []
   other_config        : {}
   ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
   protocols           : []
   rstp_enable         : false
   rstp_status         : {}
   sflow               : []
   status              : {}
   stp_enable          : false

2. Optionally, you can view logs for errors, such as if the container fails to start.

   # less /var/log/containers/neutron/openvswitch-agent.log

3. Confirm that the Poll Mode Driver CPU mask of the ovs-dpdk is pinned to the CPUs. In case of hyper threading, use sibling CPUs.

   For example, to check the sibling of CPU4, run the following command:

   # cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
   4,20

   The sibling of CPU4 is CPU20, therefore proceed with the following command:

   # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

   Display the status:

   # tuna -t ovs-vswitchd -CP
   thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
   3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
   3219   OTHER 	0    	6     	1        	0   	handler24
   3220   OTHER 	0    	6     	1        	0   	handler21
   3221   OTHER 	0    	6     	1        	0   	handler22
   3222   OTHER 	0    	6     	1        	0   	handler23
   3223   OTHER 	0    	6     	1        	0   	handler25
   3224   OTHER 	0    	6     	1        	0   	handler26
   3225   OTHER 	0    	6     	1        	0   	handler27
   3226   OTHER 	0    	6     	1        	0   	handler28
   3227   OTHER 	0    	6     	2        	0   	handler31
   3228   OTHER 	0    	6     	2        	4   	handler30
   3229   OTHER 	0    	6     	2        	5   	handler32
   3230   OTHER 	0    	6	953538      	431   revalidator29
   3231   OTHER 	0    	6   1424258      	976   revalidator33
   3232   OTHER 	0    	6   1424693      	836   revalidator34
   3233   OTHER 	0    	6	951678      	503   revalidator36
   3234   OTHER 	0    	6   1425128      	498   revalidator35
   *3235   OTHER 	0    	4	151123       	51       	pmd37*
   *3236   OTHER 	0   	20	298967       	48       	pmd38*
   3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
   3165   OTHER 	0    	6	237634        	0   vhost_thread1
   3166   OTHER 	0    	6  	3665        	0       	urcu2

Procedure

1. Deploy an overcloud with NovaComputeCpuSharedSet defined for a given role. The value of NovaComputeCpuSharedSet applies to the cpu_shared_set parameter in the nova.conf file for hosts within that role.

   parameter_defaults:
       ComputeOvsDpdkParameters:
           NovaComputeCpuSharedSet: "0-1,16-17"
           NovaComputeCpuDedicatedSet: "2-15,18-31"

2. Create a flavor to build instances with emulator threads separated into a shared pool.

   openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <vcpus> <flavor>

3. Add the hw:emulator_threads_policy extra specification, and set the value to share. Instances created with this flavor will use the instance CPUs defined in the cpu_share_set parameter in the nova.conf file.

   openstack flavor set <flavor> --property hw:emulator_threads_policy=share

Verification

1. Identify the host and name for a given instance.

   openstack server show <instance_id>

2. Use SSH to log on to the identified host as tripleo-admin.

   ssh tripleo-admin@compute-1
   [compute-1]$ sudo virsh dumpxml instance-00001 | grep `'emulatorpin cpuset'`

1. Add the NeutronPhysicalDevMappings parameter in the parameter_defaults section to link between the logical network name and the physical interface.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2

2. Add the new property, trusted, to the SR-IOV parameters.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2
     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1572"
         physical_network: "sriov2"
         trusted: "true"

   Note

   You must include double quotation marks around the value "true".

1. Create a network of type vlan.

   openstack network create trusted_vf_network  --provider-network-type vlan \
    --provider-segment 111 --provider-physical-network sriov2 \
    --external --disable-port-security

2. Create a subnet.

   openstack subnet create --network trusted_vf_network \
     --ip-version 4 --subnet-range 192.168.111.0/24 --no-dhcp \
    subnet-trusted_vf_network

3. Create a port. Set the vnic-type option to direct, and the binding-profile option to true.

   openstack port create --network sriov111 \
   --vnic-type direct --binding-profile trusted=true \
   sriov111_port_trusted

4. Create an instance, and bind it to the previously-created trusted port.

   openstack server create --image rhel --flavor dpdk  --network internal --port trusted_vf_network_port_trusted --config-drive True --wait rhel-dpdk-sriov_trusted

1. On the compute node that you created the instance, enter the following command:

   # ip link
   7: p5p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether b4:96:91:1c:40:fa brd ff:ff:ff:ff:ff:ff
       vf 6 MAC fa:16:3e:b8:91:c2, vlan 111, spoof checking off, link-state auto, trust on, query_rss off
       vf 7 MAC fa:16:3e:84:cf:c8, vlan 111, spoof checking off, link-state auto, trust off, query_rss off

2. Verify that the trust status of the VF is trust on. The example output contains details of an environment that contains two ports. Note that vf 6 contains the text trust on.
3. You can disable spoof checking if you set port_security_enabled: false in the Networking service (neutron) network, or if you include the argument --disable-port-security when you run the openstack port create command.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a custom environment YAML file and under parameter_defaults add the following definitions to increase the RX and TX queue size:

   parameter_defaults:
     NovaLibvirtRxQueueSize: 1024
     NovaLibvirtTxQueueSize: 1024

4. Run the deployment command and include the core heat templates, other environment files, the environment file that contains your RX and TX queue size changes:

   Example

   $ openstack overcloud deploy --templates \
   -e <other_environment_files> \
   -e /home/stack/my_tx-rx_queue_sizes.yaml

Verification

1. Observe the values for RX queue size and TX queue size in the nova.conf file.

   $ egrep "^[rt]x_queue_size" /var/lib/config-data/puppet-generated/\
   nova_libvirt/etc/nova/nova.conf

   You should see the following:

   rx_queue_size=1024
   tx_queue_size=1024

2. Check the values for RX queue size and TX queue size in the VM instance XML file generated by libvirt on the Compute host:

   1. Create a new instance.

   2. Obtain the Compute host and and instance name:

      $ openstack server show testvm-queue-sizes -c OS-EXT-SRV-ATTR:\
      hypervisor_hostname -c OS-EXT-SRV-ATTR:instance_name

      Sample output

      You should see output similar to the following:

      +-------------------------------------+------------------------------------+
      | Field                               | Value                              |
      +-------------------------------------+------------------------------------+
      | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.sales      |
      | OS-EXT-SRV-ATTR:instance_name       | instance-00000059                  |
      +-------------------------------------+------------------------------------+

   3. Log into the Compute host and dump the instance definition.

      Example

      $ podman exec nova_libvirt virsh dumpxml instance-00000059

      Sample output

      You should see output similar to the following:

      ...
         <interface type='vhostuser'>
           <mac address='56:48:4f:4d:5e:6f'/>
           <source type='unix' path='/tmp/vhost-user1' mode='server'/>
           <model type='virtio'/>
           <driver name='vhost' rx_queue_size='1024'   tx_queue_size='1024' />
           <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
         </interface>
      ...

Procedure

1. Set a new NeutronPhysnetNUMANodesMapping parameter to map the physical network to the NUMA node that you associate with the physical network.

2. If you use tunnels, such as VxLAN or GRE, you must also set the NeutronTunnelNUMANodes parameter.

   parameter_defaults:
     NeutronPhysnetNUMANodesMapping: {<physnet_name>: [<NUMA_NODE>]}
     NeutronTunnelNUMANodes: <NUMA_NODE>,<NUMA_NODE>

   Example

   Here is an example with two physical networks tunneled to NUMA node 0:

   • one project network associated with NUMA node 0

   • one management network without any affinity

     parameter_defaults:
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysnetNUMANodesMapping: {tenant: [1], mgmt: [0,1]}
       NeutronTunnelNUMANodes: 0

   • In this example, assign the physnet of the device named eno2 to NUMA number 0.

     # ethtool -i eno2
     bus-info: 0000:18:00.1
     
     # cat /sys/devices/pci0000:16/0000:16:02.0/0000:18:00.1/numa_node
     0

     Observe the physnet settings in the example heat template:

     NeutronBridgeMappings: 'physnet1:br-physnet1'
     NeutronPhysnetNUMANodesMapping: {physnet1: [0] }
     
     - type: ovs_user_bridge
                     name: br-physnet1
                     mtu: 9000
                     members:
                       - type: ovs_dpdk_port
                         name: dpdk2
                         members:
                           - type: interface
                             name: eno2

1. Observe the configuration in the file /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf:

   [neutron_physnet_tenant]
   numa_nodes=1
   [neutron_tunnel]
   numa_nodes=1

2. Confirm the new configuration with the lscpu command:

   $ lscpu

3. Launch a VM with the NIC attached to the appropriate network.

Building the real-time image

1. Install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud) [stack@undercloud-0 ~]$ sudo dnf install libguestfs-tools

   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

2. Extract the images:

   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/overcloud-hardened-uefi-full-17.1.x86_64.tar
   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent-17.1.x86_64.tar

3. Copy the default image:

   (undercloud) [stack@undercloud-0 ~]$ cp overcloud-hardened-uefi-full.qcow2 overcloud-realtime-compute.qcow2

4. Register your image to enable Red Hat repositories relevant to your customizations. Replace [username] and [password] with valid credentials in the following example.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager register --username=[username] --password=[password]' \
   subscription-manager release --set 9.0

   Note

   For security, you can remove credentials from the history file if they are used on the command prompt. You can delete individual lines in history using the history -d command followed by the line number.

5. Find a list of pool IDs from your account’s subscriptions, and attach the appropriate pool ID to your image.

   sudo subscription-manager list --all --available | less
   ...
   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager attach --pool [pool-ID]'

6. Add the repositories necessary for Red Hat OpenStack Platform with NFV.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms \
   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.9-for-rhel-9-x86_64-rpms \
   --enable=rhel-9-for-x86_64-nfv-rpms
   --enable=fast-datapath-for-rhel-9-x86_64-rpms'

7. Create a script to configure real-time capabilities on the image.

   (undercloud) [stack@undercloud-0 ~]$ cat <<'EOF' > rt.sh
     #!/bin/bash
   
     set -eux
   
     dnf -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
     grubby --set-default /boot/vmlinuz*rt*
     EOF

8. Run the script to configure the real-time image:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

   Note

   If you see the following line in the rt.sh script output, "grubby fatal error: unable to find a suitable template", you can ignore this error.

9. Examine the virt-customize.log file that resulted from the previous command, to check that the packages installed correctly using the rt.sh script .

   (undercloud) [stack@undercloud-0 ~]$ cat virt-customize.log | grep Verifying
   
     Verifying  : kernel-3.10.0-957.el7.x86_64                                 1/1
     Verifying  : 10:qemu-kvm-tools-rhev-2.12.0-18.el7_6.1.x86_64              1/8
     Verifying  : tuned-profiles-realtime-2.10.0-6.el7_6.3.noarch              2/8
     Verifying  : linux-firmware-20180911-69.git85c5d90.el7.noarch             3/8
     Verifying  : tuned-profiles-nfv-host-2.10.0-6.el7_6.3.noarch              4/8
     Verifying  : kernel-rt-kvm-3.10.0-957.10.1.rt56.921.el7.x86_64            5/8
     Verifying  : tuna-0.13-6.el7.noarch                                       6/8
     Verifying  : kernel-rt-3.10.0-957.10.1.rt56.921.el7.x86_64                7/8
     Verifying  : rt-setup-2.0-6.el7.x86_64                                    8/8

10. Relabel SELinux:

    (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

11. Extract vmlinuz and initrd:

    (undercloud) [stack@undercloud-0 ~]$ mkdir image
    (undercloud) [stack@undercloud-0 ~]$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud) [stack@undercloud-0 ~]$ guestunmount image

    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

12. Upload the image:

    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2

1. Create an RT-KVM flavor on the overcloud:

   $ openstack flavor create r1.small --id 99 --ram 4096 --disk 20 --vcpus 4
   $ openstack flavor set --property hw:cpu_policy=dedicated 99
   $ openstack flavor set --property hw:cpu_realtime=yes 99
   $ openstack flavor set --property hw:mem_page_size=1GB 99
   $ openstack flavor set --property hw:cpu_realtime_mask="^0-1" 99
   $ openstack flavor set --property hw:cpu_emulator_threads=isolate 99

2. Launch an RT-KVM instance:

   $ openstack server create  --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt

3. To verify that the instance uses the assigned emulator threads, run the following command:

   $ virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

1. Under parameter_defaults, set the tunnel type to vxlan, and the network type to vxlan,vlan:

   NeutronTunnelTypes: 'vxlan'
   NeutronNetworkType: 'vxlan,vlan'

2. Under parameters_defaults, set the bridge mapping:

   # The OVS logical->physical bridge mappings to use.
   NeutronBridgeMappings:
     - dpdk-mgmt:br-link0

3. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role:

     ##########################
     # OVS DPDK configuration #
     ##########################
     ComputeOvsDpdkSriovParameters:
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2-19,22-39"
       NovaComputeCpuDedicatedSet: ['4-19,24-39']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "3072,1024"
       OvsDpdkMemoryChannels: "4"
       OvsPmdCoreList: "2,22,3,23"
       NovaComputeCpuSharedSet: [0,20,1,21]
       NovaLibvirtRxQueueSize: 1024
       NovaLibvirtTxQueueSize: 1024

   Note

   To prevent failures during guest creation, assign at least one CPU with sibling thread on each NUMA node. In the example, the values for the OvsPmdCoreList parameter denote cores 2 and 22 from NUMA 0, and cores 3 and 23 from NUMA 1.

   Note

   These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the virtual machines is the boot parameter minus the OvsDpdkSocketMemory.

   You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

   Note

   OvsDpdkMemoryChannels is a required setting for this procedure. For optimum operation, ensure you deploy DPDK with appropriate parameters and values.

4. Configure the role-specific parameters for SR-IOV:

     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.0"
         trusted: "true"
         physical_network: "sriov-1"
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.1"
         trusted: "true"
         physical_network: "sriov-2"

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic2
         primary: true

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageMgmtNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageMgmtIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: ExternalNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: ExternalIpSubnet
       routes:
       - default: true
         next_hop:
           get_param: ExternalInterfaceDefaultRoute

3. Create the OVS bridge to access neutron-dhcp-agent and neutron-metadata-agent services.

     - type: ovs_bridge
       name: br-link0
       use_dhcp: false
       mtu: 9000
       members:
       - type: interface
         name: nic3
         mtu: 9000
       - type: vlan
         vlan_id:
           get_param: TenantNetworkVlanID
         mtu: 9000
         addresses:
         - ip_netmask:
             get_param: TenantIpSubnet

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic3
         primary: true
       - type: interface
         name: nic4

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet

3. Set a bridge with a DPDK port to link to the controller.

     - type: ovs_user_bridge
       name: br-link0
       use_dhcp: false
       ovs_extra:
         - str_replace:
             template: set port br-link0 tag=_VLAN_TAG_
             params:
               _VLAN_TAG_:
                  get_param: TenantNetworkVlanID
       addresses:
         - ip_netmask:
             get_param: TenantIpSubnet
       members:
         - type: ovs_dpdk_bond
           name: dpdkbond0
           mtu: 9000
           rx_queue: 2
           members:
             - type: ovs_dpdk_port
               name: dpdk0
               members:
                 - type: interface
                   name: nic7
             - type: ovs_dpdk_port
               name: dpdk1
               members:
                 - type: interface
                   name: nic8

   Note

   To include multiple DPDK devices, repeat the type code section for each DPDK device that you want to add.

   Note

   When using OVS-DPDK, all bridges on the same Compute node must be of type ovs_user_bridge. Red Hat OpenStack Platform does not support both ovs_bridge and ovs_user_bridge located on the same node.