Chapter 12. Enabling RT-KVM for NFV Workloads

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   [stack@director ~]$ source ~/stackrc

3. Based on the /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml file, create a compute-real-time.yaml environment file that sets the parameters for the ComputeRealTime role.

4. Generate a new roles data file named roles_data_rt.yaml that includes the ComputeRealTime role, along with any other roles that you need for the overcloud. The following example generates the roles data file roles_data_rt.yaml, which includes the roles Controller, Compute, and ComputeRealTime:

   (undercloud)$ openstack overcloud roles generate \
   -o /home/stack/templates/roles_data_rt.yaml \
   ComputeRealTime Compute Controller

5. Update the roles_data_rt.yaml file for the ComputeRealTime role:

   ###################################################
   # Role: ComputeRealTime                                                         #
   ###################################################
   - name: ComputeRealTime
     description: |
       Real Time Compute Node role
     CountDefault: 1
     # Create external Neutron bridge
     tags:
       - compute
       - external_bridge
     networks:
       InternalApi:
         subnet: internal_api_subnet
       Tenant:
         subnet: tenant_subnet
       Storage:
         subnet: storage_subnet
     HostnameFormatDefault: '%stackname%-computert-%index%'
     deprecated_nic_config_name: compute-rt.yaml

6. Register the ComputeRealTime nodes for the overcloud by adding them to your node definition template: node.json or node.yaml.

   For more information, see Registering nodes for the overcloud in Installing and managing Red Hat OpenStack Platform with director.

7. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in Installing and managing Red Hat OpenStack Platform with director.

8. Tag each bare-metal node that you want to designate for ComputeRealTime with a custom ComputeRealTime resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.RTCOMPUTE <node>

   Replace <node> with the name or UUID of the bare-metal node.

9. Add the ComputeRealTime role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
     ...
   - name: Compute
     count: 3
     ...
   - name: ComputeRealTime
     count: 1
     defaults:
       resource_class: baremetal.RTCOMPUTE
       network_config:
         template: /home/stack/templates/nic-config/<role_topology_file>

   • Replace <role_topology_file> with the name of the topology file to use for the ComputeRealTime role, for example, myRoleTopology.j2. You can reuse an existing network topology or create a new custom network interface template for the role.

     For more information, see Defining custom network interface templates in Installing and managing Red Hat OpenStack Platform with director. To use the default network definition settings, do not include network_config in the role definition.

     For more information about the properties you can use to configure node attributes in your node definition file, see Bare-metal node provisioning attributes in Installing and managing Red Hat OpenStack Platform with director.

     For an example node definition file, see Example node definition file in Installing and managing Red Hat OpenStack Platform with director.

10. Create the following Ansible playbook to configure the kernel during the node provisioning, and save the playbook as /home/stack/templates/fix_rt_kernel.yaml:

    # RealTime KVM fix until BZ #2122949 is closed-
    - name: Fix RT Kernel
      hosts: allovercloud
      any_errors_fatal: true
      gather_facts: false
      vars:
        reboot_wait_timeout: 900
      pre_tasks:
        - name: Wait for provisioned nodes to boot
          wait_for_connection:
            timeout: 600
            delay: 10
      tasks:
        - name: Fix bootloader entry
          become: true
          shell: |-
            set -eux
            new_entry=$(grep saved_entry= /boot/grub2/grubenv | sed -e s/saved_entry=//)
            source /etc/default/grub
            sed -i "s/options.*/options root=$GRUB_DEVICE ro $GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT/" /boot/loader/entries/$(</etc/machine-id)$new_entry.conf
            cp -f /boot/grub2/grubenv /boot/efi/EFI/redhat/grubenv
      post_tasks:
        - name: Configure reboot after new kernel
          become: true
          reboot:
            reboot_timeout: "{{ reboot_wait_timeout }}"
          when: reboot_wait_timeout is defined

11. Include /home/stack/templates/fix_rt_kernel.yaml as a playbook in the ComputeOvsDpdkSriovRT role definition in your node provisioning file:

    - name: ComputeOvsDpdkSriovRT
      ...
      ansible_playbooks:
        - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
          extra_vars:
            kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=64 iommu=pt intel_iommu=on tsx=off isolcpus=2-19,22-39"
            reboot_wait_timeout: 900
            tuned_profile: "cpu-partitioning"
            tuned_isolated_cores: "2-19,22-39"
            defer_reboot: true
        - playbook: /home/stack/templates/fix_rt_kernel.yaml
          extra_vars:
            reboot_wait_timeout: 1800

    For more information about the properties you can use to configure node attributes in your node definition file, see Bare-metal node provisioning attributes in Installing and managing Red Hat OpenStack Platform with director.

    For an example node definition file, see Example node definition file in Installing and managing Red Hat OpenStack Platform with director.

12. Provision the new nodes for your role:

    (undercloud)$ openstack overcloud node provision \
    [--stack <stack> \ ]
    [--network-config \]
    --output <deployment_file> \
    /home/stack/templates/overcloud-baremetal-deploy.yaml

    • Optional: Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. The default is overcloud.
    • Optional: Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

13. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

14. If you ran the provisioning command without the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
      ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
      ComputeAMDSEVNetworkConfigTemplate: /home/stack/templates/nic-configs/<rt_compute>.j2
      ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Replace <rt_compute> with the name of the file that contains the network topology of the ComputeRealTime role, for example, computert.yaml to use the default network topology.

15. Add your environment file to the stack with your other environment files and deploy the overcloud:

    (undercloud)$ openstack overcloud deploy --templates \
     -r /home/stack/templates/roles_data_rt.yaml \
     -e /home/stack/templates/overcloud-baremetal-deployed.yaml
     -e /home/stack/templates/node-info.yaml \
     -e [your environment files] \
     -e /home/stack/templates/compute-real-time.yaml

1. Under parameter_defaults, set the tunnel type to vxlan, and the network type to vxlan,vlan:

   NeutronTunnelTypes: 'vxlan'
   NeutronNetworkType: 'vxlan,vlan'

2. Under parameters_defaults, set the bridge mapping:

   # The OVS logical->physical bridge mappings to use.
   NeutronBridgeMappings:
     - dpdk-mgmt:br-link0

3. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role:

     ##########################
     # OVS DPDK configuration #
     ##########################
     ComputeOvsDpdkSriovParameters:
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2-19,22-39"
       NovaComputeCpuDedicatedSet: ['4-19,24-39']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "3072,1024"
       OvsDpdkMemoryChannels: "4"
       OvsPmdCoreList: "2,22,3,23"
       NovaComputeCpuSharedSet: [0,20,1,21]
       NovaLibvirtRxQueueSize: 1024
       NovaLibvirtTxQueueSize: 1024

   Note

   To prevent failures during guest creation, assign at least one CPU with sibling thread on each NUMA node. In the example, the values for the OvsPmdCoreList parameter denote cores 2 and 22 from NUMA 0, and cores 3 and 23 from NUMA 1.

   Note

   These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the virtual machines is the boot parameter minus the OvsDpdkSocketMemory.

   You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

   Note

   OvsDpdkMemoryChannels is a required setting for this procedure. For optimum operation, ensure you deploy DPDK with appropriate parameters and values.

4. Configure the role-specific parameters for SR-IOV:

     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.0"
         trusted: "true"
         physical_network: "sriov-1"
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.1"
         trusted: "true"
         physical_network: "sriov-2"