Chapter 9. Tuning a Red Hat OpenStack Platform environment

1. Deploy an overcloud with NovaComputeCpuSharedSet defined for a given role. The value of NovaComputeCpuSharedSet applies to the cpu_shared_set parameter in the nova.conf file for hosts within that role.

   parameter_defaults:
       ComputeOvsDpdkParameters:
           OvsDpdkCoreList: “0-1,16-17”
           NovaComputeCpuSharedSet: “0-1,16-17”
           NovaComputeCpuDedicatedSet: “2-15,18-31”

   Copy to Clipboard Toggle word wrap

2. Create a flavor to build instances with emulator threads separated into a shared pool.

   openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <vcpus> <flavor>

   Copy to Clipboard Toggle word wrap

3. Add the hw:emulator_threads_policy extra specification, and set the value to share. Instances created with this flavor will use the instance CPUs defined in the cpu_share_set parameter in the nova.conf file.

   openstack flavor set <flavor> --property hw:emulator_threads_policy=share

   Copy to Clipboard Toggle word wrap

1. Identify the host and name for a given instance.

   openstack server show <instance_id>

   Copy to Clipboard Toggle word wrap

2. Use SSH to log on to the identified host as heat-admin.

   ssh heat-admin@compute-1
   [compute-1]$ sudo virsh dumpxml instance-00001 | grep `'emulatorpin cpuset'`

   Copy to Clipboard Toggle word wrap

Building the real-time image

1. Install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud) [stack@undercloud-0 ~]$ sudo dnf install libguestfs-tools

   Copy to Clipboard Toggle word wrap
   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

   Copy to Clipboard Toggle word wrap

2. Extract the images:

   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/overcloud-full.tar
   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar

   Copy to Clipboard Toggle word wrap

3. Copy the default image:

   (undercloud) [stack@undercloud-0 ~]$ cp overcloud-full.qcow2 overcloud-realtime-compute.qcow2

   Copy to Clipboard Toggle word wrap

4. Register your image to enable Red Hat repositories relevant to your customizations. Replace [username] and [password] with valid credentials in the following example.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager register --username=[username] --password=[password]' \
   subscription-manager release --set 8.1

   Copy to Clipboard Toggle word wrap
   Note

   For security, you can remove credentials from the history file if they are used on the command prompt. You can delete individual lines in history using the history -d command followed by the line number.

5. Find a list of pool IDs from your account’s subscriptions, and attach the appropriate pool ID to your image.

   sudo subscription-manager list --all --available | less
   ...
   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager attach --pool [pool-ID]'

   Copy to Clipboard Toggle word wrap

6. Add the repositories necessary for Red Hat OpenStack Platform with NFV.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms \
   --enable=rhel-8-for-x86_64-appstream-eus-rpms \
   --enable=rhel-8-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.8-for-rhel-8-x86_64-rpms \
   --enable=openstack-16-for-rhel-8-x86_64-rpms \
   --enable=rhel-8-for-x86_64-nfv-rpms \
   --enable=advanced-virt-for-rhel-8-x86_64-rpms \
   --enable=fast-datapath-for-rhel-8-x86_64-rpms'

   Copy to Clipboard Toggle word wrap

7. Create a script to configure real-time capabilities on the image.

   (undercloud) [stack@undercloud-0 ~]$ cat <<'EOF' > rt.sh
     #!/bin/bash
   
     set -eux
   
     dnf -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
     EOF

   Copy to Clipboard Toggle word wrap

8. Run the script to configure the real-time image:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

   Copy to Clipboard Toggle word wrap
   Note

   If you see the following line in the rt.sh script output, "grubby fatal error: unable to find a suitable template", you can ignore this error.

9. Examine the virt-customize.log file that resulted from the previous command, to check that the packages installed correctly using the rt.sh script .

   (undercloud) [stack@undercloud-0 ~]$ cat virt-customize.log | grep Verifying
   
     Verifying  : kernel-3.10.0-957.el7.x86_64                                 1/1
     Verifying  : 10:qemu-kvm-tools-rhev-2.12.0-18.el7_6.1.x86_64              1/8
     Verifying  : tuned-profiles-realtime-2.10.0-6.el7_6.3.noarch              2/8
     Verifying  : linux-firmware-20180911-69.git85c5d90.el7.noarch             3/8
     Verifying  : tuned-profiles-nfv-host-2.10.0-6.el7_6.3.noarch              4/8
     Verifying  : kernel-rt-kvm-3.10.0-957.10.1.rt56.921.el7.x86_64            5/8
     Verifying  : tuna-0.13-6.el7.noarch                                       6/8
     Verifying  : kernel-rt-3.10.0-957.10.1.rt56.921.el7.x86_64                7/8
     Verifying  : rt-setup-2.0-6.el7.x86_64                                    8/8

   Copy to Clipboard Toggle word wrap

10. Relabel SELinux:

    (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

    Copy to Clipboard Toggle word wrap

11. Extract vmlinuz and initrd:

    (undercloud) [stack@undercloud-0 ~]$ mkdir image
    (undercloud) [stack@undercloud-0 ~]$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud) [stack@undercloud-0 ~]$ guestunmount image

    Copy to Clipboard Toggle word wrap
    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

12. Upload the image:

    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2

    Copy to Clipboard Toggle word wrap

1. Create an RT-KVM flavor on the overcloud:

   # openstack flavor create  r1.small 99 4096 20 4
   # openstack flavor set --property hw:cpu_policy=dedicated 99
   # openstack flavor set --property hw:cpu_realtime=yes 99
   # openstack flavor set --property hw:mem_page_size=1GB 99
   # openstack flavor set --property hw:cpu_realtime_mask="^0-1" 99
   # openstack flavor set --property hw:cpu_emulator_threads=isolate 99

   Copy to Clipboard Toggle word wrap

2. Launch an RT-KVM instance:

   # openstack server create  --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt

   Copy to Clipboard Toggle word wrap

3. To verify that the instance uses the assigned emulator threads, run the following command:

   # virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

   Copy to Clipboard Toggle word wrap

1. Add the NeutronPhysicalDevMappings parameter in the parameter_defaults section to link between the logical network name and the physical interface.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2

   Copy to Clipboard Toggle word wrap

2. Add the new property, trusted, to the SR-IOV parameters.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2
     NovaPCIPassthrough:
       - devname: "p5p2"
         physical_network: "sriov2"
         trusted: "true"

   Copy to Clipboard Toggle word wrap
   Note

   You must include double quotation marks around the value "true".

   Important

   Complete the following step in trusted environments, as it allows trusted port binding by non-administrative accounts.

3. Modify permissions to allow users to create and update port bindings.

   parameter_defaults:
     NeutronApiPolicies: {
       operator_create_binding_profile: { key: 'create_port:binding:profile', value: 'rule:admin_or_network_owner'},
       operator_get_binding_profile: { key: 'get_port:binding:profile', value: 'rule:admin_or_network_owner'},
       operator_update_binding_profile: { key: 'update_port:binding:profile', value: 'rule:admin_or_network_owner'}
     }

   Copy to Clipboard Toggle word wrap

1. Create a network of type vlan.

   openstack network create trusted_vf_network  --provider-network-type vlan \
    --provider-segment 111 --provider-physical-network sriov2 \
    --external --disable-port-security

   Copy to Clipboard Toggle word wrap

2. Create a subnet.

   openstack subnet create --network trusted_vf_network \
     --ip-version 4 --subnet-range 192.168.111.0/24 --no-dhcp \
    subnet-trusted_vf_network

   Copy to Clipboard Toggle word wrap

3. Create a port. Set the vnic-type option to direct, and the binding-profile option to true.

   openstack port create --network sriov111 \
   --vnic-type direct --binding-profile trusted=true \
   sriov111_port_trusted

   Copy to Clipboard Toggle word wrap

4. Create an instance, and bind it to the previously-created trusted port.

   openstack server create --image rhel --flavor dpdk  --network internal --port trusted_vf_network_port_trusted --config-drive True --wait rhel-dpdk-sriov_trusted

   Copy to Clipboard Toggle word wrap

Verify the trusted VF configuration on the hypervisor

1. On the compute node that you created the instance, run the following command:

1. Verify that the trust status of the VF is trust on. The example output contains details of an environment that contains two ports. Note that vf 6 contains the text trust on.