Chapter 10. Configuring Real-Time Compute

1. Install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud) [stack@undercloud-0 ~]$ sudo yum install libguestfs-tools

   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

2. Extract the images:

   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/overcloud-full.tar
   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent.tar

3. Copy the default image:

   (undercloud) [stack@undercloud-0 ~]$ cp overcloud-full.qcow2 overcloud-realtime-compute.qcow2

4. Register the image and configure the required subscriptions:

   (undercloud) [stack@undercloud-0 ~]$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager register --username=[username] --password=[password]'
   [  0.0] Examining the guest ...
   [ 10.0] Setting a random seed
   [ 10.0] Running: subscription-manager register --username=[username] --password=[password]
   [ 24.0] Finishing off

   Replace the username and password values with your Red Hat customer account details. For general information about building a Real-time overcloud image, see the Modifying the Red Hat Enterprise Linux OpenStack Platform Overcloud Image with virt-customize knowledgebase article.

5. Find the SKU of the Red Hat OpenStack Platform for Real Time subscription. The SKU might be located on a system that is already registered to the Red Hat Subscription Manager with the same account and credentials. For example:

   $ sudo subscription-manager list

6. Attach the Red Hat OpenStack Platform for Real Time subscription to the image:

   (undercloud) [stack@undercloud-0 ~]$  virt-customize -a overcloud-realtime-compute.qcow2 --run-command 'subscription-manager attach --pool [subscription-pool]'

7. Create a script to configure rt on the image:

   (undercloud) [stack@undercloud-0 ~]$ cat rt.sh
     #!/bin/bash
   
     set -eux
   
     subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-openstack-13-rpms --enable=rhel-7-server-nfv-rpms
     yum -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     yum -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
   
     # END OF SCRIPT

8. Run the script to configure the real-time image:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

9. Re-label SELinux:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

10. Extract vmlinuz and initrd:

    (undercloud) [stack@undercloud-0 ~]$ mkdir image
    (undercloud) [stack@undercloud-0 ~]$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud) [stack@undercloud-0 ~]$ guestunmount image

    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

11. Upload the image:

    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2

1. Based on the /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml file, create a compute-real-time.yaml environment file that sets the parameters for the ComputeRealTime role.

   cp /usr/share/openstack-tripleo-heat-templates/environments/compute-real-time-example.yaml /home/stack/templates/compute-real-time.yaml

   The file must include values for the following parameters:

   • IsolCpusList and NovaVcpuPinSet. List of isolated CPU cores and virtual CPU pins to reserve for real-time workloads. This value depends on the CPU hardware of your Real-time Compute nodes.
   • KernelArgs. Arguments to pass to the kernel of the Real-time Compute nodes. For example, you can use default_hugepagesz=1G hugepagesz=1G hugepages=<number_of_1G_pages_to_reserve> hugepagesz=2M hugepages=<number_of_2M_pages> to define the memory requirements of guests that have huge pages with multiple sizes. In this example, the default size is 1GB but you can also reserve 2M huge pages.

2. Add the ComputeRealTime role to your roles data file and regenerate the file. For example:

   $ openstack overcloud roles generate -o /home/stack/templates/rt_roles_data.yaml Controller Compute ComputeRealTime

   This command generates a ComputeRealTime role with contents similar to the following example, and also sets the ImageDefault option to overcloud-realtime-compute.

   ###############################################################
   # Role: ComputeRealTime                                                               #
   ###############################################################
   
   - name: ComputeRealTime
     description: |
       Compute role that is optimized for real-time behaviour. When using this role
       it is mandatory that an overcloud-realtime-compute image is available and
       the role specific parameters IsolCpusList and NovaVcpuPinSet are set
       accordingly to the hardware of the real-time compute nodes.
     CountDefault: 1
     networks:
       - InternalApi
       - Tenant
       - Storage
     HostnameFormatDefault: '%stackname%-computerealtime-%index%'
     disable_upgrade_deployment: True
     ImageDefault: overcloud-realtime-compute
     RoleParametersDefault:
       TunedProfileName: "realtime-virtual-host"
       KernelArgs: ""      # these must be set in an environment file or similar
       IsolCpusList: ""    # according to the hardware of real-time nodes
       NovaVcpuPinSet: ""  #
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::SkydiveAgent
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent
       - OS::TripleO::Services::Ptp

   For general information about custom roles and about the roles-data.yaml, see the Roles section.

3. Create the compute-realtime flavor to tag nodes that you want to designate for real-time workloads. For example:

   $ source ~/stackrc
   $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-realtime
   $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-realtime" compute-realtime

4. Tag each node that you want to designate for real-time workloads with the compute-realtime profile.

   $ openstack baremetal node set --property capabilities='profile:compute-realtime,boot_option:local' <NODE UUID>

5. Map the ComputeRealTime role to the compute-realtime flavor by creating an environment file with the following content:

   parameter_defaults:
     OvercloudComputeRealTimeFlavor: compute-realtime

6. Run the openstack overcloud deploy command with the -e option and specify all the environment files that you created, as well as the new roles file. For example:

   $ openstack overcloud deploy -r /home/stack/templates/rt~/my_roles_data.yaml  -e /home/stack/templates/compute-real-time.yaml <FLAVOR_ENV_FILE>

   Note

   If you want to run additional real-time instances on the same Compute node, you can change the priority of the instances in the realtime_schedule_priority paremeter in the nova.conf file.

1. Create the compute-real-time.yaml file with the following parameters:

   parameter_defaults:
     ComputeRealTimeParameters:
       IsolCpusList: "1"
       NovaVcpuPinSet: "1"
       KernelArgs: "default_hugepagesz=1G hugepagesz=1G hugepages=16"

2. Create a new rt_roles_data.yaml file with the ComputeRealTime role.

   $ openstack overcloud roles generate -o ~/rt_roles_data.yaml Controller ComputeRealTime

3. Deploy the overcloud, adding both your new real-time roles data file and your real-time environment file to the stack along with your other environment files:

   (undercloud) $ openstack overcloud deploy --templates \
     -r /home/stack/rt_roles_data.yaml
     -e [your environment files]
     -e /home/stack/templates/compute-real-time.yaml

   This command deploys one Controller node and one Real-time Compute node.

4. Log into the Real-time Compute node and check the following parameters. Replace <...> with the values of the relevant parameters from the compute-real-time.yaml.

   [root@overcloud-computerealtime-0 ~]# uname -a
   Linux overcloud-computerealtime-0 3.10.0-693.11.1.rt56.632.el7.x86_64 #1 SMP PREEMPT RT Wed Dec 13 13:37:53 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
   [root@overcloud-computerealtime-0 ~]# cat /proc/cmdline
   BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.11.1.rt56.632.el7.x86_64 root=UUID=45ae42d0-58e7-44fe-b5b1-993fe97b760f ro console=tty0 crashkernel=auto console=ttyS0,115200 default_hugepagesz=1G hugepagesz=1G hugepages=16
   [root@overcloud-computerealtime-0 ~]# tuned-adm active
   Current active profile: realtime-virtual-host
   [root@overcloud-computerealtime-0 ~]# grep ^isolated_cores /etc/tuned/realtime-virtual-host-variables.conf
   isolated_cores=<IsolCpusList>
   [root@overcloud-computerealtime-0 ~]# cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/module/kvm/parameters/lapic_timer_advance_ns
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
   X (X != 0)
   [root@overcloud-computerealtime-0 ~]# grep ^vcpu_pin_set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf
   vcpu_pin_set=<NovaVcpuPinSet>

Launching a real-time instance

1. Make sure that the compute-realtime flavor exists on the overcloud, as described in the Deploying the Real-time Compute Role section.

2. Launch the real-time instance.

   # openstack server create  --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt

3. If you have administrator access to the Compute host, you can optionally verify that the instance uses the assigned emulator threads.

   # virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

1. Set the emulator_thread_policy parameter to isolate. For example:

1. Configure your flavor to use a dedicated CPU policy. To do so, set the hw:cpu_policy parameter to dedicated on the flavor. For example: