Chapter 13. Low latency tuning

1. Optional: Add a node to the OpenShift Container Platform cluster. See Setting BIOS parameters for system tuning.
2. Add the label worker-rt to the worker nodes that require the real-time capability by using the oc command.

3. Create a new machine config pool for real-time nodes:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-rt
     labels:
       machineconfiguration.openshift.io/role: worker-rt
   spec:
     machineConfigSelector:
       matchExpressions:
         - {
              key: machineconfiguration.openshift.io/role,
              operator: In,
              values: [worker, worker-rt],
           }
     paused: false
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-rt: ""

   Note that a machine config pool worker-rt is created for group of nodes that have the label worker-rt.

4. Add the node to the proper machine config pool by using node role labels.

   Note

   You must decide which nodes are configured with real-time workloads. You could configure all of the nodes in the cluster, or a subset of the nodes. The Node Tuning Operator that expects all of the nodes are part of a dedicated machine config pool. If you use all of the nodes, you must point the Node Tuning Operator to the worker node role label. If you use a subset, you must group the nodes into a new machine config pool.

5. Create the PerformanceProfile with the proper set of housekeeping cores and realTimeKernel: enabled: true.

6. You must set machineConfigPoolSelector in PerformanceProfile:

     apiVersion: performance.openshift.io/v2
     kind: PerformanceProfile
     metadata:
      name: example-performanceprofile
     spec:
     ...
       realTimeKernel:
         enabled: true
       nodeSelector:
          node-role.kubernetes.io/worker-rt: ""
       machineConfigPoolSelector:
          machineconfiguration.openshift.io/role: worker-rt

7. Verify that a matching machine config pool exists with a label:

   $ oc describe mcp/worker-rt

   Example output

   Name:         worker-rt
   Namespace:
   Labels:       machineconfiguration.openshift.io/role=worker-rt

8. OpenShift Container Platform will start configuring the nodes, which might involve multiple reboots. Wait for the nodes to settle. This can take a long time depending on the specific hardware you use, but 20 minutes per node is expected.
9. Verify everything is working as expected.

Procedure

1. Create a pod with a QoS class of Guaranteed.
2. Optional: Disable CPU load balancing for DPDK.
3. Assign a proper node selector.

1. Create the pod:

   $ oc  apply -f qos-pod.yaml --namespace=qos-example

2. View detailed information about the pod:

   $ oc get pod qos-demo --namespace=qos-example --output=yaml

   Example output

   spec:
     containers:
       ...
   status:
     qosClass: Guaranteed

   Note

   If a container specifies its own memory limit, but does not specify a memory request, OpenShift Container Platform automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own CPU limit, but does not specify a CPU request, OpenShift Container Platform automatically assigns a CPU request that matches the limit.

1. Set the offline CPUs in the performance profile and save the contents of the YAML file:

   Example performance profile with offlined CPUs

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     additionalKernelArgs:
     - nmi_watchdog=0
     - audit=0
     - mce=off
     - processor.max_cstate=1
     - intel_idle.max_cstate=0
     - idle=poll
     cpu:
       isolated: "2-23,26-47"
       reserved: "0,1,24,25"
       offlined: "48-59" 1
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numa:
       topologyPolicy: single-numa-node
     realTimeKernel:
       enabled: true

   1

   Optional. You can list CPUs in the offlined field to take the specified CPUs offline.

2. Apply the updated profile by running the following command:

   $ oc apply -f my-performance-profile.yaml

Procedure

1. Generate a PerformanceProfile with per-pod-power-management set to true:

   $ podman run --entrypoint performance-profile-creator -v \
   /must-gather:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-operator:v4.12 \
   --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true \
   --split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node \
   --must-gather-dir-path /must-gather -power-consumption-mode=low-latency \ 1
   --per-pod-power-management=true > my-performance-profile.yaml

   1

   The power-consumption-mode must be default or low-latency when the per-pod-power-management is set to true.

   Example PerformanceProfile with perPodPowerManagement

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       [.....]
       workloadHints:
           realTime: true
           highPowerConsumption: false
           perPodPowerManagement: true

2. Set the default cpufreq governor as an additional kernel argument in the PerformanceProfile custom resource (CR):

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       ...
       additionalKernelArgs:
       - cpufreq.default_governor=schedutil 1

   1

   Using the schedutil governor is recommended, however, you can use other governors such as the ondemand or powersave governors.

3. Set the maximum CPU frequency in the TunedPerformancePatch CR:

   spec:
     profile:
     - data: |
         [sysfs]
         /sys/devices/system/cpu/intel_pstate/max_perf_pct = <x> 1

   1

   The max_perf_pct controls the maximum frequency the cpufreq driver is allowed to set as a percentage of the maximum supported cpu frequency. This value applies to all CPUs. You can check the maximum supported frequency in /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq. As a starting point, you can use a percentage that caps all CPUs at the All Cores Turbo frequency. The All Cores Turbo frequency is the frequency that all cores will run at when the cores are all fully occupied.

4. Add the desired annotations to your high priority workload pods. The annotations override the default settings.

   Example high priority workload annotation

   apiVersion: v1
   kind: Pod
   metadata:
     ...
     annotations:
       ...
       cpu-c-states.crio.io: "disable"
       cpu-freq-governor.crio.io: "<governor>"
       ...
     ...
   spec:
     ...
     runtimeClassName: performance-<profile_name>
     ...

5. Restart the pods.

Procedure

1. Log in to the OpenShift Container Platform cluster as a user with cluster-admin privileges.
2. Set the performance profile apiVersion to use performance.openshift.io/v2.
3. Remove the globallyDisableIrqLoadBalancing field or set it to false.

4. Set the appropriate isolated and reserved CPUs. The following snippet illustrates a profile that reserves 2 CPUs. IRQ load-balancing is enabled for pods running on the isolated CPU set:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: dynamic-irq-profile
   spec:
     cpu:
       isolated: 2-5
       reserved: 0-1
   ...

   Note

   When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.

5. Create the pod that uses exclusive CPUs, and set irq-load-balancing.crio.io and cpu-quota.crio.io annotations to disable. For example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dynamic-irq-pod
     annotations:
        irq-load-balancing.crio.io: "disable"
        cpu-quota.crio.io: "disable"
   spec:
     containers:
     - name: dynamic-irq-pod
       image: "registry.redhat.io/openshift4/cnf-tests-rhel8:v4.12"
       command: ["sleep", "10h"]
       resources:
         requests:
           cpu: 2
           memory: "200M"
         limits:
           cpu: 2
           memory: "200M"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     runtimeClassName: performance-dynamic-irq-profile
   ...

6. Enter the pod runtimeClassName in the form performance-<profile_name>, where <profile_name> is the name from the PerformanceProfile YAML, in this example, performance-dynamic-irq-profile.
7. Set the node selector to target a cnf-worker.

8. Ensure the pod is running correctly. Status should be running, and the correct cnf-worker node should be set:

   $ oc get pod -o wide

   Expected output

   NAME              READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
   dynamic-irq-pod   1/1     Running   0          5h33m   <ip-address>   <node-name>   <none>           <none>

9. Get the CPUs that the pod configured for IRQ dynamic load balancing runs on:

   $ oc exec -it dynamic-irq-pod -- /bin/bash -c "grep Cpus_allowed_list /proc/self/status | awk '{print $2}'"

   Expected output

   Cpus_allowed_list:  2-3

10. Ensure the node configuration is applied correctly. Log in to the node to verify the configuration.

    $ oc debug node/<node-name>

    Expected output

    Starting pod/<node-name>-debug ...
    To use host binaries, run `chroot /host`
    
    Pod IP: <ip-address>
    If you don't see a command prompt, try pressing enter.
    
    sh-4.4#

11. Verify that you can use the node file system:

    sh-4.4# chroot /host

    Expected output

    sh-4.4#

12. Ensure the default system CPU affinity mask does not include the dynamic-irq-pod CPUs, for example, CPUs 2 and 3.

    $ cat /proc/irq/default_smp_affinity

    Example output

    33

13. Ensure the system IRQs are not configured to run on the dynamic-irq-pod CPUs:

    find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); file=$(echo $i); echo $file: $mask' _ {} \;

    Example output

    /proc/irq/0/smp_affinity_list: 0-5
    /proc/irq/1/smp_affinity_list: 5
    /proc/irq/2/smp_affinity_list: 0-5
    /proc/irq/3/smp_affinity_list: 0-5
    /proc/irq/4/smp_affinity_list: 0
    /proc/irq/5/smp_affinity_list: 0-5
    /proc/irq/6/smp_affinity_list: 0-5
    /proc/irq/7/smp_affinity_list: 0-5
    /proc/irq/8/smp_affinity_list: 4
    /proc/irq/9/smp_affinity_list: 4
    /proc/irq/10/smp_affinity_list: 0-5
    /proc/irq/11/smp_affinity_list: 0
    /proc/irq/12/smp_affinity_list: 1
    /proc/irq/13/smp_affinity_list: 0-5
    /proc/irq/14/smp_affinity_list: 1
    /proc/irq/15/smp_affinity_list: 0
    /proc/irq/24/smp_affinity_list: 1
    /proc/irq/25/smp_affinity_list: 1
    /proc/irq/26/smp_affinity_list: 1
    /proc/irq/27/smp_affinity_list: 5
    /proc/irq/28/smp_affinity_list: 1
    /proc/irq/29/smp_affinity_list: 0
    /proc/irq/30/smp_affinity_list: 0-5

Procedure

1. Ascertain which threads are running on what CPUs for the host you want to configure.

   You can view which threads are running on the host CPUs by logging in to the cluster and running the following command:

   $ lscpu --all --extended

   Example output

   CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
   0   0    0      0    0:0:0:0       yes    4800.0000 400.0000
   1   0    0      1    1:1:1:0       yes    4800.0000 400.0000
   2   0    0      2    2:2:2:0       yes    4800.0000 400.0000
   3   0    0      3    3:3:3:0       yes    4800.0000 400.0000
   4   0    0      0    0:0:0:0       yes    4800.0000 400.0000
   5   0    0      1    1:1:1:0       yes    4800.0000 400.0000
   6   0    0      2    2:2:2:0       yes    4800.0000 400.0000
   7   0    0      3    3:3:3:0       yes    4800.0000 400.0000

   In this example, there are eight logical CPU cores running on four physical CPU cores. CPU0 and CPU4 are running on physical Core0, CPU1 and CPU5 are running on physical Core 1, and so on.

   Alternatively, to view the threads that are set for a particular physical CPU core (cpu0 in the example below), open a command prompt and run the following:

   $ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list

   Example output

   0-4

2. Apply the isolated and reserved CPUs in the PerformanceProfile YAML. For example, you can set logical cores CPU0 and CPU4 as isolated, and logical cores CPU1 to CPU3 and CPU5 to CPU7 as reserved. When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.

   ...
     cpu:
       isolated: 0,4
       reserved: 1-3,5-7
   ...

   Note

   The reserved and isolated CPU pools must not overlap and together must span all available cores in the worker node.

Procedure

1. Create a PerformanceProfile appropriate for the environment’s hardware and topology as described in the table in "Understanding workload hints". Adjust the profile to match the expected workload. In this example, we tune for the lowest possible latency.

2. Add the highPowerConsumption and realTime workload hints. Both are set to true here.

       apiVersion: performance.openshift.io/v2
       kind: PerformanceProfile
       metadata:
         name: workload-hints
       spec:
         ...
         workloadHints:
           highPowerConsumption: true 1
           realTime: true 2

   1

   If highPowerConsumption is true, the node is tuned for very low latency at the cost of increased power consumption.

   2

   Disables some debugging and monitoring features that can affect system latency.

Procedure

1. Create a performance profile appropriate for the environment’s hardware and topology.

2. Add the reserved and isolated parameters with the CPUs you want reserved and isolated for the infra and application containers:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: infra-cpus
   spec:
     cpu:
       reserved: "0-4,9" 1
       isolated: "5-8" 2
     nodeSelector: 3
       node-role.kubernetes.io/worker: ""

   1

   Specify which CPUs are for infra containers to perform cluster and operating system housekeeping duties.

   2

   Specify which CPUs are for application containers to run workloads.

   3

   Optional: Specify a node selector to apply the performance profile to specific nodes.

Procedure

1. Log in to the OpenShift Container Platform cluster running the Node Tuning Operator as a user with cluster-admin privileges.
2. Create and apply a performance profile appropriate for your hardware and topology. For guidance on creating a profile, see the "Creating a performance profile" section.

3. Edit this created performance profile:

   $ oc edit -f <your_profile_name>.yaml

4. Populate the spec field with the net object. The object list can contain two fields:

   • userLevelNetworking is a required field specified as a boolean flag. If userLevelNetworking is true, the queue count is set to the reserved CPU count for all supported devices. The default is false.

   • devices is an optional field specifying a list of devices that will have the queues set to the reserved CPU count. If the device list is empty, the configuration applies to all network devices. The configuration is as follows:

     • interfaceName: This field specifies the interface name, and it supports shell-style wildcards, which can be positive or negative.

       • Example wildcard syntax is as follows: <string> .*
       • Negative rules are prefixed with an exclamation mark. To apply the net queue changes to all devices other than the excluded list, use !<device>, for example, !eno1.
     • vendorID: The network device vendor ID represented as a 16-bit hexadecimal number with a 0x prefix.

     • deviceID: The network device ID (model) represented as a 16-bit hexadecimal number with a 0x prefix.

       Note

       When a deviceID is specified, the vendorID must also be defined. A device that matches all of the device identifiers specified in a device entry interfaceName, vendorID, or a pair of vendorID plus deviceID qualifies as a network device. This network device then has its net queues count set to the reserved CPU count.

       When two or more devices are specified, the net queues count is set to any net device that matches one of them.

5. Set the queue count to the reserved CPU count for all devices by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

6. Set the queue count to the reserved CPU count for all devices matching any of the defined device identifiers by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - interfaceName: "eth1"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

7. Set the queue count to the reserved CPU count for all devices starting with the interface name eth by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth*"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

8. Set the queue count to the reserved CPU count for all devices with an interface named anything other than eno1 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "!eno1"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

9. Set the queue count to the reserved CPU count for all devices that have an interface name eth0, vendorID of 0x1af4, and deviceID of 0x1000 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

10. Apply the updated performance profile:

    $ oc apply -f <your_profile_name>.yaml

1. The associated machine config pool is in a degraded state:

   # oc get mcp

   Example output

   NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master       rendered-master-2ee57a93fa6c9181b546ca46e1571d2d       True      False      False      3              3                   3                     0                      2d21h
   worker       rendered-worker-d6b2bdc07d9f5a59a6b68950acf25e5f       True      False      False      2              2                   2                     0                      2d21h
   worker-cnf   rendered-worker-cnf-6c838641b8a08fff08dbd8b02fb63f7c   False     True       True       2              1                   1                     1                      2d20h

2. The describe section of the MCP shows the reason:

   # oc describe mcp worker-cnf

   Example output

     Message:               Node node-worker-cnf is reporting: "prepping update:
     machineconfig.machineconfiguration.openshift.io \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not
     found"
       Reason:                1 nodes are reporting degraded status on sync

3. The degraded state should also appear under the performance profile status field marked as degraded = true:

   # oc describe performanceprofiles performance

   Example output

   Message: Machine config pool worker-cnf Degraded Reason: 1 nodes are reporting degraded status on sync.
   Machine config pool worker-cnf Degraded Message: Node yquinn-q8s5v-w-b-z5lqn.c.openshift-gce-devel.internal is
   reporting: "prepping update: machineconfig.machineconfiguration.openshift.io
   \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not found".    Reason:  MCPDegraded
      Status:  True
      Type:    Degraded

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to the Node Tuning Operator:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 1
   
    --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.12 2

   1

   The default OpenShift Container Platform must-gather image.

   2

   The must-gather image for low latency tuning diagnostics.

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

    $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.