Chapter 16. Provisioning real-time and low latency workloads

Procedure

1. Create a Pod CR for the low latency workload and apply it in the cluster, for example:

   Example Pod spec configured to use real-time processing

   apiVersion: v1
   kind: Pod
   metadata:
     name: dynamic-low-latency-pod
     annotations:
       cpu-quota.crio.io: "disable" 1
       cpu-load-balancing.crio.io: "disable" 2
       irq-load-balancing.crio.io: "disable" 3
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: dynamic-low-latency-pod
       image: "registry.redhat.io/openshift4/cnf-tests-rhel8:v4.16"
       command: ["sleep", "10h"]
       resources:
         requests:
           cpu: 2
           memory: "200M"
         limits:
           cpu: 2
           memory: "200M"
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: "" 4
     runtimeClassName: performance-dynamic-low-latency-profile 5
   # ...

   1

   Disables the CPU completely fair scheduler (CFS) quota at the pod run time.

   2

   Disables CPU load balancing.

   3

   Opts the pod out of interrupt handling on the node.

   4

   The nodeSelector label must match the label that you specify in the Node CR.

   5

   runtimeClassName must match the name of the performance profile configured in the cluster.

2. Enter the pod runtimeClassName in the form performance-<profile_name>, where <profile_name> is the name from the PerformanceProfile YAML. In the previous example, the name is performance-dynamic-low-latency-profile.

3. Ensure the pod is running correctly. Status should be running, and the correct cnf-worker node should be set:

   $ oc get pod -o wide

   Expected output

   NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE
   dynamic-low-latency-pod  1/1     Running   0          5h33m   10.131.0.10  cnf-worker.example.com

4. Get the CPUs that the pod configured for IRQ dynamic load balancing runs on:

   $ oc exec -it dynamic-low-latency-pod -- /bin/bash -c "grep Cpus_allowed_list /proc/self/status | awk '{print $2}'"

   Expected output

   Cpus_allowed_list:  2-3

1. Log in to the node to verify the configuration.

   $ oc debug node/<node-name>

2. Verify that you can use the node file system:

   sh-4.4# chroot /host

   Expected output

   sh-4.4#

3. Ensure the default system CPU affinity mask does not include the dynamic-low-latency-pod CPUs, for example, CPUs 2 and 3.

   sh-4.4# cat /proc/irq/default_smp_affinity

   Example output

   33

4. Ensure the system IRQs are not configured to run on the dynamic-low-latency-pod CPUs:

   sh-4.4# find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); file=$(echo $i); echo $file: $mask' _ {} \;

   Example output

   /proc/irq/0/smp_affinity_list: 0-5
   /proc/irq/1/smp_affinity_list: 5
   /proc/irq/2/smp_affinity_list: 0-5
   /proc/irq/3/smp_affinity_list: 0-5
   /proc/irq/4/smp_affinity_list: 0
   /proc/irq/5/smp_affinity_list: 0-5
   /proc/irq/6/smp_affinity_list: 0-5
   /proc/irq/7/smp_affinity_list: 0-5
   /proc/irq/8/smp_affinity_list: 4
   /proc/irq/9/smp_affinity_list: 4
   /proc/irq/10/smp_affinity_list: 0-5
   /proc/irq/11/smp_affinity_list: 0
   /proc/irq/12/smp_affinity_list: 1
   /proc/irq/13/smp_affinity_list: 0-5
   /proc/irq/14/smp_affinity_list: 1
   /proc/irq/15/smp_affinity_list: 0
   /proc/irq/24/smp_affinity_list: 1
   /proc/irq/25/smp_affinity_list: 1
   /proc/irq/26/smp_affinity_list: 1
   /proc/irq/27/smp_affinity_list: 5
   /proc/irq/28/smp_affinity_list: 1
   /proc/irq/29/smp_affinity_list: 0
   /proc/irq/30/smp_affinity_list: 0-5

1. Create the pod:

   $ oc  apply -f qos-pod.yaml --namespace=qos-example

2. View detailed information about the pod:

   $ oc get pod qos-demo --namespace=qos-example --output=yaml

   Example output

   spec:
     containers:
       ...
   status:
     qosClass: Guaranteed

   Note

   If you specify a memory limit for a container, but do not specify a memory request, OpenShift Container Platform automatically assigns a memory request that matches the limit. Similarly, if you specify a CPU limit for a container, but do not specify a CPU request, OpenShift Container Platform automatically assigns a CPU request that matches the limit.

Procedure

1. Add the required annotations to your high priority workload pods. The annotations override the default settings.

   Example high priority workload annotation

   apiVersion: v1
   kind: Pod
   metadata:
     #...
     annotations:
       #...
       cpu-c-states.crio.io: "disable"
       cpu-freq-governor.crio.io: "performance"
       #...
     #...
   spec:
     #...
     runtimeClassName: performance-<profile_name>
     #...

2. Restart the pods to apply the annotation.