Chapter 8. Configuring low latency

Procedure

• Install the low latency RPM package by running the following command:

  $ sudo dnf install -y microshift-low-latency

  Tip

  Wait to restart the host until after activating your TuneD profile. Restarting the host restarts MicroShift and CRI-O, which applies the low latency manifests and activates the TuneD profile.

Prerequisites

• You installed the OpenShift CLI (oc).
• You have root access to the cluster.
• You made a copy of the provided config.yaml.default file in the /etc/microshift/ directory, and renamed it config.yaml.

Procedure

• Add the kubelet configuration to the MicroShift config.yaml file:

  Example passthrough kubelet configuration

  apiServer:
  # ...
  kubelet: 1
    cpuManagerPolicy: static 2
    cpuManagerPolicyOptions:
      full-pcpus-only: "true" 3
    cpuManagerReconcilePeriod: 5s
    memoryManagerPolicy: Static 4
    topologyManagerPolicy: single-numa-node
    reservedSystemCPUs: 0-1 5
    reservedMemory:
    - limits:
        memory: 1100Mi 6
      numaNode: 0
    kubeReserved:
      memory: 500Mi
    systemReserved:
      memory: 500Mi
    evictionHard: 7
      imagefs.available: "15%" 8
      memory.available: "100Mi" 9
      nodefs.available: "10%" 10
      nodefs.inodesFree: "5%" 11
    evictionPressureTransitionPeriod: 0s
  # ...

  1

  If you change the CPU or memory managers in the kubelet configuration, you must remove files that cache the previous configuration. Restart the host to remove them automatically, or manually remove the /var/lib/kubelet/cpu_manager_state and /var/lib/kubelet/memory_manager_state files.

  2

  The name of the policy to use. Valid values are none and static. Requires the CPUManager feature gate to be enabled. Default value is none.

  3

  A set of key=value pairs for setting extra options that fine tune the behavior of the CPUManager policies. The default value is null. Requires both the CPUManager and CPUManagerPolicyOptions feature gates to be enabled.

  4

  The name of the policy used by Memory Manager. Case-sensitive. The default value is none. Requires the MemoryManager feature gate to be enabled.

  5

  Required. The reservedSystemCPUs value must be the inverse of the offlined CPUs because both values combined must account for all of the CPUs on the system. This parameter is essential to dividing the management and application workloads. Use this parameter to define a static CPU set for the host-level system and Kubernetes daemons, plus interrupts and timers. Then the rest of the CPUs on the system can be used exclusively for workloads.

  6

  The value in reservedMemory[0].limits.memory, 1100 Mi in this example, is equal to kubeReserved.memory + systemReserved.memory + evictionHard.memory.available.

  7

  The evictionHard parameters define under which conditions the kubelet evicts pods. When you change the default value of only one parameter for the evictionHard stanza, the default values of other parameters are not inherited and are set to zero. Provide all the threshold values even when you want to change just one.

  8

  The imagefs is a filesystem that container runtimes use to store container images and container writable layers. In this example, the evictionHard.imagefs.available parameter means that the pod is evicted when the available space of the image filesystem is less than 15%.

  9

  In this example, the evictionHard.memory.available parameter means that the pods are evicted when the available memory of the node drops below 100MiB.

  10

  In this example, the evictionHard.nodefs.available parameter means that the pods are evicted when the main filesystem of the node has less than 10% available space.

  11

  In this example, the evictionHard.nodefs.inodesFree parameter means that the pods are evicted when more than 15% of the node’s main filesystem’s inodes are in use.

Verification

• After you complete the next steps and restart the host, you can use a root-access account to check that your settings are in the config.yaml file in the /var/lib/microshift/resources/kubelet/config/ directory.

• Use TuneD profiles to tune your system for different use cases, such as deploying a low-latency MicroShift cluster.
• You can modify the rules defined for each profile and customize tuning for a specific device.
• When you switch to another profile or deactivate TuneD, all changes made to the system settings by the previous profile revert back to their original state.
• You can also configure TuneD to react to changes in device usage and adjusts settings to improve performance of active devices and reduce power consumption of inactive devices.

• System tuning is just as important when using the real-time kernel as it is for the standard kernel.
• Installing the real-time kernel on an untuned system running the standard kernel supplied as part of the RHEL 9 release is not likely to result in any noticeable benefit.
• Tuning the standard kernel yields 90% of possible latency gains.
• The real-time kernel provides the last 10% of latency reduction required by the most demanding workloads.

• Embedding in a RHEL for Edge image

Prerequisites

• You installed the microshift-low-latency RPM package.
• You configured workload partitioning.

Procedure

• Use the following example to set the following annotations in the pod spec:

  cpu-load-balancing.crio.io: "disable"
  irq-load-balancing.crio.io: "disable"
  cpu-quota.crio.io: "disable"
  cpu-load-balancing.crio.io: "disable"
  cpu-freq-governor.crio.io: "<governor>"

  Example pod that runs oslat test:

  apiVersion: v1
  kind: Pod
  metadata:
    name: oslat
    annotations:
      cpu-load-balancing.crio.io: "disable" 1
      irq-load-balancing.crio.io: "disable" 2
      cpu-quota.crio.io: "disable" 3
      cpu-c-states.crio.io: "disable" 4
      cpu-freq-governor.crio.io: "<governor>" 5
  spec:
    runtimeClassName: microshift-low-latency 6
    containers:
    - name: oslat
      image: quay.io/container-perf-tools/oslat
      imagePullPolicy: Always
      resources:
        requests:
          memory: "400Mi"
          cpu: "2"
        limits:
          memory: "400Mi"
          cpu: "2"
      env:
      - name: tool
        value: "oslat"
      - name: manual
        value: "n"
      - name: PRIO
        value: "1"
      - name: delay
        value: "0"
      - name: RUNTIME_SECONDS
        value: "60"
      - name: TRACE_THRESHOLD
        value: ""
      - name: EXTRA_ARGS
        value: ""
      securityContext:
        privileged: true
        capabilities:
          add:
            - SYS_NICE
            - IPC_LOCK

  1

  Disables the CPU load balancing for the pod.

  2

  Opts the pod out of interrupt handling (IRQ).

  3

  Disables the CPU completely fair scheduler (CFS) quota at the pod run time.

  4

  Enables or disables C-states for each CPU. Set the value to disable to provide the best performance for a high-priority pod.

  5

  Sets the cpufreq governor for each CPU. The performance governor is recommended for high-priority workloads.

  6

  The runtimeClassName must match the name of the performance profile configured in the cluster. For example, microshift-low-latency.

  Note

  Disable CPU load balancing only when the CPU manager static policy is enabled and for pods with guaranteed QoS that use whole CPUs. Otherwise, disabling CPU load balancing can affect the performance of other containers in the cluster.

  Important

  For the pod to have the Guaranteed QoS class, it must have the same values of CPU and memory in requests and limits. See Guaranteed (Kubernetes upstream documentation)

• Update the MicroShift config.yaml file to include the kubelet configuration file.
• Create the CRI-O systemd and configuration files.
• Create and update the systemd configuration file for the MicroShift and CRI-O services respectively.