Chapter 6. Scheduling NUMA-aware workloads

Procedure

1. Create the MachineConfigPool custom resource that enables custom kubelet configurations for worker nodes:

   1. Save the following YAML in the nro-machineconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        labels:
          cnf-worker-tuning: enabled
          machineconfiguration.openshift.io/mco-built-in: ""
          pools.operator.machineconfiguration.openshift.io/worker: ""
        name: worker
      spec:
        machineConfigSelector:
          matchLabels:
            machineconfiguration.openshift.io/role: worker
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""

   2. Create the MachineConfigPool CR by running the following command:

      $ oc create -f nro-machineconfig.yaml

2. Create the NUMAResourcesOperator custom resource:

   1. Save the following YAML in the nrop.yaml file:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 1

      1

      Should match the label applied to worker nodes in the related MachineConfigPool CR.

   2. Create the NUMAResourcesOperator CR by running the following command:

      $ oc create -f nrop.yaml

Procedure

1. Create the KubeletConfig custom resource that configures the pod admittance policy for the machine profile:

   1. Save the following YAML in the nro-kubeletconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: cnf-worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            cnf-worker-tuning: enabled
        kubeletConfig:
          cpuManagerPolicy: "static" 1
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1"
          memoryManagerPolicy: "Static" 2
          evictionHard:
            memory.available: "100Mi"
          kubeReserved:
            memory: "512Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 3
          topologyManagerScope: "pod"

      1

      For cpuManagerPolicy, static must use a lowercase s.

      2

      For memoryManagerPolicy, Static must use an uppercase S.

      3

      topologyManagerPolicy must be set to single-numa-node.

   2. Create the KubeletConfig custom resource (CR) by running the following command:

      $ oc create -f nro-kubeletconfig.yaml

2. Create the NUMAResourcesScheduler custom resource that deploys the NUMA-aware custom pod scheduler:

   1. Save the following YAML in the nro-scheduler.yaml file:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.10"

   2. Create the NUMAResourcesScheduler CR by running the following command:

      $ oc create -f nro-scheduler.yaml

Procedure

1. Get the name of the NUMA-aware scheduler that is deployed in the cluster by running the following command:

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Example output

   topo-aware-scheduler

2. Create a Deployment CR that uses scheduler named topo-aware-scheduler, for example:

   1. Save the following YAML in the nro-deployment.yaml file:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 1
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"

      1

      schedulerName must match the name of the NUMA-aware scheduler that is deployed in your cluster, for example topo-aware-scheduler.

   2. Create the Deployment CR by running the following command:

      $ oc create -f nro-deployment.yaml

Verification

1. Verify that the deployment was successful:

   $ oc get pods -n openshift-numaresources

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   numa-deployment-1-56954b7b46-pfgw8                  2/2     Running   0          129m
   numaresources-controller-manager-7575848485-bns4s   1/1     Running   0          15h
   numaresourcesoperator-worker-dvj4n                  2/2     Running   0          18h
   numaresourcesoperator-worker-lcg4t                  2/2     Running   0          16h
   secondary-scheduler-56994cf6cf-7qf4q                1/1     Running   0          18h

2. Verify that the topo-aware-scheduler is scheduling the deployed pod by running the following command:

   $ oc describe pod numa-deployment-1-56954b7b46-pfgw8 -n openshift-numaresources

   Example output

   Events:
     Type    Reason          Age   From                  Message
     ----    ------          ----  ----                  -------
     Normal  Scheduled       130m  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-56954b7b46-pfgw8 to compute-0.example.com

   Note

   Deployments that request more resources than is available for scheduling will fail with a MinimumReplicasUnavailable error. The deployment succeeds when the required resources become available. Pods remain in the Pending state until the required resources are available.

3. Verify that the expected allocated resources are listed for the node. Run the following command:

   $ oc describe noderesourcetopologies.topology.node.k8s.io

   Example output

   ...
   
   Zones:
     Costs:
       Name:   node-0
       Value:  10
       Name:   node-1
       Value:  21
     Name:     node-0
     Resources:
       Allocatable:  39
       Available:    21 1
       Capacity:     40
       Name:         cpu
       Allocatable:  6442450944
       Available:    6442450944
       Capacity:     6442450944
       Name:         hugepages-1Gi
       Allocatable:  134217728
       Available:    134217728
       Capacity:     134217728
       Name:         hugepages-2Mi
       Allocatable:  262415904768
       Available:    262206189568
       Capacity:     270146007040
       Name:         memory
     Type:           Node

   1

   The Available capacity is reduced because of the resources that have been allocated to the guaranteed pod.

   Resources consumed by guaranteed pods are subtracted from the available node resources listed under noderesourcetopologies.topology.node.k8s.io.

4. Resource allocations for pods with a Best-effort or Burstable quality of service (qosClass) are not reflected in the NUMA node resources under noderesourcetopologies.topology.node.k8s.io. If a pod’s consumed resources are not reflected in the node resource calculation, verify that the pod has qosClass of Guaranteed by running the following command:

   $ oc get pod <pod_name> -n <pod_namespace> -o jsonpath="{ .status.qosClass }"

   Example output

   Guaranteed

Procedure

1. Verify that the noderesourcetopologies CRD is deployed in the cluster by running the following command:

   $ oc get crd | grep noderesourcetopologies

   Example output

   NAME                                                              CREATED AT
   noderesourcetopologies.topology.node.k8s.io                       2022-01-18T08:28:06Z

2. Check that the NUMA-aware scheduler name matches the name specified in your NUMA-aware workloads by running the following command:

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Example output

   topo-aware-scheduler

3. Verify that NUMA-aware scheduable nodes have the noderesourcetopologies CR applied to them. Run the following command:

   $ oc get noderesourcetopologies.topology.node.k8s.io

   Example output

   NAME                    AGE
   compute-0.example.com   17h
   compute-1.example.com   17h

   Note

   The number of nodes should equal the number of worker nodes that are configured by the machine config pool (mcp) worker definition.

4. Verify the NUMA zone granularity for all scheduable nodes by running the following command:

   $ oc get noderesourcetopologies.topology.node.k8s.io -o yaml

   Example output

   apiVersion: v1
   items:
   - apiVersion: topology.node.k8s.io/v1alpha1
     kind: NodeResourceTopology
     metadata:
       annotations:
         k8stopoawareschedwg/rte-update: periodic
       creationTimestamp: "2022-06-16T08:55:38Z"
       generation: 63760
       name: worker-0
       resourceVersion: "8450223"
       uid: 8b77be46-08c0-4074-927b-d49361471590
     topologyPolicies:
     - SingleNUMANodeContainerLevel
     zones:
     - costs:
       - name: node-0
         value: 10
       - name: node-1
         value: 21
       name: node-0
       resources:
       - allocatable: "38"
         available: "38"
         capacity: "40"
         name: cpu
       - allocatable: "134217728"
         available: "134217728"
         capacity: "134217728"
         name: hugepages-2Mi
       - allocatable: "262352048128"
         available: "262352048128"
         capacity: "270107316224"
         name: memory
       - allocatable: "6442450944"
         available: "6442450944"
         capacity: "6442450944"
         name: hugepages-1Gi
       type: Node
     - costs:
       - name: node-0
         value: 21
       - name: node-1
         value: 10
       name: node-1
       resources:
       - allocatable: "268435456"
         available: "268435456"
         capacity: "268435456"
         name: hugepages-2Mi
       - allocatable: "269231067136"
         available: "269231067136"
         capacity: "270573244416"
         name: memory
       - allocatable: "40"
         available: "40"
         capacity: "40"
         name: cpu
       - allocatable: "1073741824"
         available: "1073741824"
         capacity: "1073741824"
         name: hugepages-1Gi
       type: Node
   - apiVersion: topology.node.k8s.io/v1alpha1
     kind: NodeResourceTopology
     metadata:
       annotations:
         k8stopoawareschedwg/rte-update: periodic
       creationTimestamp: "2022-06-16T08:55:37Z"
       generation: 62061
       name: worker-1
       resourceVersion: "8450129"
       uid: e8659390-6f8d-4e67-9a51-1ea34bba1cc3
     topologyPolicies:
     - SingleNUMANodeContainerLevel
     zones: 1
     - costs:
       - name: node-0
         value: 10
       - name: node-1
         value: 21
       name: node-0
       resources: 2
       - allocatable: "38"
         available: "38"
         capacity: "40"
         name: cpu
       - allocatable: "6442450944"
         available: "6442450944"
         capacity: "6442450944"
         name: hugepages-1Gi
       - allocatable: "134217728"
         available: "134217728"
         capacity: "134217728"
         name: hugepages-2Mi
       - allocatable: "262391033856"
         available: "262391033856"
         capacity: "270146301952"
         name: memory
       type: Node
     - costs:
       - name: node-0
         value: 21
       - name: node-1
         value: 10
       name: node-1
       resources:
       - allocatable: "40"
         available: "40"
         capacity: "40"
         name: cpu
       - allocatable: "1073741824"
         available: "1073741824"
         capacity: "1073741824"
         name: hugepages-1Gi
       - allocatable: "268435456"
         available: "268435456"
         capacity: "268435456"
         name: hugepages-2Mi
       - allocatable: "269192085504"
         available: "269192085504"
         capacity: "270534262784"
         name: memory
       type: Node
   kind: List
   metadata:
     resourceVersion: ""
     selfLink: ""

   1

   Each stanza under zones describes the resources for a single NUMA zone.

   2

   resources describes the current state of the NUMA zone resources. Check that resources listed under items.zones.resources.available correspond to the exclusive NUMA zone resources allocated to each guaranteed pod.