Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 12. Scheduling NUMA-aware workloads

To deploy high performance workloads with optimal efficiency, use NUMA-aware scheduling. This feature aligns pods with the underlying hardware topology in your OpenShift Container Platform cluster, minimizing latency and maximizing resource utilization.

By using the NUMA Resources Operator, you can schedule high-performance workloads in the same NUMA zone. The Operator deploys a node resources exporting agent that reports on available cluster node NUMA resources, and a secondary scheduler that manages the workloads.

12.1. About NUMA
Copy link

To reduce latency in multiprocessor systems, Non-Uniform Memory Access (NUMA) architecture allows CPUs to access local memory faster than remote memory. This design optimizes performance by prioritizing memory resources that are physically closer to the processor.

A CPU with multiple memory controllers can use any available memory across CPU complexes, regardless of where the memory is located. However, this increased flexibility comes at the expense of performance.

NUMA resource topology refers to the physical locations of CPUs, memory, and PCI devices relative to each other in a NUMA zone. In a NUMA architecture, a NUMA zone is a group of CPUs that has its own processors and memory. Colocated resources are said to be in the same NUMA zone, and CPUs in a zone have faster access to the same local memory than CPUs outside of that zone.

A CPU processing a workload using memory that is outside its NUMA zone is slower than a workload processed in a single NUMA zone. For I/O-constrained workloads, the network interface on a distant NUMA zone slows down how quickly information can reach the application.

Applications can achieve better performance by containing data and processing within the same NUMA zone. For high-performance workloads and applications, such as telecommunications workloads, the cluster must process pod workloads in a single NUMA zone so that the workload can operate to specification.

12.2. About NUMA-aware scheduling
Copy link

To process latency-sensitive or high-performance workloads efficiently, use NUMA-aware scheduling. This feature aligns cluster compute resources, such as CPUs, memory, and devices, in the same NUMA zone, optimizing resource efficiency and improving pod density per compute node.

By integrating the performance profile of the Node Tuning Operator with NUMA-aware scheduling, you can further configure CPU affinity to optimize performance for latency-sensitive workloads.

The default OpenShift Container Platform pod scheduler scheduling logic considers the available resources of the entire compute node, not individual NUMA zones. If the most restrictive resource alignment is requested in the kubelet topology manager, error conditions can occur when admitting the pod to a node.

Conversely, if the most restrictive resource alignment is not requested, the pod can be admitted to the node without proper resource alignment, leading to worse or unpredictable performance. For example, runaway pod creation with Topology Affinity Error statuses can occur when the pod scheduler makes suboptimal scheduling decisions for guaranteed pod workloads without knowing if the pod’s requested resources are available. Scheduling mismatch decisions can cause indefinite pod startup delays. Also, depending on the cluster state and resource allocation, poor pod scheduling decisions can cause extra load on the cluster because of failed startup attempts.

The NUMA Resources Operator deploys a custom NUMA resources secondary scheduler and other resources to mitigate against the shortcomings of the default OpenShift Container Platform pod scheduler. The following diagram provides a high-level overview of NUMA-aware pod scheduling.

Figure 12.1. NUMA-aware scheduling overview

Diagram of NUMA-aware scheduling that shows how the various components interact with each other in the cluster
NodeResourceTopology API
The NodeResourceTopology API describes the available NUMA zone resources in each compute node.
NUMA-aware scheduler
The NUMA-aware secondary scheduler receives information about the available NUMA zones from the NodeResourceTopology API and schedules high-performance workloads on a node where it can be optimally processed.
Node topology exporter
The node topology exporter exposes the available NUMA zone resources for each compute node to the NodeResourceTopology API. The node topology exporter daemon tracks the resource allocation from the kubelet by using the PodResources API.
PodResources API
The PodResources API is local to each node and exposes the resource topology and available resources to the kubelet.
Note

The List endpoint of the PodResources API exposes exclusive CPUs allocated to a particular container. The API does not expose CPUs that belong to a shared pool.

The GetAllocatableResources endpoint exposes allocatable resources available on a node.

12.3. NUMA resource scheduling strategies
Copy link

To optimize the placement of high-performance workloads, the secondary scheduler uses NUMA-aware scoring strategies to select the most suitable compute nodes. This process assigns workloads based on resource availability while allowing local node managers to handle final resource pinning.

When scheduling high-performance workloads, the secondary scheduler determines which compute node is best suited for the task based on its internal NUMA resource distribution. While the scheduler uses NUMA-level data to score and select a compute node, the actual resource pinning within that node is managed by the local Topology Manager and CPU Manager.

When a high-performance workload is scheduled in a NUMA-aware cluster, the following steps occur:

  1. Node filtering: The scheduler first filters the entire cluster to find a shortlist of feasible nodes. A node is only kept if the node meets all requirements, such as matching labels, respecting taints and tolerations, and, importantly, having sufficient available resources within its specific NUMA zones. If a node cannot satisfy the NUMA affinity of the workload, the node is filtered out at this stage.
  2. Node selection: When a shortlist of suitable nodes is established, the scheduler evaluates them to find the best fit. The scheduler applies a NUMA-aware scoring strategy to rank these candidates based on their resource distribution. The node with the highest score is then selected for the workload.
  3. Local Allocation: When the pod is assigned to a compute node, the node-level components (CPU, memory, device, and topology managers) perform the authoritative allocation of specific CPUs and memory. The scheduler does not influence this final selection.

The following table summarizes the different OpenShift Container Platform strategies and their outcomes:

Expand
Table 12.1. Scoring strategy summary
StrategyDescriptionOutcome

LeastAllocated

Favors compute nodes that contain NUMA zones with the most available resources.

Distributes workloads across the cluster to nodes with the highest available headroom.

MostAllocated

Favors compute nodes where the requested resources fit into NUMA zones that are already highly utilized.

Consolidates workloads on already utilized nodes, potentially leaving other nodes idle.

BalancedAllocation

Favors compute nodes with the most balanced CPU and memory usage across NUMA zones.

Prevents skewed usage patterns where one resource type, such as CPU, is exhausted while another, such as memory, remains idle.

12.4. Installing the NUMA Resources Operator
Copy link

NUMA Resources Operator deploys resources that allow you to schedule NUMA-aware workloads and deployments. You can install the NUMA Resources Operator using the OpenShift Container Platform CLI or the web console.

To enable NUMA-aware scheduling for high-performance workloads, install the NUMA Resources Operator by using the OpenShift CLI (oc). As a cluster administrator, you can deploy the Operator efficiently without using the web console.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.

Procedure

  1. Create a namespace for the NUMA Resources Operator:

    1. Save the following YAML in the nro-namespace.yaml file: 

      apiVersion: v1
kind: Namespace
metadata:
  name: openshift-numaresources
# ...
      Copy to Clipboard Toggle word wrap

    2. Create the Namespace CR by running the following command: 

      $ oc create -f nro-namespace.yaml
      Copy to Clipboard Toggle word wrap

  2. Create the Operator group for the NUMA Resources Operator:

    1. Save the following YAML in the nro-operatorgroup.yaml file: 

      apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: numaresources-operator
  namespace: openshift-numaresources
spec:
  targetNamespaces:
  - openshift-numaresources
# ...
      Copy to Clipboard Toggle word wrap

    2. Create the OperatorGroup CR by running the following command: 

      $ oc create -f nro-operatorgroup.yaml
      Copy to Clipboard Toggle word wrap

  3. Create the subscription for the NUMA Resources Operator:

    1. Save the following YAML in the nro-sub.yaml file: 

      apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: numaresources-operator
  namespace: openshift-numaresources
spec:
  channel: "4.19"
  name: numaresources-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
# ...
      Copy to Clipboard Toggle word wrap

    2. Create the Subscription CR by running the following command: 

      $ oc create -f nro-sub.yaml
      Copy to Clipboard Toggle word wrap

Verification

  1. Verify that the installation succeeded by inspecting the CSV resource in the openshift-numaresources namespace. Run the following command: 

    $ oc get csv -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                             DISPLAY                  VERSION   REPLACES   PHASE
numaresources-operator.v4.19.2   numaresources-operator   4.19.2               Succeeded
    Copy to Clipboard Toggle word wrap

To enable NUMA-aware scheduling for high-performance workloads, install the NUMA Resources Operator by using the web console. As a cluster administrator, you can deploy the Operator through the graphical interface.

Procedure

  1. Create a namespace for the NUMA Resources Operator:

    1. In the OpenShift Container Platform web console, click Administration Namespaces.
    2. Click Create Namespace, enter openshift-numaresources in the Name field, and then click Create.

  2. Install the NUMA Resources Operator:

    1. In the OpenShift Container Platform web console, click Operators OperatorHub.
    2. Choose numaresources-operator from the list of available Operators, and then click Install.
    3. In the Installed Namespaces field, select the openshift-numaresources namespace, and then click Install.

  3. Optional: Verify that the NUMA Resources Operator installed successfully:

    1. Switch to the Operators Installed Operators page.

    2. Ensure that NUMA Resources Operator is listed in the openshift-numaresources namespace with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the Operators Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads Pods page and check the logs for pods in the default project.

12.5. Configuring a single NUMA node policy
Copy link

To enable the NUMA Resources Operator, configure a single NUMA node policy on your cluster. You can implement this policy by creating a performance profile or by configuring a KubeletConfig custom resource (CR).

Note

The preferred way to configure a single NUMA node policy is to apply a performance profile. You can use the Performance Profile Creator (PPC) tool to create the performance profile. If a performance profile is created on the cluster, the PPC tool automatically creates other tuning components like KubeletConfig and the tuned profile.

For more information about creating a performance profile, see "About the Performance Profile Creator" in the "Additional resources" section.

12.5.1. Sample performance profile
Copy link

Reference an example YAML to understand how to use the performance profile creator (PPC) tool to create a performance profile. 

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: performance
spec:
  cpu:
    isolated: "3"
    reserved: 0-2
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/worker: ""
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  numa:
    topologyPolicy: single-numa-node
  realTimeKernel:
    enabled: true
  workloadHints:
    highPowerConsumption: true
    perPodPowerManagement: false
    realTime: true
Copy to Clipboard Toggle word wrap

where:

spec.pools.operator.machineconfiguration.openshift.io/worker
Specifies the value that must match the MachineConfigPool value that you want to configure the NUMA Resources Operator on. For example, you might create a MachineConfigPool object named worker-cnf that designates a set of nodes that run telecommunications workloads. The value for MachineConfigPool must match the machineConfigPoolSelector value in the NUMAResourcesOperator CR that you configure later in "Creating the NUMAResourcesOperator custom resource".
spec.numa.topologyPolicy

Specifies that the topologyPolicy field is set to single-numa-node by setting the topology-manager-policy argument to single-numa-node when you run the PPC tool.

Note

For hosted control plane clusters, the machineConfigPoolSelector does not have any functional effect. Node association is instead determined by the specified NodePool object.

12.5.2. Creating a KubeletConfig CR
Copy link

To configure a single NUMA node policy, create and apply a KubeletConfig custom resource (CR). While applying a performance profile is recommended, you can use the alternative method to manually manage the configuration on your cluster.

Procedure

  1. Create the KubeletConfig custom resource (CR) that configures the pod admittance policy for the machine profile:

    1. Save the following YAML in the nro-kubeletconfig.yaml file: 

      apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    cpuManagerPolicy: "static"
    cpuManagerReconcilePeriod: "5s"
    reservedSystemCPUs: "0,1"
    memoryManagerPolicy: "Static"
    evictionHard:
      memory.available: "100Mi"
    kubeReserved:
      memory: "512Mi"
    reservedMemory:
      - numaNode: 0
        limits:
          memory: "1124Mi"
    systemReserved:
      memory: "512Mi"
    topologyManagerPolicy: "single-numa-node"
      Copy to Clipboard Toggle word wrap

      where:

      spec.machineConfigPoolSelector.matchLabels.pools.operator.machineconfiguration.openshift.io/worker
      Specifies that this label matches the machineConfigPoolSelector setting in the NUMAResourcesOperator CR that you configure later in "Creating the NUMAResourcesOperator custom resource".
      spec.kubeletConfig.cpuManagerPolicy
      Specifies the static value. You must use a lowercase s.
      spec.kubeletConfig.reservedSystemCPUs
      Adjust the field based on the CPU on your nodes.
      spec.kubeletConfig.memoryManagerPolicy
      Specifies Static. You must use an uppercase S.
      spec.kubeletConfig.topologyManagerPolicy

      Specifies the value as single-numa-node.

      Note

      For hosted control plane clusters, the machineConfigPoolSelector setting does not have any functional effect. Node association is instead determined by the specified NodePool object. To apply a KubeletConfig for hosted control plane clusters, you must create a ConfigMap that contains the configuration, and then reference that ConfigMap within the spec.config field of a NodePool.

    2. Create the KubeletConfig CR by running the following command: 

      $ oc create -f nro-kubeletconfig.yaml
      Copy to Clipboard Toggle word wrap
      Note

      Applying performance profile or KubeletConfig automatically triggers rebooting of the nodes. If no reboot is triggered, you can troubleshoot the issue by looking at the labels in KubeletConfig that address the node group.

12.6. Scheduling NUMA-aware workloads
Copy link

To process latency-sensitive and high-performance workloads efficiently, configure your OpenShift Container Platform cluster for NUMA-aware scheduling. This process aligns pods with specific NUMA zones to minimize network delays and maximize compute resource utilization.

Clusters running latency-sensitive workloads typically feature performance profiles that help to minimize workload latency and optimize performance. The NUMA-aware scheduler deploys workloads based on available node NUMA resources and with respect to any performance profile settings applied to the node. The combination of NUMA-aware deployments, and the performance profile of the workload, ensures that workloads are scheduled in a way that maximizes performance.

For the NUMA Resources Operator to be fully operational, you must deploy the NUMAResourcesOperator custom resource and the NUMA-aware secondary pod scheduler.

After you have installed the NUMA Resources Operator, you can create the NUMAResourcesOperator custom resource (CR). This CR instructs the NUMA Resources Operator to install all the cluster infrastructure that is needed to support the NUMA-aware scheduler, including daemon sets and APIs.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.
  • Installed the NUMA Resources Operator.

Procedure

  1. Create the NUMAResourcesOperator custom resource:

    1. Save the following minimal required YAML file example as nrop.yaml: 

      apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesOperator
metadata:
  name: numaresourcesoperator
spec:
  nodeGroups:
  - machineConfigPoolSelector:
      matchLabels:
        pools.operator.machineconfiguration.openshift.io/worker: ""
# ...
      Copy to Clipboard Toggle word wrap
      • pools.operator.machineconfiguration.openshift.io/worker: Specifies a value that must match the MachineConfigPool resource that you want to configure the NUMA Resources Operator on. For example, you might have created a MachineConfigPool resource named worker-cnf that designates a set of nodes expected to run telecommunications workloads. Each NodeGroup must match exactly one MachineConfigPool. Configurations where NodeGroup matches more than one MachineConfigPool are not supported.

    2. Create the NUMAResourcesOperator CR by running the following command: 

      $ oc create -f nrop.yaml
      Copy to Clipboard Toggle word wrap

  2. Optional: To enable NUMA-aware scheduling for multiple machine config pools (MCPs), define a separate NodeGroup for each pool. For example, define three NodeGroups for worker-cnf, worker-ht, and worker-other, in the NUMAResourcesOperator CR as shown in the following example:

    Example YAML definition for a NUMAResourcesOperator CR with multiple NodeGroups

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesOperator
metadata:
  name: numaresourcesoperator
spec:
  logLevel: Normal
  nodeGroups:
    - machineConfigPoolSelector:
        matchLabels:
          machineconfiguration.openshift.io/role: worker-ht
    - machineConfigPoolSelector:
        matchLabels:
          machineconfiguration.openshift.io/role: worker-cnf
    - machineConfigPoolSelector:
        matchLabels:
          machineconfiguration.openshift.io/role: worker-other
# ...
    Copy to Clipboard Toggle word wrap

Verification

  1. Verify that the NUMA Resources Operator deployed successfully by running the following command: 

    $ oc get numaresourcesoperators.nodetopology.openshift.io
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                    AGE
numaresourcesoperator   27s
    Copy to Clipboard Toggle word wrap

  2. After a few minutes, run the following command to verify that the required resources deployed successfully: 

    $ oc get all -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                    READY   STATUS    RESTARTS   AGE
pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s
    Copy to Clipboard Toggle word wrap

After you install the NUMA Resources Operator, create the NUMAResourcesOperator custom resource (CR). The CR instructs the NUMA Resources Operator to install all the cluster infrastructure that is needed to support the NUMA-aware scheduler on hosted control planes, including daemon sets and APIs.

Important

Creating the NUMAResourcesOperator custom resource for hosted control planes is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.
  • Installed the NUMA Resources Operator.

Procedure

  1. Export the management cluster kubeconfig file by running the following command: 

    $ export KUBECONFIG=<path-to-management-cluster-kubeconfig>
    Copy to Clipboard Toggle word wrap

  2. Find the node-pool-name for your cluster by running the following command: 

    $ oc --kubeconfig="$MGMT_KUBECONFIG" get np -A
    Copy to Clipboard Toggle word wrap

    Example output

    NAMESPACE   NAME                     CLUSTER       DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
clusters    democluster-us-east-1a   democluster   1               1               False         False        4.19.0    False             False
    Copy to Clipboard Toggle word wrap

    The node-pool-name is the NAME field in the output. In this example, the node-pool-name is democluster-us-east-1a.

  3. Create a YAML file named nrop-hcp.yaml with at least the following content: 

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesOperator
metadata:
  name: numaresourcesoperator
spec:
  nodeGroups:
  - poolName: democluster-us-east-1a
# ...
    Copy to Clipboard Toggle word wrap
    • spec.nodeGroups.poolName: Specifies the poolName. The example shows the node-pool-name pool name that was retrieved from a previous step.

  4. On the management cluster, run the following command to list the available secrets: 

    $ oc get secrets -n clusters
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                              TYPE                      DATA   AGE
builder-dockercfg-25qpp           kubernetes.io/dockercfg   1      128m
default-dockercfg-mkvlz           kubernetes.io/dockercfg   1      128m
democluster-admin-kubeconfig      Opaque                    1      127m
democluster-etcd-encryption-key   Opaque                    1      128m
democluster-kubeadmin-password    Opaque                    1      126m
democluster-pull-secret           Opaque                    1      128m
deployer-dockercfg-8lfpd          kubernetes.io/dockercfg   1      128m
    Copy to Clipboard Toggle word wrap

  5. Extract the kubeconfig file for the hosted cluster by running the following command: 

    $ oc get secret <SECRET_NAME> -n clusters -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig
    Copy to Clipboard Toggle word wrap

    Example

    $ oc get secret democluster-admin-kubeconfig -n clusters -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig
    Copy to Clipboard Toggle word wrap

  6. Export the hosted cluster kubeconfig file by running the following command: 

    $ export HC_KUBECONFIG=<path_to_hosted-cluster-kubeconfig>
    Copy to Clipboard Toggle word wrap

  7. Create the NUMAResourcesOperator CR by running the following command on the hosted cluster: 

    $ oc create -f nrop-hcp.yaml
    Copy to Clipboard Toggle word wrap

Verification

  1. Verify that the NUMA Resources Operator deployed successfully by running the following command: 

    $ oc get numaresourcesoperators.nodetopology.openshift.io
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                    AGE
numaresourcesoperator   27s
    Copy to Clipboard Toggle word wrap

  2. After a few minutes, run the following command to verify that the required resources deployed successfully: 

    $ oc get all -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                    READY   STATUS    RESTARTS   AGE
pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
pod/numaresourcesoperator-democluster-7d96r             2/2     Running   0          97s
pod/numaresourcesoperator-democluster-crsht             2/2     Running   0          97s
pod/numaresourcesoperator-democluster-jp9mw             2/2     Running   0          97s
    Copy to Clipboard Toggle word wrap

After you install the NUMA Resources Operator, follow this procedure to deploy the NUMA-aware secondary pod scheduler.

Procedure

  1. Create the NUMAResourcesScheduler custom resource that deploys the NUMA-aware custom pod scheduler:

    1. Save the following minimal required YAML in the nro-scheduler.yaml file: 

      apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-rhel9:v4.19"
      1
      Copy to Clipboard Toggle word wrap
      1
      In a disconnected environment, make sure to configure the resolution of this image by either:
      • Creating an ImageTagMirrorSet custom resource (CR). For more information, see "Configuring image registry repository mirroring" in the "Additional resources" section.
      • Setting the URL to the disconnected registry.

    2. Create the NUMAResourcesScheduler CR by running the following command: 

      $ oc create -f nro-scheduler.yaml
      Copy to Clipboard Toggle word wrap
      Note

      In a hosted control plane cluster, run this command on the hosted control plane node.

  2. After a few seconds, run the following command to confirm the successful deployment of the required resources: 

    $ oc get all -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                    READY   STATUS    RESTARTS   AGE
pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s
pod/secondary-scheduler-847cb74f84-9whlm                1/1     Running   0          10m

NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/numaresourcesoperator-worker   3         3         3       3            3           node-role.kubernetes.io/worker=   98s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/numaresources-controller-manager   1/1     1            1           12m
deployment.apps/secondary-scheduler                1/1     1            1           10m

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/numaresources-controller-manager-7d9d84c58d   1         1         1       12m
replicaset.apps/secondary-scheduler-847cb74f84                1         1         1       10m
    Copy to Clipboard Toggle word wrap

To schedule workloads with the NUMA-aware scheduler, use deployment CRs that specify the minimum required resources. This ensures your cluster processes the workloads efficiently.

Before you schedule workloads with the NUMA-aware scheduler, ensure that you previouslu installed the topo-aware-scheduler, you applied the NUMAResourcesOperator and NUMAResourcesScheduler CRs, and that your cluster has a matching performance profile or kubeletconfig.

The example in the procedure uses NUMA-aware scheduling for a sample workload.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.

Procedure

  1. Get the name of the NUMA-aware scheduler that is deployed in the cluster by running the following command: 

    $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'
    Copy to Clipboard Toggle word wrap

    Example output

    "topo-aware-scheduler"
    Copy to Clipboard Toggle word wrap

  2. Create a Deployment CR that uses scheduler named topo-aware-scheduler, for example:

    1. Save the following YAML in the nro-deployment.yaml file: 

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: numa-deployment-1
  namespace: openshift-numaresources
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      schedulerName: topo-aware-scheduler
      containers:
      - name: ctnr
        image: quay.io/openshifttest/hello-openshift:openshift
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "100Mi"
            cpu: "10"
          requests:
            memory: "100Mi"
            cpu: "10"
      - name: ctnr2
        image: registry.access.redhat.com/rhel:latest
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c"]
        args: [ "while true; do sleep 1h; done;" ]
        resources:
          limits:
            memory: "100Mi"
            cpu: "8"
          requests:
            memory: "100Mi"
            cpu: "8"
      Copy to Clipboard Toggle word wrap

      spec.schedulerName: Specifies the scheduler name that must match the name of the NUMA-aware scheduler that is deployed in your cluster, such as topo-aware-scheduler.

    2. Create the Deployment CR by running the following command: 

      $ oc create -f nro-deployment.yaml
      Copy to Clipboard Toggle word wrap

Verification

  1. Verify that the deployment was successful: 

    $ oc get pods -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                READY   STATUS    RESTARTS   AGE
numa-deployment-1-6c4f5bdb84-wgn6g                  2/2     Running   0          5m2s
numaresources-controller-manager-7d9d84c58d-4v65j   1/1     Running   0          18m
numaresourcesoperator-worker-7d96r                  2/2     Running   4          43m
numaresourcesoperator-worker-crsht                  2/2     Running   2          43m
numaresourcesoperator-worker-jp9mw                  2/2     Running   2          43m
secondary-scheduler-847cb74f84-fpncj                1/1     Running   0          18m
    Copy to Clipboard Toggle word wrap

  2. Verify that the topo-aware-scheduler is scheduling the deployed pod by running the following command: 

    $ oc describe pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Example output

    Events:
  Type    Reason          Age    From                  Message
  ----    ------          ----   ----                  -------
  Normal  Scheduled       4m45s  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-6c4f5bdb84-wgn6g to worker-1
    Copy to Clipboard Toggle word wrap

    Note

    Deployments that request more resources than is available for scheduling will fail with a MinimumReplicasUnavailable error. The deployment succeeds when the required resources become available. Pods remain in the Pending state until the required resources are available.

  3. Verify that the expected allocated resources are listed for the node.

    1. Identify the node that is running the deployment pod by running the following command: 

      $ oc get pods -n openshift-numaresources -o wide
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
numa-deployment-1-6c4f5bdb84-wgn6g   0/2     Running   0          82m   10.128.2.50   worker-1   <none>  <none>
      Copy to Clipboard Toggle word wrap

    2. Run the following command with the name of that node that is running the deployment pod. 

      $ oc describe noderesourcetopologies.topology.node.k8s.io worker-1
      Copy to Clipboard Toggle word wrap

      Example output

      ...

Zones:
  Costs:
    Name:   node-0
    Value:  10
    Name:   node-1
    Value:  21
  Name:     node-0
  Resources:
    Allocatable:  39
    Available:    21
    Capacity:     40
    Name:         cpu
    Allocatable:  6442450944
    Available:    6442450944
    Capacity:     6442450944
    Name:         hugepages-1Gi
    Allocatable:  134217728
    Available:    134217728
    Capacity:     134217728
    Name:         hugepages-2Mi
    Allocatable:  262415904768
    Available:    262206189568
    Capacity:     270146007040
    Name:         memory
  Type:           Node
      Copy to Clipboard Toggle word wrap

      Resources.Available: Specifies the Available capacity that is reduced because of the resources that have been allocated to the guaranteed pod. Resources consumed by guaranteed pods are subtracted from the available node resources listed under noderesourcetopologies.topology.node.k8s.io.

  4. Resource allocations for pods with a Best-effort or Burstable quality of service (qosClass) are not reflected in the NUMA node resources under noderesourcetopologies.topology.node.k8s.io. If a pod’s consumed resources are not reflected in the node resource calculation, verify that the pod has qosClass of Guaranteed and the CPU request is an integer value, not a decimal value. You can verify the that the pod has a qosClass of Guaranteed by running the following command: 

    $ oc get pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources -o jsonpath="{ .status.qosClass }"
    Copy to Clipboard Toggle word wrap

    Example output

    Guaranteed
    Copy to Clipboard Toggle word wrap

As an optional task, you can improve scheduling behavior and troubleshoot suboptimal scheduling decisions by configuring the spec.nodeGroups specification in the NUMAResourcesOperator custom resource (CR). This configuration fine-tunes how daemons poll for available NUMA resources, providing advanced control over your polling operations.

The configuration options are listed as follows:

  • infoRefreshMode: Determines the trigger condition for polling the kubelet. The NUMA Resources Operator reports the resulting information to the API server.
  • infoRefreshPeriod: Determines the duration between polling updates.
  • podsFingerprinting: Determines if point-in-time information for the current set of pods running on a node is exposed in polling updates.
Note

The default value for podsFingerprinting is EnabledExclusiveResources. To optimize scheduler performance, set podsFingerprinting to either EnabledExclusiveResources or Enabled. Additionally, configure the cacheResyncPeriod in the NUMAResourcesScheduler custom resource (CR) to a value greater than 0. The cacheResyncPeriod specification helps to report more exact resource availability by monitoring pending resources on nodes.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.
  • Installed the NUMA Resources Operator.

Procedure

  • Configure the spec.nodeGroups specification in your NUMAResourcesOperator CR: 

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesOperator
metadata:
  name: numaresourcesoperator
spec:
  nodeGroups:
  - config:
      infoRefreshMode: Periodic
      infoRefreshPeriod: 10s
      podsFingerprinting: Enabled
    name: worker
# ...
    Copy to Clipboard Toggle word wrap

    where:

    spec.nodeGroups.config.infoRefreshMode
    Valid values are Periodic, Events, PeriodicAndEvents. Use Periodic to poll the kubelet at intervals that you define in infoRefreshPeriod. Use Events to poll the kubelet at every pod lifecycle event. Use PeriodicAndEvents to enable both methods.
    spec.nodeGroups.config.infoRefreshPeriod
    Specifies the polling interval for Periodic or PeriodicAndEvents refresh modes. The field is ignored if the refresh mode is Events.
    spec.nodeGroups.config.podsFingerprinting
    Valid values are Enabled, Disabled, and EnabledExclusiveResources. Setting to Enabled or EnabledExclusiveResources is a requirement for the cacheResyncPeriod specification in the NUMAResourcesScheduler.

Verification

  1. After you deploy the NUMA Resources Operator, verify that the node group configurations were applied by running the following command: 

    $ oc get numaresop numaresourcesoperator -o json | jq '.status'
    Copy to Clipboard Toggle word wrap

    Example output

          ...

        "config": {
        "infoRefreshMode": "Periodic",
        "infoRefreshPeriod": "10s",
        "podsFingerprinting": "Enabled"
      },
      "name": "worker"

      ...
    Copy to Clipboard Toggle word wrap

12.8. Troubleshooting NUMA-aware scheduling
Copy link

To troubleshoot common problems with NUMA-aware pod scheduling, perform the following steps.

Prerequisites

  • Install the OpenShift Container Platform CLI (oc).
  • Log in as a user with cluster-admin privileges.
  • Install the NUMA Resources Operator and deploy the NUMA-aware secondary scheduler.

Procedure

  1. Verify that the noderesourcetopologies CRD is deployed in the cluster by running the following command: 

    $ oc get crd | grep noderesourcetopologies
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                              CREATED AT
noderesourcetopologies.topology.node.k8s.io                       2022-01-18T08:28:06Z
    Copy to Clipboard Toggle word wrap

  2. Check that the NUMA-aware scheduler name matches the name specified in your NUMA-aware workloads by running the following command: 

    $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'
    Copy to Clipboard Toggle word wrap

    Example output

    topo-aware-scheduler
    Copy to Clipboard Toggle word wrap

  3. Verify that NUMA-aware schedulable nodes have the noderesourcetopologies CR applied to them. Run the following command: 

    $ oc get noderesourcetopologies.topology.node.k8s.io
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                    AGE
compute-0.example.com   17h
compute-1.example.com   17h
    Copy to Clipboard Toggle word wrap

    Note

    The number of nodes should equal the number of worker nodes that are configured by the machine config pool (mcp) worker definition.

  4. Verify the NUMA zone granularity for all schedulable nodes by running the following command: 

    $ oc get noderesourcetopologies.topology.node.k8s.io -o yaml
    Copy to Clipboard Toggle word wrap

    Example output

    apiVersion: v1
items:
- apiVersion: topology.node.k8s.io/v1
  kind: NodeResourceTopology
  metadata:
    annotations:
      k8stopoawareschedwg/rte-update: periodic
    creationTimestamp: "2022-06-16T08:55:38Z"
    generation: 63760
    name: worker-0
    resourceVersion: "8450223"
    uid: 8b77be46-08c0-4074-927b-d49361471590
  topologyPolicies:
  - SingleNUMANodeContainerLevel
  zones:
  - costs:
    - name: node-0
      value: 10
    - name: node-1
      value: 21
    name: node-0
    resources:
    - allocatable: "38"
      available: "38"
      capacity: "40"
      name: cpu
    - allocatable: "134217728"
      available: "134217728"
      capacity: "134217728"
      name: hugepages-2Mi
    - allocatable: "262352048128"
      available: "262352048128"
      capacity: "270107316224"
      name: memory
    - allocatable: "6442450944"
      available: "6442450944"
      capacity: "6442450944"
      name: hugepages-1Gi
    type: Node
  - costs:
    - name: node-0
      value: 21
    - name: node-1
      value: 10
    name: node-1
    resources:
    - allocatable: "268435456"
      available: "268435456"
      capacity: "268435456"
      name: hugepages-2Mi
    - allocatable: "269231067136"
      available: "269231067136"
      capacity: "270573244416"
      name: memory
    - allocatable: "40"
      available: "40"
      capacity: "40"
      name: cpu
    - allocatable: "1073741824"
      available: "1073741824"
      capacity: "1073741824"
      name: hugepages-1Gi
    type: Node
- apiVersion: topology.node.k8s.io/v1
  kind: NodeResourceTopology
  metadata:
    annotations:
      k8stopoawareschedwg/rte-update: periodic
    creationTimestamp: "2022-06-16T08:55:37Z"
    generation: 62061
    name: worker-1
    resourceVersion: "8450129"
    uid: e8659390-6f8d-4e67-9a51-1ea34bba1cc3
  topologyPolicies:
  - SingleNUMANodeContainerLevel
  zones:
    1 
    
  - costs:
    - name: node-0
      value: 10
    - name: node-1
      value: 21
    name: node-0
    resources:
    2 
    
    - allocatable: "38"
      available: "38"
      capacity: "40"
      name: cpu
    - allocatable: "6442450944"
      available: "6442450944"
      capacity: "6442450944"
      name: hugepages-1Gi
    - allocatable: "134217728"
      available: "134217728"
      capacity: "134217728"
      name: hugepages-2Mi
    - allocatable: "262391033856"
      available: "262391033856"
      capacity: "270146301952"
      name: memory
    type: Node
  - costs:
    - name: node-0
      value: 21
    - name: node-1
      value: 10
    name: node-1
    resources:
    - allocatable: "40"
      available: "40"
      capacity: "40"
      name: cpu
    - allocatable: "1073741824"
      available: "1073741824"
      capacity: "1073741824"
      name: hugepages-1Gi
    - allocatable: "268435456"
      available: "268435456"
      capacity: "268435456"
      name: hugepages-2Mi
    - allocatable: "269192085504"
      available: "269192085504"
      capacity: "270534262784"
      name: memory
    type: Node
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
    Copy to Clipboard Toggle word wrap

    1
    Each stanza under zones describes the resources for a single NUMA zone.
    2
    resources describes the current state of the NUMA zone resources. Check that resources listed under items.zones.resources.available correspond to the exclusive NUMA zone resources allocated to each guaranteed pod.

12.8.1. Reporting more exact resource availability
Copy link

Enable the cacheResyncPeriod specification to help the NUMA Resources Operator report more exact resource availability by monitoring pending resources on nodes and synchronizing this information in the scheduler cache at a defined interval. This also helps to minimize Topology Affinity Error errors because of sub-optimal scheduling decisions. The lower the interval, the greater the network load. The cacheResyncPeriod specification is disabled by default.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Delete the currently running NUMAResourcesScheduler resource:

    1. Get the active NUMAResourcesScheduler by running the following command: 

      $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   92m
      Copy to Clipboard Toggle word wrap

    2. Delete the secondary scheduler resource by running the following command: 

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  2. Save the following YAML in the file nro-scheduler-cacheresync.yaml. This example changes the log level to Debug: 

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.19"
  cacheResyncPeriod: "5s"
    1
    Copy to Clipboard Toggle word wrap
    1
    Enter an interval value in seconds for synchronization of the scheduler cache. A value of 5s is typical for most implementations.

  3. Create the updated NUMAResourcesScheduler resource by running the following command: 

    $ oc create -f nro-scheduler-cacheresync.yaml
    Copy to Clipboard Toggle word wrap

    Example output

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

Verification steps

  1. Check that the NUMA-aware scheduler was successfully deployed:

    1. Run the following command to check that the CRD is created successfully: 

      $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                                              CREATED AT
numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. Check that the new custom scheduler is available by running the following command: 

      $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

  2. Check that the logs for the scheduler show the increased log level:

    1. Get the list of pods running in the openshift-numaresources namespace by running the following command: 

      $ oc get pods -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                               READY   STATUS    RESTARTS   AGE
numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m
      Copy to Clipboard Toggle word wrap

    2. Get the logs for the secondary scheduler pod by running the following command: 

      $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      Example output

      ...
I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      Copy to Clipboard Toggle word wrap

To optimize the processing of high-performance workloads, change the default placement behavior of the NUMA-aware secondary scheduler. With this configuration, you can assign workloads to a specific NUMA node within a compute node instead of relying on default resource availability.

If you want to change where the workloads run, you can add the scoringStrategy setting to the NUMAResourcesScheduler custom resource and set its value to either MostAllocated or BalancedAllocation.

Prerequisites

  • Installed the OpenShift CLI (oc).
  • Logged in as a user with cluster-admin privileges.

Procedure

  1. Delete the currently running NUMAResourcesScheduler resource by using the following steps:

    1. Get the active NUMAResourcesScheduler by running the following command: 

      $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   92m
      Copy to Clipboard Toggle word wrap

    2. Delete the secondary scheduler resource by running the following command: 

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  2. Save the following YAML in the file nro-scheduler-mostallocated.yaml. This example changes the scoringStrategy to MostAllocated: 

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v{product-version}"
  scoringStrategy:
        type: "MostAllocated"
# ...
    Copy to Clipboard Toggle word wrap

    spec.imageSpec.scoringStrategy: If the scoringStrategy configuration is omitted, the default of LeastAllocated applies.

  3. Create the updated NUMAResourcesScheduler resource by running the following command: 

    $ oc create -f nro-scheduler-mostallocated.yaml
    Copy to Clipboard Toggle word wrap

    Example output

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

Verification

  1. Check that the NUMA-aware scheduler was successfully deployed by using the following steps:

    1. Run the following command to check that the custom resource definition (CRD) is created successfully: 

      $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                                              CREATED AT
numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. Check that the new custom scheduler is available by running the following command: 

      $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

  2. Verify that the ScoringStrategy has been applied correctly by running the following command to check the relevant ConfigMap resource for the scheduler: 

    $ oc get -n openshift-numaresources cm topo-aware-scheduler-config -o yaml | grep scoring -A 1
    Copy to Clipboard Toggle word wrap

    Example output

    scoringStrategy:
  type: MostAllocated
    Copy to Clipboard Toggle word wrap

12.8.3. Checking the NUMA-aware scheduler logs
Copy link

Troubleshoot problems with the NUMA-aware scheduler by reviewing the logs. If required, you can increase the scheduler log level by modifying the spec.logLevel field of the NUMAResourcesScheduler resource. Acceptable values are Normal, Debug, and Trace, with Trace being the most verbose option.

Note

To change the log level of the secondary scheduler, delete the running scheduler resource and re-deploy it with the changed log level. The scheduler is unavailable for scheduling new workloads during this downtime.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Delete the currently running NUMAResourcesScheduler resource:

    1. Get the active NUMAResourcesScheduler by running the following command: 

      $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   90m
      Copy to Clipboard Toggle word wrap

    2. Delete the secondary scheduler resource by running the following command: 

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  2. Save the following YAML in the file nro-scheduler-debug.yaml. This example changes the log level to Debug: 

    apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.19"
  logLevel: Debug
    Copy to Clipboard Toggle word wrap

  3. Create the updated Debug logging NUMAResourcesScheduler resource by running the following command: 

    $ oc create -f nro-scheduler-debug.yaml
    Copy to Clipboard Toggle word wrap

    Example output

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

Verification steps

  1. Check that the NUMA-aware scheduler was successfully deployed:

    1. Run the following command to check that the CRD is created successfully: 

      $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                                              CREATED AT
numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. Check that the new custom scheduler is available by running the following command: 

      $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     AGE
numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

  2. Check that the logs for the scheduler shows the increased log level:

    1. Get the list of pods running in the openshift-numaresources namespace by running the following command: 

      $ oc get pods -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                               READY   STATUS    RESTARTS   AGE
numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m
      Copy to Clipboard Toggle word wrap

    2. Get the logs for the secondary scheduler pod by running the following command: 

      $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      Example output

      ...
I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      Copy to Clipboard Toggle word wrap

Troubleshoot noderesourcetopologies objects where unexpected results are occurring by inspecting the corresponding resource-topology-exporter logs.

Note

It is recommended that NUMA resource topology exporter instances in the cluster are named for nodes they refer to. For example, a worker node with the name worker should have a corresponding noderesourcetopologies object called worker.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Get the daemonsets managed by the NUMA Resources Operator. Each daemonset has a corresponding nodeGroup in the NUMAResourcesOperator CR. Run the following command: 

    $ oc get numaresourcesoperators.nodetopology.openshift.io numaresourcesoperator -o jsonpath="{.status.daemonsets[0]}"
    Copy to Clipboard Toggle word wrap

    Example output

    {"name":"numaresourcesoperator-worker","namespace":"openshift-numaresources"}
    Copy to Clipboard Toggle word wrap

  2. Get the label for the daemonset of interest using the value for name from the previous step: 

    $ oc get ds -n openshift-numaresources numaresourcesoperator-worker -o jsonpath="{.spec.selector.matchLabels}"
    Copy to Clipboard Toggle word wrap

    Example output

    {"name":"resource-topology"}
    Copy to Clipboard Toggle word wrap

  3. Get the pods using the resource-topology label by running the following command: 

    $ oc get pods -n openshift-numaresources -l name=resource-topology -o wide
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                 READY   STATUS    RESTARTS   AGE    IP            NODE
numaresourcesoperator-worker-5wm2k   2/2     Running   0          2d1h   10.135.0.64   compute-0.example.com
numaresourcesoperator-worker-pb75c   2/2     Running   0          2d1h   10.132.2.33   compute-1.example.com
    Copy to Clipboard Toggle word wrap

  4. Examine the logs of the resource-topology-exporter container running on the worker pod that corresponds to the node you are troubleshooting. Run the following command: 

    $ oc logs -n openshift-numaresources -c resource-topology-exporter numaresourcesoperator-worker-pb75c
    Copy to Clipboard Toggle word wrap

    Example output

    I0221 13:38:18.334140       1 main.go:206] using sysinfo:
reservedCpus: 0,1
reservedMemory:
  "0": 1178599424
I0221 13:38:18.334370       1 main.go:67] === System information ===
I0221 13:38:18.334381       1 sysinfo.go:231] cpus: reserved "0-1"
I0221 13:38:18.334493       1 sysinfo.go:237] cpus: online "0-103"
I0221 13:38:18.546750       1 main.go:72]
cpus: allocatable "2-103"
hugepages-1Gi:
  numa cell 0 -> 6
  numa cell 1 -> 1
hugepages-2Mi:
  numa cell 0 -> 64
  numa cell 1 -> 128
memory:
  numa cell 0 -> 45758Mi
  numa cell 1 -> 48372Mi
    Copy to Clipboard Toggle word wrap

If you install the NUMA Resources Operator in a cluster with misconfigured cluster settings, in some circumstances, the Operator is shown as active but the logs of the resource topology exporter (RTE) daemon set pods show that the configuration for the RTE is missing, for example: 

Info: couldn't find configuration in "/etc/resource-topology-exporter/config.yaml"
Copy to Clipboard Toggle word wrap

This log message indicates that the kubeletconfig with the required configuration was not properly applied in the cluster, resulting in a missing RTE configmap. For example, the following cluster is missing a numaresourcesoperator-worker configmap custom resource (CR): 

$ oc get configmap
Copy to Clipboard Toggle word wrap

Example output

NAME                           DATA   AGE
0e2a6bd3.openshift-kni.io      0      6d21h
kube-root-ca.crt               1      6d21h
openshift-service-ca.crt       1      6d21h
topo-aware-scheduler-config    1      6d18h
Copy to Clipboard Toggle word wrap

In a correctly configured cluster, oc get configmap also returns a numaresourcesoperator-worker configmap CR.

Prerequisites

  • Install the OpenShift Container Platform CLI (oc).
  • Log in as a user with cluster-admin privileges.
  • Install the NUMA Resources Operator and deploy the NUMA-aware secondary scheduler.

Procedure

  1. Compare the values for spec.machineConfigPoolSelector.matchLabels in kubeletconfig and metadata.labels in the MachineConfigPool (mcp) worker CR using the following commands:

    1. Check the kubeletconfig labels by running the following command: 

      $ oc get kubeletconfig -o yaml
      Copy to Clipboard Toggle word wrap

      Example output

      machineConfigPoolSelector:
  matchLabels:
    cnf-worker-tuning: enabled
      Copy to Clipboard Toggle word wrap

    2. Check the mcp labels by running the following command: 

      $ oc get mcp worker -o yaml
      Copy to Clipboard Toggle word wrap

      Example output

      labels:
  machineconfiguration.openshift.io/mco-built-in: ""
  pools.operator.machineconfiguration.openshift.io/worker: ""
      Copy to Clipboard Toggle word wrap

      The cnf-worker-tuning: enabled label is not present in the MachineConfigPool object.

  2. Edit the MachineConfigPool CR to include the missing label, for example: 

    $ oc edit mcp worker -o yaml
    Copy to Clipboard Toggle word wrap

    Example output

    labels:
  machineconfiguration.openshift.io/mco-built-in: ""
  pools.operator.machineconfiguration.openshift.io/worker: ""
  cnf-worker-tuning: enabled
    Copy to Clipboard Toggle word wrap

  3. Apply the label changes and wait for the cluster to apply the updated configuration. Run the following command:

Verification

  • Check that the missing numaresourcesoperator-worker configmap CR is applied: 

    $ oc get configmap
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                           DATA   AGE
0e2a6bd3.openshift-kni.io      0      6d21h
kube-root-ca.crt               1      6d21h
numaresourcesoperator-worker   1      5m
openshift-service-ca.crt       1      6d21h
topo-aware-scheduler-config    1      6d18h
    Copy to Clipboard Toggle word wrap

12.8.6. Collecting NUMA Resources Operator data
Copy link

You can use the oc adm must-gather CLI command to collect information about your cluster, including features and objects associated with the NUMA Resources Operator.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift CLI (oc).

Procedure

  • To collect NUMA Resources Operator data with must-gather, you must specify the NUMA Resources Operator must-gather image. 

    $ oc adm must-gather --image=registry.redhat.io/openshift4/numaresources-must-gather-rhel9:v4.19
    Copy to Clipboard Toggle word wrap
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top