Scalability and performance

1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} 1
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 2
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} 3
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 4
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} 5
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring

   1

   The default value of Prometheus retention is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.

   2 4

   The storage class for your cluster.

   3

   A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

   5

   A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Add values for the retention period, storage class, and storage sizes.
3. Save the file.

4. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yaml

Procedure

1. Attach the new disk to the cluster and verify that the disk is detected in the node by running the lsblk command in a debug shell:

   $ oc debug node/<node_name>

   # lsblk

   Note the device name of the new disk reported by the lsblk command.

2. Create the following script and name it etcd-find-secondary-device.sh:

   #!/bin/bash
   set -uo pipefail
   
   for device in <device_type_glob>; do 1
   /usr/sbin/blkid "${device}" &> /dev/null
    if [ $? == 2  ]; then
       echo "secondary device found ${device}"
       echo "creating filesystem for etcd mount"
       mkfs.xfs -L var-lib-etcd -f "${device}" &> /dev/null
       udevadm settle
       touch /etc/var-lib-etcd-mount
       exit
    fi
   done
   echo "Couldn't find secondary block device!" >&2
   exit 77

   1

   Replace <device_type_glob> with a shell glob for your block device type. For SCSI or SATA drives, use /dev/sd*; for virtual drives, use /dev/vd*; for NVMe drives, use /dev/nvme*[0-9]*n*.

3. Create a base64-encoded string from the etcd-find-secondary-device.sh script and note its contents:

   $ base64 -w0 etcd-find-secondary-device.sh

4. Create a MachineConfig YAML file named etcd-mc.yml with contents such as the following:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 98-var-lib-etcd
   spec:
     config:
       ignition:
         version: 3.1.0
       storage:
         files:
           - path: /etc/find-secondary-device
             mode: 0755
             contents:
               source: data:text/plain;charset=utf-8;base64,<encoded_etcd_find_secondary_device_script> 1
       systemd:
         units:
           - name: find-secondary-device.service
             enabled: true
             contents: |
               [Unit]
               Description=Find secondary device
               DefaultDependencies=false
               After=systemd-udev-settle.service
               Before=local-fs-pre.target
               ConditionPathExists=!/etc/var-lib-etcd-mount
   
               [Service]
               RemainAfterExit=yes
               ExecStart=/etc/find-secondary-device
   
               RestartForceExitStatus=77
   
               [Install]
               WantedBy=multi-user.target
           - name: var-lib-etcd.mount
             enabled: true
             contents: |
               [Unit]
               Before=local-fs.target
   
               [Mount]
               What=/dev/disk/by-label/var-lib-etcd
               Where=/var/lib/etcd
               Type=xfs
               TimeoutSec=120s
   
               [Install]
               RequiredBy=local-fs.target
           - name: sync-var-lib-etcd-to-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Sync etcd data if new mount is empty
               DefaultDependencies=no
               After=var-lib-etcd.mount var.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 1
               ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
               ExecStart=/usr/sbin/semanage fcontext -a -t container_var_lib_t '/var/lib/etcd(/.*)?'
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 0
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target
           - name: restorecon-var-lib-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Restore recursive SELinux security contexts
               DefaultDependencies=no
               After=var-lib-etcd.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecStart=/sbin/restorecon -R /var/lib/etcd/
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target

   1

   Replace <encoded_etcd_find_secondary_device_script> with the encoded script contents that you noted.

1. gp3 disks with a baseline performance of 3000 IOPS and 125 MiB per second are used for control plane/etcd nodes because etcd is latency sensitive. gp3 volumes do not use burst performance.
2. Infra nodes are used to host Monitoring, Ingress, and Registry components to ensure they have enough resources to run at large scale.
3. Workload node is dedicated to run performance and scalability workload generators.
4. Larger disk size is used so that there is enough space to store the large amounts of data that is collected during the performance and scalability test run.
5. Cluster is scaled in iterations and performance and scalability tests are executed at the specified node counts.

1. io1 disks with 120 / 10 IOPS per GiB are used for control plane/etcd nodes as etcd is I/O intensive and latency sensitive.
2. Infra nodes are used to host Monitoring, Ingress, and Registry components to ensure they have enough resources to run at large scale.
3. Workload node is dedicated to run performance and scalability workload generators.
4. Larger disk size is used so that there is enough space to store the large amounts of data that is collected during the performance and scalability test run.
5. Cluster is scaled in iterations.

1. Nodes are distributed between two logical control units (LCUs) to optimize disk I/O load of the control plane/etcd nodes as etcd is I/O intensive and latency sensitive. Etcd I/O demand should not interfere with other workloads.
2. Four compute nodes are used for the tests running several iterations with 100/250/500 pods at the same time. First, idling pods were used to evaluate if pods can be instanced. Next, a network and CPU demanding client/server workload were used to evaluate the stability of the system under stress. Client and server pods were pairwise deployed and each pair was spread over two compute nodes.
3. No separate workload node was used. The workload simulates a microservice workload between two compute nodes.
4. Physical number of processors used is six Integrated Facilities for Linux (IFLs).
5. Total physical memory used is 512 GiB.

Procedure

1. To determine how many GPUs are available on a node in your cluster, use the following command:

   $ oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'

   Example output

                       openshift.com/gpu-accelerator=true
   Capacity:
    nvidia.com/gpu:  2
   Allocatable:
    nvidia.com/gpu:  2
    nvidia.com/gpu:  0           0

   In this example, 2 GPUs are available.

2. Use this command to set a quota in the namespace nvidia. In this example, the quota is 1:

   $ cat gpu-quota.yaml

   Example output

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: gpu-quota
     namespace: nvidia
   spec:
     hard:
       requests.nvidia.com/gpu: 1

3. Create the quota with the following command:

   $ oc create -f gpu-quota.yaml

   Example output

   resourcequota/gpu-quota created

4. Verify that the namespace has the correct quota set using the following command:

   $ oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  0     1

5. Run a pod that asks for a single GPU with the following command:

   $ oc create pod gpu-pod.yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: gpu-pod-s46h7
     namespace: nvidia
   spec:
     restartPolicy: OnFailure
     containers:
     - name: rhel7-gpu-pod
       image: rhel7
       env:
         - name: NVIDIA_VISIBLE_DEVICES
           value: all
         - name: NVIDIA_DRIVER_CAPABILITIES
           value: "compute,utility"
         - name: NVIDIA_REQUIRE_CUDA
           value: "cuda>=5.0"
   
       command: ["sleep"]
       args: ["infinity"]
   
       resources:
         limits:
           nvidia.com/gpu: 1

6. Verify that the pod is running bwith the following command:

   $ oc get pods

   Example output

   NAME              READY     STATUS      RESTARTS   AGE
   gpu-pod-s46h7     1/1       Running     0          1m

7. Verify that the quota Used counter is correct by running the following command:

   $ oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  1     1

8. Using the following command, attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:

   $ oc create -f gpu-pod.yaml

   Example output

   Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1

   This Forbidden error message occurs because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.

1. First, get the list of quotas defined in the project. For example, for a project called demoproject:

   $ oc get quota -n demoproject
   NAME                AGE
   besteffort          11m
   compute-resources   2m
   core-object-counts  29m

2. Describe the quota you are interested in, for example the core-object-counts quota:

   $ oc describe quota core-object-counts -n demoproject
   Name:			core-object-counts
   Namespace:		demoproject
   Resource		Used	Hard
   --------		----	----
   configmaps		3	10
   persistentvolumeclaims	0	4
   replicationcontrollers	3	20
   secrets			9	10
   services		2	10

Procedure

1. Create the object:

   $ oc create -f <limit_range_file> -n <project>

Procedure

1. Get the list of limit range objects that are defined in the project. For example, a project called demoproject:

   $ oc get limits -n demoproject

   Example Output

   NAME              AGE
   resource-limits   6d

2. Describe the limit range. For example, for a limit range called resource-limits:

   $ oc describe limits resource-limits -n demoproject

   Example Output

   Name:                           resource-limits
   Namespace:                      demoproject
   Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
   ----                            --------                ---     ---     --------------- -------------   -----------------------
   Pod                             cpu                     200m    2       -               -               -
   Pod                             memory                  6Mi     1Gi     -               -               -
   Container                       cpu                     100m    2       200m            300m            10
   Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
   openshift.io/Image              storage                 -       1Gi     -               -               -
   openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
   openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -

Procedure

1. Copy the following MCO sample profile into a YAML file. For example, enable-rfs.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 50-enable-rfs
   spec:
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all%20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D%3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20-c%20%27for%20x%20in%20/sys/%24DEVPATH/queues/rx-%2A%3B%20do%20echo%208192%20%3E%20%24x/rps_flow_cnt%3B%20%20done%27%22%0A
           filesystem: root
           mode: 0644
           path: /etc/udev/rules.d/70-persistent-net.rules
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20define%20sock%20flow%20enbtried%20for%20%20Receive%20Flow%20Steering%20%28RFS%29%0Anet.core.rps_sock_flow_entries%3D8192%0A
           filesystem: root
           mode: 0644
           path: /etc/sysctl.d/95-enable-rps.conf

2. Create the MCO profile:

   $ oc create -f enable-rfs.yaml

3. Verify that an entry named 50-enable-rfs is listed:

   $ oc get mc

4. To deactivate, enter:

   $ oc delete mc 50-enable-rfs

Procedure

1. Copy the following MCO sample profile into a YAML file for the control plane node. For example, 05-master-kernelarg-hpav.yaml:

   $ cat 05-master-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 05-master-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

2. Copy the following MCO sample profile into a YAML file for the compute node. For example, 05-worker-kernelarg-hpav.yaml:

   $ cat 05-worker-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 05-worker-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

   Note

   You must modify the rd.dasd arguments to fit the device IDs.

3. Create the MCO profiles:

   $ oc create -f 05-master-kernelarg-hpav.yaml

   $ oc create -f 05-worker-kernelarg-hpav.yaml

4. To deactivate, enter:

   $ oc delete -f 05-master-kernelarg-hpav.yaml

   $ oc delete -f 05-worker-kernelarg-hpav.yaml

Procedure

1. Open /etc/libvirt/qemu.conf with an editor of your choice.
2. Go to the cgroup_controllers line.
3. Duplicate the entire line and remove the leading number sign (#) from the copy.

4. Remove the cpuset entry, as follows:

   cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ]

5. For the new setting to take effect, you must restart the libvirtd daemon:

   1. Stop all virtual machines.

   2. Run the following command:

      # systemctl restart libvirtd

   3. Restart the virtual machines.

Procedure

1. Create a namespace for the NUMA Resources Operator:

   1. Save the following YAML in the nro-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-numaresources

   2. Create the Namespace CR by running the following command:

      $ oc create -f nro-namespace.yaml

2. Create the Operator group for the NUMA Resources Operator:

   1. Save the following YAML in the nro-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        targetNamespaces:
        - openshift-numaresources

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f nro-operatorgroup.yaml

3. Create the subscription for the NUMA Resources Operator:

   1. Save the following YAML in the nro-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        channel: "4.12"
        name: numaresources-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR by running the following command:

      $ oc create -f nro-sub.yaml

Verification

1. Verify that the installation succeeded by inspecting the CSV resource in the openshift-numaresources namespace. Run the following command:

   $ oc get csv -n openshift-numaresources

   Example output

   NAME                             DISPLAY                  VERSION   REPLACES   PHASE
   numaresources-operator.v4.12.2   numaresources-operator   4.12.2               Succeeded

Procedure

1. Create a namespace for the NUMA Resources Operator:

   1. In the OpenShift Container Platform web console, click Administration → Namespaces.
   2. Click Create Namespace, enter openshift-numaresources in the Name field, and then click Create.

2. Install the NUMA Resources Operator:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose numaresources-operator from the list of available Operators, and then click Install.
   3. In the Installed Namespaces field, select the openshift-numaresources namespace, and then click Install.

3. Optional: Verify that the NUMA Resources Operator installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that NUMA Resources Operator is listed in the openshift-numaresources namespace with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the default project.

Procedure

1. Create the MachineConfigPool custom resource that enables custom kubelet configurations for worker nodes:

   1. Save the following YAML in the nro-machineconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        labels:
          cnf-worker-tuning: enabled
          machineconfiguration.openshift.io/mco-built-in: ""
          pools.operator.machineconfiguration.openshift.io/worker: ""
        name: worker
      spec:
        machineConfigSelector:
          matchLabels:
            machineconfiguration.openshift.io/role: worker
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""

   2. Create the MachineConfigPool CR by running the following command:

      $ oc create -f nro-machineconfig.yaml

2. Create the NUMAResourcesOperator custom resource:

   1. Save the following minimal required YAML file example as nrop.yaml:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 1

      1

      This should match the MachineConfigPool that you want to configure the NUMA Resources Operator on. For example, you might have created a MachineConfigPool named worker-cnf that designates a set of nodes expected to run telecommunications workloads.

   2. Create the NUMAResourcesOperator CR by running the following command:

      $ oc create -f nrop.yaml

      Note

      Creating the NUMAResourcesOperator triggers a reboot on the corresponding machine config pool and therefore the affected node.

Verification

1. Verify that the NUMA Resources Operator deployed successfully by running the following command:

   $ oc get numaresourcesoperators.nodetopology.openshift.io

   Example output

   NAME                    AGE
   numaresourcesoperator   27s

2. After a few minutes, run the following command to verify that the required resources deployed successfully:

   $ oc get all -n openshift-numaresources

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
   pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
   pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
   pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s

Procedure

1. Create the KubeletConfig custom resource that configures the pod admittance policy for the machine profile:

2. Create the NUMAResourcesScheduler custom resource that deploys the NUMA-aware custom pod scheduler:

   1. Save the following minimal required YAML in the nro-scheduler.yaml file:

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-rhel9:v4.12"

   2. Create the NUMAResourcesScheduler CR by running the following command:

      $ oc create -f nro-scheduler.yaml

3. After a few seconds, run the following command to confirm the successful deployment of the required resources:

   $ oc get all -n openshift-numaresources

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
   pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
   pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
   pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s
   pod/secondary-scheduler-847cb74f84-9whlm                1/1     Running   0          10m
   
   NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
   daemonset.apps/numaresourcesoperator-worker   3         3         3       3            3           node-role.kubernetes.io/worker=   98s
   
   NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/numaresources-controller-manager   1/1     1            1           12m
   deployment.apps/secondary-scheduler                1/1     1            1           10m
   
   NAME                                                          DESIRED   CURRENT   READY   AGE
   replicaset.apps/numaresources-controller-manager-7d9d84c58d   1         1         1       12m
   replicaset.apps/secondary-scheduler-847cb74f84                1         1         1       10m

Procedure

1. Create the KubeletConfig custom resource (CR) that configures the pod admittance policy for the machine profile:

   1. Save the following YAML in the nro-kubeletconfig.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: "" 1
        kubeletConfig:
          cpuManagerPolicy: "static" 2
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1" 3
          memoryManagerPolicy: "Static" 4
          evictionHard:
            memory.available: "100Mi"
          kubeReserved:
            memory: "512Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 5

      1

      Adjust this label to match the machineConfigPoolSelector in the NUMAResourcesOperator CR.

      2

      For cpuManagerPolicy, static must use a lowercase s.

      3

      Adjust this based on the CPU on your nodes.

      4

      For memoryManagerPolicy, Static must use an uppercase S.

      5

      topologyManagerPolicy must be set to single-numa-node.

   2. Create the KubeletConfig CR by running the following command:

      $ oc create -f nro-kubeletconfig.yaml

      Note

      Applying performance profile or KubeletConfig automatically triggers rebooting of the nodes. If no reboot is triggered, you can troubleshoot the issue by looking at the labels in KubeletConfig that address the node group.

Procedure

1. Get the name of the NUMA-aware scheduler that is deployed in the cluster by running the following command:

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Example output

   "topo-aware-scheduler"

2. Create a Deployment CR that uses scheduler named topo-aware-scheduler, for example:

   1. Save the following YAML in the nro-deployment.yaml file:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 1
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"

      1

      schedulerName must match the name of the NUMA-aware scheduler that is deployed in your cluster, for example topo-aware-scheduler.

   2. Create the Deployment CR by running the following command:

      $ oc create -f nro-deployment.yaml

Verification

1. Verify that the deployment was successful:

   $ oc get pods -n openshift-numaresources

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   numa-deployment-1-6c4f5bdb84-wgn6g                  2/2     Running   0          5m2s
   numaresources-controller-manager-7d9d84c58d-4v65j   1/1     Running   0          18m
   numaresourcesoperator-worker-7d96r                  2/2     Running   4          43m
   numaresourcesoperator-worker-crsht                  2/2     Running   2          43m
   numaresourcesoperator-worker-jp9mw                  2/2     Running   2          43m
   secondary-scheduler-847cb74f84-fpncj                1/1     Running   0          18m

2. Verify that the topo-aware-scheduler is scheduling the deployed pod by running the following command:

   $ oc describe pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources

   Example output

   Events:
     Type    Reason          Age    From                  Message
     ----    ------          ----   ----                  -------
     Normal  Scheduled       4m45s  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-6c4f5bdb84-wgn6g to worker-1

   Note

   Deployments that request more resources than is available for scheduling will fail with a MinimumReplicasUnavailable error. The deployment succeeds when the required resources become available. Pods remain in the Pending state until the required resources are available.

3. Verify that the expected allocated resources are listed for the node.

   1. Identify the node that is running the deployment pod by running the following command:

      $ oc get pods -n openshift-numaresources -o wide

      Example output

      NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
      numa-deployment-1-6c4f5bdb84-wgn6g   0/2     Running   0          82m   10.128.2.50   worker-1   <none>  <none>

   2. Run the following command with the name of that node that is running the deployment pod.

      $ oc describe noderesourcetopologies.topology.node.k8s.io worker-1

      Example output

      ...
      
      Zones:
        Costs:
          Name:   node-0
          Value:  10
          Name:   node-1
          Value:  21
        Name:     node-0
        Resources:
          Allocatable:  39
          Available:    21 1
          Capacity:     40
          Name:         cpu
          Allocatable:  6442450944
          Available:    6442450944
          Capacity:     6442450944
          Name:         hugepages-1Gi
          Allocatable:  134217728
          Available:    134217728
          Capacity:     134217728
          Name:         hugepages-2Mi
          Allocatable:  262415904768
          Available:    262206189568
          Capacity:     270146007040
          Name:         memory
        Type:           Node

      1

      The Available capacity is reduced because of the resources that have been allocated to the guaranteed pod.

      Resources consumed by guaranteed pods are subtracted from the available node resources listed under noderesourcetopologies.topology.node.k8s.io.

4. Resource allocations for pods with a Best-effort or Burstable quality of service (qosClass) are not reflected in the NUMA node resources under noderesourcetopologies.topology.node.k8s.io. If a pod’s consumed resources are not reflected in the node resource calculation, verify that the pod has qosClass of Guaranteed by running the following command:

   $ oc get pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources -o jsonpath="{ .status.qosClass }"

   Example output

   Guaranteed

Procedure

1. Delete the currently running NUMAResourcesScheduler resource:

   1. Get the active NUMAResourcesScheduler by running the following command:

      $ oc get NUMAResourcesScheduler

      Example output

      NAME                     AGE
      numaresourcesscheduler   92m

   2. Delete the secondary scheduler resource by running the following command:

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted

2. Save the following YAML in the file nro-scheduler-cacheresync.yaml. This example changes the log level to Debug:

   apiVersion: nodetopology.openshift.io/v1
   kind: NUMAResourcesScheduler
   metadata:
     name: numaresourcesscheduler
   spec:
     imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"
     cacheResyncPeriod: "5s" 1

   1

   Enter an interval value in seconds for synchronization of the scheduler cache. A value of 5s is typical for most implementations.

3. Create the updated NUMAResourcesScheduler resource by running the following command:

   $ oc create -f nro-scheduler-cacheresync.yaml

   Example output

   numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created

Verification steps

1. Check that the NUMA-aware scheduler was successfully deployed:

   1. Run the following command to check that the CRD is created successfully:

      $ oc get crd | grep numaresourcesschedulers

      Example output

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z

   2. Check that the new custom scheduler is available by running the following command:

      $ oc get numaresourcesschedulers.nodetopology.openshift.io

      Example output

      NAME                     AGE
      numaresourcesscheduler   3h26m

2. Check that the logs for the scheduler show the increased log level:

   1. Get the list of pods running in the openshift-numaresources namespace by running the following command:

      $ oc get pods -n openshift-numaresources

      Example output

      NAME                                               READY   STATUS    RESTARTS   AGE
      numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
      numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
      numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
      secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m

   2. Get the logs for the secondary scheduler pod by running the following command:

      $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources

      Example output

      ...
      I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
      I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
      I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
      I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
      I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"

Procedure

1. Delete the currently running NUMAResourcesScheduler resource:

   1. Get the active NUMAResourcesScheduler by running the following command:

      $ oc get NUMAResourcesScheduler

      Example output

      NAME                     AGE
      numaresourcesscheduler   90m

   2. Delete the secondary scheduler resource by running the following command:

      $ oc delete NUMAResourcesScheduler numaresourcesscheduler

      Example output

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted

2. Save the following YAML in the file nro-scheduler-debug.yaml. This example changes the log level to Debug:

   apiVersion: nodetopology.openshift.io/v1alpha1
   kind: NUMAResourcesScheduler
   metadata:
     name: numaresourcesscheduler
   spec:
     imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"
     logLevel: Debug

3. Create the updated Debug logging NUMAResourcesScheduler resource by running the following command:

   $ oc create -f nro-scheduler-debug.yaml

   Example output

   numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created

Verification steps

1. Check that the NUMA-aware scheduler was successfully deployed:

   1. Run the following command to check that the CRD is created successfully:

      $ oc get crd | grep numaresourcesschedulers

      Example output

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z

   2. Check that the new custom scheduler is available by running the following command:

      $ oc get numaresourcesschedulers.nodetopology.openshift.io

      Example output

      NAME                     AGE
      numaresourcesscheduler   3h26m

2. Check that the logs for the scheduler shows the increased log level:

   1. Get the list of pods running in the openshift-numaresources namespace by running the following command:

      $ oc get pods -n openshift-numaresources

      Example output

      NAME                                               READY   STATUS    RESTARTS   AGE
      numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
      numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
      numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
      secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m

   2. Get the logs for the secondary scheduler pod by running the following command:

      $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources

      Example output

      ...
      I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
      I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
      I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
      I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
      I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"

Procedure

1. Get the daemonsets managed by the NUMA Resources Operator. Each daemonset has a corresponding nodeGroup in the NUMAResourcesOperator CR. Run the following command:

   $ oc get numaresourcesoperators.nodetopology.openshift.io numaresourcesoperator -o jsonpath="{.status.daemonsets[0]}"

   Example output

   {"name":"numaresourcesoperator-worker","namespace":"openshift-numaresources"}

2. Get the label for the daemonset of interest using the value for name from the previous step:

   $ oc get ds -n openshift-numaresources numaresourcesoperator-worker -o jsonpath="{.spec.selector.matchLabels}"

   Example output

   {"name":"resource-topology"}

3. Get the pods using the resource-topology label by running the following command:

   $ oc get pods -n openshift-numaresources -l name=resource-topology -o wide

   Example output

   NAME                                 READY   STATUS    RESTARTS   AGE    IP            NODE
   numaresourcesoperator-worker-5wm2k   2/2     Running   0          2d1h   10.135.0.64   compute-0.example.com
   numaresourcesoperator-worker-pb75c   2/2     Running   0          2d1h   10.132.2.33   compute-1.example.com

4. Examine the logs of the resource-topology-exporter container running on the worker pod that corresponds to the node you are troubleshooting. Run the following command:

   $ oc logs -n openshift-numaresources -c resource-topology-exporter numaresourcesoperator-worker-pb75c

   Example output

   I0221 13:38:18.334140       1 main.go:206] using sysinfo:
   reservedCpus: 0,1
   reservedMemory:
     "0": 1178599424
   I0221 13:38:18.334370       1 main.go:67] === System information ===
   I0221 13:38:18.334381       1 sysinfo.go:231] cpus: reserved "0-1"
   I0221 13:38:18.334493       1 sysinfo.go:237] cpus: online "0-103"
   I0221 13:38:18.546750       1 main.go:72]
   cpus: allocatable "2-103"
   hugepages-1Gi:
     numa cell 0 -> 6
     numa cell 1 -> 1
   hugepages-2Mi:
     numa cell 0 -> 64
     numa cell 1 -> 128
   memory:
     numa cell 0 -> 45758Mi
     numa cell 1 -> 48372Mi

Procedure

1. Compare the values for spec.machineConfigPoolSelector.matchLabels in kubeletconfig and metadata.labels in the MachineConfigPool (mcp) worker CR using the following commands:

   1. Check the kubeletconfig labels by running the following command:

      $ oc get kubeletconfig -o yaml

      Example output

      machineConfigPoolSelector:
        matchLabels:
          cnf-worker-tuning: enabled

   2. Check the mcp labels by running the following command:

      $ oc get mcp worker -o yaml

      Example output

      labels:
        machineconfiguration.openshift.io/mco-built-in: ""
        pools.operator.machineconfiguration.openshift.io/worker: ""

      The cnf-worker-tuning: enabled label is not present in the MachineConfigPool object.

2. Edit the MachineConfigPool CR to include the missing label, for example:

   $ oc edit mcp worker -o yaml

   Example output

   labels:
     machineconfiguration.openshift.io/mco-built-in: ""
     pools.operator.machineconfiguration.openshift.io/worker: ""
     cnf-worker-tuning: enabled

3. Apply the label changes and wait for the cluster to apply the updated configuration. Run the following command:

1. NetApp NFS supports dynamic PV provisioning when using the Trident plugin.

Procedure

1. Create a file called mount_namespace_config.yaml with the following YAML:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-kubens-master
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: kubens.service
   ---
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-kubens-worker
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: kubens.service

2. Apply the mount namespace MachineConfig CR by running the following command:

   $ oc apply -f mount_namespace_config.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-kubens-master created
   machineconfig.machineconfiguration.openshift.io/99-kubens-worker created

3. The MachineConfig CR can take up to 30 minutes to finish being applied in the cluster. You can check the status of the MachineConfig CR by running the following command:

   $ oc get mcp

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-03d4bc4befb0f4ed3566a2c8f7636751   False     True       False      3              0                   0                     0                      45m
   worker   rendered-worker-10577f6ab0117ed1825f8af2ac687ddf   False     True       False      3              1                   1

4. Wait for the MachineConfig CR to be applied successfully across all control plane and worker nodes after running the following command:

   $ oc wait --for=condition=Updated mcp --all --timeout=30m

   Example output

   machineconfigpool.machineconfiguration.openshift.io/master condition met
   machineconfigpool.machineconfiguration.openshift.io/worker condition met

1. Open a debug shell to the cluster host:

   $ oc debug node/<node_name>

2. Open a chroot session:

   sh-4.4# chroot /host

3. Check the systemd mount namespace:

   sh-4.4# readlink /proc/1/ns/mnt

   Example output

   mnt:[4026531953]

4. Check kubelet mount namespace:

   sh-4.4# readlink /proc/$(pgrep kubelet)/ns/mnt

   Example output

   mnt:[4026531840]

5. Check the CRI-O mount namespace:

   sh-4.4# readlink /proc/$(pgrep crio)/ns/mnt

   Example output

   mnt:[4026531840]

Procedure

1. Open a remote SSH shell to the cluster host. For example:

   $ ssh core@<node_name>

2. Run commands using the provided kubensenter script as the root user. To run a single command inside the Kubernetes namespace, provide the command and any arguments to the kubensenter script. For example, to run the findmnt command inside the Kubernetes namespace, run the following command:

   [core@control-plane-1 ~]$ sudo kubensenter findmnt

   Example output

   kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt
   TARGET                                SOURCE                 FSTYPE     OPTIONS
   /                                     /dev/sda4[/ostree/deploy/rhcos/deploy/32074f0e8e5ec453e56f5a8a7bc9347eaa4172349ceab9c22b709d9d71a3f4b0.0]
   |                                                            xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
                                         shm                    tmpfs
   ...

3. To start a new interactive shell inside the Kubernetes namespace, run the kubensenter script without any arguments:

   [core@control-plane-1 ~]$ sudo kubensenter

   Example output

   kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New with Dialog.
3. Specify a unique name for the new bare metal host.
4. Set the Boot MAC address.
5. Set the Baseboard Management Console (BMC) Address.
6. Enter the user credentials for the host’s baseboard management controller (BMC).
7. Select to power on the host after creation, and select Create.
8. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machine replicas in the cluster by selecting Edit Machine count from the Actions drop-down menu.

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New from YAML.

3. Copy and paste the below YAML, modifying the relevant fields with the details of your host:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: <bare_metal_host_name>
   spec:
     online: true
     bmc:
       address: <bmc_address>
       credentialsName: <secret_credentials_name>  1
       disableCertificateVerification: True 2
     bootMACAddress: <host_boot_mac_address>

   1

   credentialsName must reference a valid Secret CR. The baremetal-operator cannot manage the bare metal host without a valid Secret referenced in the credentialsName. For more information about secrets and how to create them, see Understanding secrets.

   2

   Setting disableCertificateVerification to true disables TLS host validation between the cluster and the baseboard management controller (BMC).

4. Select Create to save the YAML and create the new bare metal host.

5. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machines in the cluster by selecting Edit Machine count from the Actions drop-down menu.

   Note

   You can also manage the number of bare metal nodes using the oc scale command and the appropriate bare metal compute machine set.

Procedure

1. Annotate the compute machine set that you want to configure for automatic scaling by adding the metal3.io/autoscale-to-hosts annotation. Replace <machineset> with the name of the compute machine set.

   $ oc annotate machineset <machineset> -n openshift-machine-api 'metal3.io/autoscale-to-hosts=<any_value>'

   Wait for the new scaled machines to start.

Procedure

1. Annotate the compute machine set that you want to remove from the provisioner node by adding the baremetalhost.metal3.io/detached annotation.

   $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached'

   Wait for the new machines to start.

   Note

   When you use a BareMetalHost object to create a machine in the cluster and labels or selectors are subsequently changed on the BareMetalHost, the BareMetalHost object continues be counted against the MachineSet that the Machine object was created from.

2. In the provisioning use case, remove the annotation after the reboot is complete by using the following command:

   $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached-'

Procedure

1. Create a namespace for the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-bare-metal-events
        labels:
          name: openshift-bare-metal-events
          openshift.io/cluster-monitoring: "true"

   2. Create the Namespace CR:

      $ oc create -f bare-metal-events-namespace.yaml

2. Create an Operator group for the Bare Metal Event Relay Operator.

   1. Save the following YAML in the bare-metal-events-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: bare-metal-event-relay-group
        namespace: openshift-bare-metal-events
      spec:
        targetNamespaces:
        - openshift-bare-metal-events

   2. Create the OperatorGroup CR:

      $ oc create -f bare-metal-events-operatorgroup.yaml

3. Subscribe to the Bare Metal Event Relay.

   1. Save the following YAML in the bare-metal-events-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: bare-metal-event-relay-subscription
        namespace: openshift-bare-metal-events
      spec:
        channel: "stable"
        name: bare-metal-event-relay
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR:

      $ oc create -f bare-metal-events-sub.yaml

Procedure

1. Install the Bare Metal Event Relay using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Bare Metal Event Relay from the list of available Operators, and then click Install.
   3. On the Install Operator page, select or create a Namespace, select openshift-bare-metal-events, and then click Install.

1. Switch to the Operators → Installed Operators page.

2. Ensure that Bare Metal Event Relay is listed in the project with a Status of InstallSucceeded.

   Note

   During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Procedure

1. Confirm that the node hardware has the Redfish EventService enabled by running the following curl command:

   $ curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   {
      "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
      "@odata.id": "/redfish/v1/EventService",
      "@odata.type": "#EventService.v1_0_2.EventService",
      "Actions": {
         "#EventService.SubmitTestEvent": {
            "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
            "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
         }
      },
      "DeliveryRetryAttempts": 3,
      "DeliveryRetryIntervalSeconds": 30,
      "Description": "Event Service represents the properties for the service",
      "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
      "EventTypesForSubscription@odata.count": 5,
      "Id": "EventService",
      "Name": "Event Service",
      "ServiceEnabled": true,
      "Status": {
         "Health": "OK",
         "HealthRollup": "OK",
         "State": "Enabled"
      },
      "Subscriptions": {
         "@odata.id": "/redfish/v1/EventService/Subscriptions"
      }
   }

2. Get the Bare Metal Event Relay service route for the cluster by running the following command:

   $ oc get route -n openshift-bare-metal-events

   Example output

   NAME            HOST/PORT   PATH                                                                    SERVICES                 PORT   TERMINATION   WILDCARD
   hw-event-proxy              hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com   hw-event-proxy-service   9087   edge          None

3. Create a BMCEventSubscription resource to subscribe to the Redfish events:

   1. Save the following YAML in the bmc_sub.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: BMCEventSubscription
      metadata:
        name: sub-01
        namespace: openshift-machine-api
      spec:
         hostName: <hostname> 1
         destination: <proxy_service_url> 2
         context: ''

      1

      Specifies the name or UUID of the worker node where the Redfish events are generated.

      2

      Specifies the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   2. Create the BMCEventSubscription CR:

      $ oc create -f bmc_sub.yaml

4. Optional: To delete the BMC event subscription, run the following command:

   $ oc delete -f bmc_sub.yaml

5. Optional: To manually create a Redfish event subscription without creating a BMCEventSubscription CR, run the following curl command, specifying the BMC username and password.

   $ curl -i -k -X POST -H "Content-Type: application/json"  -d '{"Destination": "https://<proxy_service_url>", "Protocol" : "Redfish", "EventTypes": ["Alert"], "Context": "root"}' -u <bmc_username>:<password> 'https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions' –v

   where:

   proxy_service_url

   is the bare-metal event proxy service, for example, https://hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com/webhook.

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   HTTP/1.1 201 Created
   Server: AMI MegaRAC Redfish Service
   Location: /redfish/v1/EventService/Subscriptions/1
   Allow: GET, POST
   Access-Control-Allow-Origin: *
   Access-Control-Expose-Headers: X-Auth-Token
   Access-Control-Allow-Headers: X-Auth-Token
   Access-Control-Allow-Credentials: true
   Cache-Control: no-cache, must-revalidate
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>; rel=describedby
   Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>
   Link: </redfish/v1/EventService/Subscriptions>; path=
   ETag: "1651135676"
   Content-Type: application/json; charset=UTF-8
   OData-Version: 4.0
   Content-Length: 614
   Date: Thu, 28 Apr 2022 08:47:57 GMT

Procedure

1. Check the current subscriptions for the BMC by running the following curl command:

   $ curl --globoff -H "Content-Type: application/json" -k -X GET --user <bmc_username>:<password> https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

   Example output

   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed
   100 435 100 435 0 0 399 0 0:00:01 0:00:01 --:--:-- 399
   {
     "@odata.context": "/redfish/v1/$metadata#EventDestinationCollection.EventDestinationCollection",
     "@odata.etag": ""
     1651137375 "",
     "@odata.id": "/redfish/v1/EventService/Subscriptions",
     "@odata.type": "#EventDestinationCollection.EventDestinationCollection",
     "Description": "Collection for Event Subscriptions",
     "Members": [
     {
       "@odata.id": "/redfish/v1/EventService/Subscriptions/1"
     }],
     "Members@odata.count": 1,
     "Name": "Event Subscriptions Collection"
   }

   In this example, a single subscription is configured: /redfish/v1/EventService/Subscriptions/1.

2. Optional: To remove the /redfish/v1/EventService/Subscriptions/1 subscription with curl, run the following command, specifying the BMC username and password:

   $ curl --globoff -L -w "%{http_code} %{url_effective}\n" -k -u <bmc_username>:<password >-H "Content-Type: application/json" -d '{}' -X DELETE https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions/1

   where:

   bmc_ip_address

   is the IP address of the BMC where the Redfish events are generated.

Procedure

1. Create the HardwareEvent custom resource (CR):

   Note

   Multiple HardwareEvent resources are not permitted.

   1. Save the following YAML in the hw-event.yaml file:

      apiVersion: "event.redhat-cne.org/v1alpha1"
      kind: "HardwareEvent"
      metadata:
        name: "hardware-event"
      spec:
        nodeSelector:
          node-role.kubernetes.io/hw-event: "" 1
        logLevel: "debug" 2
        msgParserTimeout: "10" 3

      1

      Required. Use the nodeSelector field to target nodes with the specified label, for example, node-role.kubernetes.io/hw-event: "".

      Note

      In OpenShift Container Platform 4.12 or later, you do not need to set the spec.transportHost field in the HardwareEvent resource when you use HTTP transport for bare-metal events. Set transportHost only when you use AMQP transport for bare-metal events.

      2

      Optional. The default value is debug. Sets the log level in hw-event-proxy logs. The following log levels are available: fatal, error, warning, info, debug, trace.

      3

      Optional. Sets the timeout value in milliseconds for the Message Parser. If a message parsing request is not responded to within the timeout duration, the original hardware event message is passed to the cloud native event framework. The default value is 10.

   2. Apply the HardwareEvent CR in the cluster:

      $ oc create -f hardware-event.yaml

2. Create a BMC username and password Secret CR that enables the hardware events proxy to access the Redfish message registry for the bare-metal host.

   1. Save the following YAML in the hw-event-bmc-secret.yaml file:

      apiVersion: v1
      kind: Secret
      metadata:
        name: redfish-basic-auth
      type: Opaque
      stringData: 1
        username: <bmc_username>
        password: <bmc_password>
        # BMC host DNS or IP address
        hostaddr: <bmc_host_ip_address>

      1

      Enter plain text values for the various items under stringData.

   2. Create the Secret CR:

      $ oc create -f hw-event-bmc-secret.yaml

1. Optional: Add a node to the OpenShift Container Platform cluster. See Setting BIOS parameters for system tuning.
2. Add the label worker-rt to the worker nodes that require the real-time capability by using the oc command.

3. Create a new machine config pool for real-time nodes:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-rt
     labels:
       machineconfiguration.openshift.io/role: worker-rt
   spec:
     machineConfigSelector:
       matchExpressions:
         - {
              key: machineconfiguration.openshift.io/role,
              operator: In,
              values: [worker, worker-rt],
           }
     paused: false
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-rt: ""

   Note that a machine config pool worker-rt is created for group of nodes that have the label worker-rt.

4. Add the node to the proper machine config pool by using node role labels.

   Note

   You must decide which nodes are configured with real-time workloads. You could configure all of the nodes in the cluster, or a subset of the nodes. The Node Tuning Operator that expects all of the nodes are part of a dedicated machine config pool. If you use all of the nodes, you must point the Node Tuning Operator to the worker node role label. If you use a subset, you must group the nodes into a new machine config pool.

5. Create the PerformanceProfile with the proper set of housekeeping cores and realTimeKernel: enabled: true.

6. You must set machineConfigPoolSelector in PerformanceProfile:

     apiVersion: performance.openshift.io/v2
     kind: PerformanceProfile
     metadata:
      name: example-performanceprofile
     spec:
     ...
       realTimeKernel:
         enabled: true
       nodeSelector:
          node-role.kubernetes.io/worker-rt: ""
       machineConfigPoolSelector:
          machineconfiguration.openshift.io/role: worker-rt

7. Verify that a matching machine config pool exists with a label:

   $ oc describe mcp/worker-rt

   Example output

   Name:         worker-rt
   Namespace:
   Labels:       machineconfiguration.openshift.io/role=worker-rt

8. OpenShift Container Platform will start configuring the nodes, which might involve multiple reboots. Wait for the nodes to settle. This can take a long time depending on the specific hardware you use, but 20 minutes per node is expected.
9. Verify everything is working as expected.

Procedure

1. Create a pod with a QoS class of Guaranteed.
2. Optional: Disable CPU load balancing for DPDK.
3. Assign a proper node selector.

1. Create the pod:

   $ oc  apply -f qos-pod.yaml --namespace=qos-example

2. View detailed information about the pod:

   $ oc get pod qos-demo --namespace=qos-example --output=yaml

   Example output

   spec:
     containers:
       ...
   status:
     qosClass: Guaranteed

   Note

   If a container specifies its own memory limit, but does not specify a memory request, OpenShift Container Platform automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own CPU limit, but does not specify a CPU request, OpenShift Container Platform automatically assigns a CPU request that matches the limit.

1. Set the offline CPUs in the performance profile and save the contents of the YAML file:

   Example performance profile with offlined CPUs

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     additionalKernelArgs:
     - nmi_watchdog=0
     - audit=0
     - mce=off
     - processor.max_cstate=1
     - intel_idle.max_cstate=0
     - idle=poll
     cpu:
       isolated: "2-23,26-47"
       reserved: "0,1,24,25"
       offlined: "48-59" 1
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numa:
       topologyPolicy: single-numa-node
     realTimeKernel:
       enabled: true

   1

   Optional. You can list CPUs in the offlined field to take the specified CPUs offline.

2. Apply the updated profile by running the following command:

   $ oc apply -f my-performance-profile.yaml

Procedure

1. Generate a PerformanceProfile with per-pod-power-management set to true:

   $ podman run --entrypoint performance-profile-creator -v \
   /must-gather:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-operator:v4.12 \
   --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true \
   --split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node \
   --must-gather-dir-path /must-gather -power-consumption-mode=low-latency \ 1
   --per-pod-power-management=true > my-performance-profile.yaml

   1

   The power-consumption-mode must be default or low-latency when the per-pod-power-management is set to true.

   Example PerformanceProfile with perPodPowerManagement

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       [.....]
       workloadHints:
           realTime: true
           highPowerConsumption: false
           perPodPowerManagement: true

2. Set the default cpufreq governor as an additional kernel argument in the PerformanceProfile custom resource (CR):

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
        name: performance
   spec:
       ...
       additionalKernelArgs:
       - cpufreq.default_governor=schedutil 1

   1

   Using the schedutil governor is recommended, however, you can use other governors such as the ondemand or powersave governors.

3. Set the maximum CPU frequency in the TunedPerformancePatch CR:

   spec:
     profile:
     - data: |
         [sysfs]
         /sys/devices/system/cpu/intel_pstate/max_perf_pct = <x> 1

   1

   The max_perf_pct controls the maximum frequency the cpufreq driver is allowed to set as a percentage of the maximum supported cpu frequency. This value applies to all CPUs. You can check the maximum supported frequency in /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq. As a starting point, you can use a percentage that caps all CPUs at the All Cores Turbo frequency. The All Cores Turbo frequency is the frequency that all cores will run at when the cores are all fully occupied.

4. Add the desired annotations to your high priority workload pods. The annotations override the default settings.

   Example high priority workload annotation

   apiVersion: v1
   kind: Pod
   metadata:
     ...
     annotations:
       ...
       cpu-c-states.crio.io: "disable"
       cpu-freq-governor.crio.io: "<governor>"
       ...
     ...
   spec:
     ...
     runtimeClassName: performance-<profile_name>
     ...

5. Restart the pods.