6.3. 调度 NUMA 感知工作负载

流程

1. 创建 MachineConfigPool 自定义资源，为 worker 节点启用自定义 kubelet 配置：

   1. 将以下 YAML 保存到 nro-machineconfig.yaml 文件中：

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        labels:
          cnf-worker-tuning: enabled
          machineconfiguration.openshift.io/mco-built-in: ""
          pools.operator.machineconfiguration.openshift.io/worker: ""
        name: worker
      spec:
        machineConfigSelector:
          matchLabels:
            machineconfiguration.openshift.io/role: worker
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""

      Copy to Clipboard Toggle word wrap

   2. 运行以下命令来创建 MachineConfigPool CR：

      $ oc create -f nro-machineconfig.yaml

      Copy to Clipboard Toggle word wrap

2. 创建 NUMAResourcesOperator 自定义资源：

   1. 将以下 YAML 保存到 nrop.yaml 文件中：

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 

      1

      Copy to Clipboard Toggle word wrap

      1

      应该与相关 MachineConfigPool CR 中的 worker 节点匹配。

   2. 运行以下命令来创建 NUMAResourcesOperator CR：

      $ oc create -f nrop.yaml

      Copy to Clipboard Toggle word wrap

流程

1. 创建 KubeletConfig 自定义资源，为机器配置集配置 pod admittance 策略：

   1. 将以下 YAML 保存到 nro-kubeletconfig.yaml 文件中：

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: cnf-worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            cnf-worker-tuning: enabled
        kubeletConfig:
          cpuManagerPolicy: "static" 

      1

      
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1"
          memoryManagerPolicy: "Static" 

      2

      
          evictionHard:
            memory.available: "100Mi"
          kubeReserved:
            memory: "512Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 

      3

      
          topologyManagerScope: "pod"

      Copy to Clipboard Toggle word wrap

      1

      对于 cpuManagerPolicy，static 必须使用小写 s。

      2

      对于 memoryManagerPolicy，Static 必须使用大写 S。

      3

      topologyManagerPolicy 必须设置为 single-numa-node。

   2. 运行以下命令来创建 KubeletConfig 自定义资源 (CR)：

      $ oc create -f nro-kubeletconfig.yaml

      Copy to Clipboard Toggle word wrap

2. 创建 NUMAResourcesScheduler 自定义资源来部署 NUMA 感知自定义 pod 调度程序：

   1. 将以下 YAML 保存到 nro-scheduler.yaml 文件中：

      apiVersion: nodetopology.openshift.io/v1alpha1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.12"

      Copy to Clipboard Toggle word wrap

   2. 运行以下命令来创建 NUMAResourcesScheduler CR：

      $ oc create -f nro-scheduler.yaml

      Copy to Clipboard Toggle word wrap

流程

1. 运行以下命令，获取集群中部署的 NUMA 感知调度程序名称：

   $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'

   Copy to Clipboard Toggle word wrap

   输出示例

   topo-aware-scheduler

   Copy to Clipboard Toggle word wrap

2. 创建一个 Deployment CR，它使用名为 topo-aware-scheduler 的调度程序，例如：

   1. 将以下 YAML 保存到 nro-deployment.yaml 文件中：

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 

      1

      
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"

      Copy to Clipboard Toggle word wrap

      1

      schedulerName 必须与集群中部署的 NUMA 感知调度程序的名称匹配，如 topo-aware-scheduler。

   2. 运行以下命令来创建 Deployment CR：

      $ oc create -f nro-deployment.yaml

      Copy to Clipboard Toggle word wrap

验证

1. 验证部署是否成功：

   $ oc get pods -n openshift-numaresources

   Copy to Clipboard Toggle word wrap

   输出示例

   NAME                                                READY   STATUS    RESTARTS   AGE
   numa-deployment-1-56954b7b46-pfgw8                  2/2     Running   0          129m
   numaresources-controller-manager-7575848485-bns4s   1/1     Running   0          15h
   numaresourcesoperator-worker-dvj4n                  2/2     Running   0          18h
   numaresourcesoperator-worker-lcg4t                  2/2     Running   0          16h
   secondary-scheduler-56994cf6cf-7qf4q                1/1     Running   0          18h

   Copy to Clipboard Toggle word wrap

2. 运行以下命令，验证 topo-aware-scheduler 是否在调度部署的 pod：

   $ oc describe pod numa-deployment-1-56954b7b46-pfgw8 -n openshift-numaresources

   Copy to Clipboard Toggle word wrap

   输出示例

   Events:
     Type    Reason          Age   From                  Message
     ----    ------          ----  ----                  -------
     Normal  Scheduled       130m  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-56954b7b46-pfgw8 to compute-0.example.com

   Copy to Clipboard Toggle word wrap

   注意

   请求的资源超过可用于调度的部署将失败，并显示 MinimumReplicasUnavailable 错误。当所需资源可用时，部署会成功。Pod 会一直处于 Pending 状态，直到所需资源可用。

3. 验证是否为节点列出了预期的分配资源。运行以下命令:

   $ oc describe noderesourcetopologies.topology.node.k8s.io

   Copy to Clipboard Toggle word wrap

   输出示例

   ...
   
   Zones:
     Costs:
       Name:   node-0
       Value:  10
       Name:   node-1
       Value:  21
     Name:     node-0
     Resources:
       Allocatable:  39
       Available:    21 

   1

   
       Capacity:     40
       Name:         cpu
       Allocatable:  6442450944
       Available:    6442450944
       Capacity:     6442450944
       Name:         hugepages-1Gi
       Allocatable:  134217728
       Available:    134217728
       Capacity:     134217728
       Name:         hugepages-2Mi
       Allocatable:  262415904768
       Available:    262206189568
       Capacity:     270146007040
       Name:         memory
     Type:           Node

   Copy to Clipboard Toggle word wrap

   1

   由于已分配给有保证 pod 的资源，可用的容量会减少。

   通过保证 pod 使用的资源从 noderesourcetopologies.topology.node.k8s.io 中列出的可用节点资源中减去。

4. 对具有 Best-effort 或 Burstable 服务质量 (qosClass) 的pod 的资源分配不会反映在 noderesourcetopologies.topology.node.k8s.io 下的 NUMA 节点资源中。如果 pod 消耗的资源没有反映在节点资源计算中，请运行以下命令验证 pod 的 Guaranteed 具有 qosClass：

   $ oc get pod <pod_name> -n <pod_namespace> -o jsonpath="{ .status.qosClass }"

   Copy to Clipboard Toggle word wrap

   输出示例

   Guaranteed

   Copy to Clipboard Toggle word wrap