5.7. 配置 GPU Operator

流程

1. 运行以下命令，检查 Operator pod 是否正在运行以查看命名空间中的 pod：

   $ oc get pods -n nvidia-gpu-operator

   输出示例

   NAME                          READY   STATUS    RESTARTS   AGE
   gpu-operator-b4cb7d74-zxpwq   1/1     Running   0          32s

2. 创建 GPU 集群策略自定义资源文件，如下例所示：

   apiVersion: nvidia.com/v1
   kind: ClusterPolicy
   metadata:
     name: gpu-cluster-policy
   spec:
     vgpuDeviceManager:
       config:
         default: default
       enabled: true
     migManager:
       config:
         default: all-disabled
         name: default-mig-parted-config
       enabled: true
     operator:
       defaultRuntime: crio
       initContainer: {}
       runtimeClass: nvidia
       use_ocp_driver_toolkit: true
     dcgm:
       enabled: true
     gfd:
       enabled: true
     dcgmExporter:
       config:
         name: ''
       serviceMonitor:
         enabled: true
       enabled: true
     cdi:
       default: false
       enabled: false
     driver:
       licensingConfig:
         nlsEnabled: true
         configMapName: ''
       certConfig:
         name: ''
       rdma:
         enabled: false
       kernelModuleConfig:
         name: ''
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: false
           enable: false
           force: false
           timeoutSeconds: 300
         maxParallelUpgrades: 1
         maxUnavailable: 25%
         podDeletion:
           deleteEmptyDir: false
           force: false
           timeoutSeconds: 300
         waitForCompletion:
           timeoutSeconds: 0
       repoConfig:
         configMapName: ''
       virtualTopology:
         config: ''
       enabled: true
       useNvidiaDriverCRD: false
       useOpenKernelModules: true
     devicePlugin:
       config:
         name: ''
         default: ''
       mps:
         root: /run/nvidia/mps
       enabled: true
     gdrcopy:
       enabled: true
     kataManager:
       config:
         artifactsDir: /opt/nvidia-gpu-operator/artifacts/runtimeclasses
     mig:
       strategy: single
     sandboxDevicePlugin:
       enabled: true
     validator:
       plugin:
         env:
           - name: WITH_WORKLOAD
             value: 'false'
     nodeStatusExporter:
       enabled: true
     daemonsets:
       rollingUpdate:
         maxUnavailable: '1'
       updateStrategy: RollingUpdate
     sandboxWorkloads:
       defaultWorkload: container
       enabled: false
     gds:
       enabled: true
       image: nvidia-fs
       version: 2.20.5
       repository: nvcr.io/nvidia/cloud-native
     vgpuManager:
       enabled: false
     vfioManager:
       enabled: true
     toolkit:
       installDir: /usr/local/nvidia
       enabled: true

3. 当生成 GPU ClusterPolicy 自定义资源时，运行以下命令在集群中创建资源：

   $ oc create -f gpu-cluster-policy.yaml

   输出示例

   clusterpolicy.nvidia.com/gpu-cluster-policy created

4. 运行以下命令验证 Operator 是否已安装并正在运行：

   $ oc get pods -n nvidia-gpu-operator

   输出示例

   NAME                                                  READY   STATUS      RESTARTS   AGE
   gpu-feature-discovery-d5ngn                           1/1     Running     0          3m20s
   gpu-feature-discovery-z42rx                           1/1     Running     0          3m23s
   gpu-operator-6bb4d4b4c5-njh78                         1/1     Running     0          4m35s
   nvidia-container-toolkit-daemonset-bkh8l              1/1     Running     0          3m20s
   nvidia-container-toolkit-daemonset-c4hzm              1/1     Running     0          3m23s
   nvidia-cuda-validator-4blvg                           0/1     Completed   0          106s
   nvidia-cuda-validator-tw8sl                           0/1     Completed   0          112s
   nvidia-dcgm-exporter-rrw4g                            1/1     Running     0          3m20s
   nvidia-dcgm-exporter-xc78t                            1/1     Running     0          3m23s
   nvidia-dcgm-nvxpf                                     1/1     Running     0          3m20s
   nvidia-dcgm-snj4j                                     1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-fk2xz                  1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-wq87j                  1/1     Running     0          3m20s
   nvidia-driver-daemonset-416.94.202410211619-0-ngrjg   4/4     Running     0          3m58s
   nvidia-driver-daemonset-416.94.202410211619-0-tm4x6   4/4     Running     0          3m58s
   nvidia-node-status-exporter-jlzxh                     1/1     Running     0          3m57s
   nvidia-node-status-exporter-zjffs                     1/1     Running     0          3m57s
   nvidia-operator-validator-l49hx                       1/1     Running     0          3m20s
   nvidia-operator-validator-n44nn                       1/1     Running     0          3m23s

5. 可选：验证 pod 正在运行时，远程 shell 到 NVIDIA 驱动程序 daemonset pod，并确认已载入 NVIDIA 模块。具体来说，请确保载入 nvidia_peermem。

   $ oc rsh -n nvidia-gpu-operator $(oc -n nvidia-gpu-operator get pod -o name -l app.kubernetes.io/component=nvidia-driver)
   sh-4.4# lsmod|grep nvidia

   输出示例

   nvidia_fs             327680  0
   nvidia_peermem         24576  0
   nvidia_modeset       1507328  0
   video                  73728  1 nvidia_modeset
   nvidia_uvm           6889472  8
   nvidia               8810496  43 nvidia_uvm,nvidia_peermem,nvidia_fs,gdrdrv,nvidia_modeset
   ib_uverbs             217088  3 nvidia_peermem,rdma_ucm,mlx5_ib
   drm                   741376  5 drm_kms_helper,drm_shmem_helper,nvidia,mgag200

6. 可选：运行 nvidia-smi 工具来显示驱动程序和硬件的详情：

1. 当您仍然在驱动程序 pod 中时，使用 nvidia-smi 命令将 GPU 时钟设置为最大值：

   $ oc rsh -n nvidia-gpu-operator nvidia-driver-daemonset-416.94.202410172137-0-ndhzc
   sh-4.4# nvidia-smi -i 0 -lgc $(nvidia-smi -i 0 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   输出示例

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:61:00.0
   All done.

   sh-4.4# nvidia-smi -i 1 -lgc $(nvidia-smi -i 1 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   输出示例

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:E1:00.0
   All done.

2. 运行以下命令，从节点描述视角验证资源是否可用：

   $ oc describe node -l node-role.kubernetes.io/worker=| grep -E 'Capacity:|Allocatable:' -A9

   输出示例

   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596712Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445736Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   --
   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596672Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445696Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63