5.7. GPU 운영자 구성

프로세스

1. 다음 명령을 실행하여 네임스페이스 아래의 포드를 살펴보려면 Operator 포드가 실행 중인지 확인하세요.

   $ oc get pods -n nvidia-gpu-operator

   출력 예

   NAME                          READY   STATUS    RESTARTS   AGE
   gpu-operator-b4cb7d74-zxpwq   1/1     Running   0          32s

2. 다음 예와 유사한 GPU 클러스터 정책 사용자 정의 리소스 파일을 만듭니다.

   apiVersion: nvidia.com/v1
   kind: ClusterPolicy
   metadata:
     name: gpu-cluster-policy
   spec:
     vgpuDeviceManager:
       config:
         default: default
       enabled: true
     migManager:
       config:
         default: all-disabled
         name: default-mig-parted-config
       enabled: true
     operator:
       defaultRuntime: crio
       initContainer: {}
       runtimeClass: nvidia
       use_ocp_driver_toolkit: true
     dcgm:
       enabled: true
     gfd:
       enabled: true
     dcgmExporter:
       config:
         name: ''
       serviceMonitor:
         enabled: true
       enabled: true
     cdi:
       default: false
       enabled: false
     driver:
       licensingConfig:
         nlsEnabled: true
         configMapName: ''
       certConfig:
         name: ''
       rdma:
         enabled: false
       kernelModuleConfig:
         name: ''
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: false
           enable: false
           force: false
           timeoutSeconds: 300
         maxParallelUpgrades: 1
         maxUnavailable: 25%
         podDeletion:
           deleteEmptyDir: false
           force: false
           timeoutSeconds: 300
         waitForCompletion:
           timeoutSeconds: 0
       repoConfig:
         configMapName: ''
       virtualTopology:
         config: ''
       enabled: true
       useNvidiaDriverCRD: false
       useOpenKernelModules: true
     devicePlugin:
       config:
         name: ''
         default: ''
       mps:
         root: /run/nvidia/mps
       enabled: true
     gdrcopy:
       enabled: true
     kataManager:
       config:
         artifactsDir: /opt/nvidia-gpu-operator/artifacts/runtimeclasses
     mig:
       strategy: single
     sandboxDevicePlugin:
       enabled: true
     validator:
       plugin:
         env:
           - name: WITH_WORKLOAD
             value: 'false'
     nodeStatusExporter:
       enabled: true
     daemonsets:
       rollingUpdate:
         maxUnavailable: '1'
       updateStrategy: RollingUpdate
     sandboxWorkloads:
       defaultWorkload: container
       enabled: false
     gds:
       enabled: true
       image: nvidia-fs
       version: 2.20.5
       repository: nvcr.io/nvidia/cloud-native
     vgpuManager:
       enabled: false
     vfioManager:
       enabled: true
     toolkit:
       installDir: /usr/local/nvidia
       enabled: true

3. GPU ClusterPolicy 사용자 지정 리소스가 생성되면 다음 명령을 실행하여 클러스터에 리소스를 만듭니다.

   $ oc create -f gpu-cluster-policy.yaml

   출력 예

   clusterpolicy.nvidia.com/gpu-cluster-policy created

4. 다음 명령을 실행하여 Operator가 설치되고 실행 중인지 확인하세요.

   $ oc get pods -n nvidia-gpu-operator

   출력 예

   NAME                                                  READY   STATUS      RESTARTS   AGE
   gpu-feature-discovery-d5ngn                           1/1     Running     0          3m20s
   gpu-feature-discovery-z42rx                           1/1     Running     0          3m23s
   gpu-operator-6bb4d4b4c5-njh78                         1/1     Running     0          4m35s
   nvidia-container-toolkit-daemonset-bkh8l              1/1     Running     0          3m20s
   nvidia-container-toolkit-daemonset-c4hzm              1/1     Running     0          3m23s
   nvidia-cuda-validator-4blvg                           0/1     Completed   0          106s
   nvidia-cuda-validator-tw8sl                           0/1     Completed   0          112s
   nvidia-dcgm-exporter-rrw4g                            1/1     Running     0          3m20s
   nvidia-dcgm-exporter-xc78t                            1/1     Running     0          3m23s
   nvidia-dcgm-nvxpf                                     1/1     Running     0          3m20s
   nvidia-dcgm-snj4j                                     1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-fk2xz                  1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-wq87j                  1/1     Running     0          3m20s
   nvidia-driver-daemonset-416.94.202410211619-0-ngrjg   4/4     Running     0          3m58s
   nvidia-driver-daemonset-416.94.202410211619-0-tm4x6   4/4     Running     0          3m58s
   nvidia-node-status-exporter-jlzxh                     1/1     Running     0          3m57s
   nvidia-node-status-exporter-zjffs                     1/1     Running     0          3m57s
   nvidia-operator-validator-l49hx                       1/1     Running     0          3m20s
   nvidia-operator-validator-n44nn                       1/1     Running     0          3m23s

5. 선택 사항: 포드가 실행 중인지 확인한 후 NVIDIA 드라이버 데몬셋 포드에 원격 셸을 실행하여 NVIDIA 모듈이 로드되었는지 확인합니다. 특히, nvidia_peermem 이 로드되었는지 확인하세요.

   $ oc rsh -n nvidia-gpu-operator $(oc -n nvidia-gpu-operator get pod -o name -l app.kubernetes.io/component=nvidia-driver)
   sh-4.4# lsmod|grep nvidia

   출력 예

   nvidia_fs             327680  0
   nvidia_peermem         24576  0
   nvidia_modeset       1507328  0
   video                  73728  1 nvidia_modeset
   nvidia_uvm           6889472  8
   nvidia               8810496  43 nvidia_uvm,nvidia_peermem,nvidia_fs,gdrdrv,nvidia_modeset
   ib_uverbs             217088  3 nvidia_peermem,rdma_ucm,mlx5_ib
   drm                   741376  5 drm_kms_helper,drm_shmem_helper,nvidia,mgag200

6. 선택 사항: nvidia-smi 유틸리티를 실행하여 드라이버 및 하드웨어에 대한 세부 정보를 표시합니다.

1. 드라이버 포드에 있는 동안 nvidia-smi 명령을 사용하여 GPU 클럭을 최대로 설정하세요.

   $ oc rsh -n nvidia-gpu-operator nvidia-driver-daemonset-416.94.202410172137-0-ndhzc
   sh-4.4# nvidia-smi -i 0 -lgc $(nvidia-smi -i 0 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   출력 예

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:61:00.0
   All done.

   sh-4.4# nvidia-smi -i 1 -lgc $(nvidia-smi -i 1 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   출력 예

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:E1:00.0
   All done.

2. 다음 명령을 실행하여 노드 설명 관점에서 리소스를 사용할 수 있는지 확인하세요.

   $ oc describe node -l node-role.kubernetes.io/worker=| grep -E 'Capacity:|Allocatable:' -A9

   출력 예

   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596712Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445736Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   --
   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596672Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445696Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63