하드웨어 가속기

다음 단계

1. Node Feature Discovery Operator 를 설치합니다.
2. Kernel Module Management Operator 를 설치합니다.
3. AMD GPU Operator 를 설치하고 구성합니다.

프로세스

1. ROCmInfo를 테스트하는 YAML 파일을 생성합니다.

   $ cat << EOF > rocminfo.yaml
   
   apiVersion: v1
   kind: Pod
   metadata:
    name: rocminfo
   spec:
    containers:
    - image: docker.io/rocm/pytorch:latest
      name: rocminfo
      command: ["/bin/sh","-c"]
      args: ["rocminfo"]
      resources:
       limits:
         amd.com/gpu: 1
       requests:
         amd.com/gpu: 1
    restartPolicy: Never
   EOF

   Copy to Clipboard Toggle word wrap

2. rocminfo Pod를 생성합니다.

   $ oc create -f rocminfo.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   apiVersion: v1
   pod/rocminfo created

   Copy to Clipboard Toggle word wrap

3. MI210 GPU 1개로 rocmnfo 로그를 확인합니다.

   $ oc logs rocminfo | grep -A5 "Agent"

   Copy to Clipboard Toggle word wrap

   출력 예

   HSA Agents
   ==========
   *******
   Agent 1
   *******
     Name:                    Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
     Uuid:                    CPU-XX
     Marketing Name:          Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
     Vendor Name:             CPU
   --
   Agent 2
   *******
     Name:                    Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
     Uuid:                    CPU-XX
     Marketing Name:          Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
     Vendor Name:             CPU
   --
   Agent 3
   *******
     Name:                    gfx90a
     Uuid:                    GPU-024b776f768a638b
     Marketing Name:          AMD Instinct MI210
     Vendor Name:             AMD

   Copy to Clipboard Toggle word wrap

4. Pod를 삭제합니다.

   $ oc delete -f rocminfo.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   pod "rocminfo" deleted

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 실행하여 다음 머신 구성 파일을 생성합니다.

   $ cat <<EOF > 99-machine-config-blacklist-irdma.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-blacklist-irdma
   spec:
     kernelArguments:
       - "module_blacklist=irdma"

   Copy to Clipboard Toggle word wrap

2. 클러스터에 머신 구성을 생성하고 다음 명령을 실행하여 노드가 재부팅될 때까지 기다립니다.

   $ oc create -f 99-machine-config-blacklist-irdma.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   machineconfig.machineconfiguration.openshift.io/99-worker-blacklist-irdma created

   Copy to Clipboard Toggle word wrap

3. 다음 명령을 실행하여 모듈이 로드되지 않은 각 노드의 디버그 Pod에서 확인합니다.

   $ oc debug node/nvd-srv-32.nvidia.eng.rdu2.dc.redhat.com
   Starting pod/nvd-srv-32nvidiaengrdu2dcredhatcom-debug-btfj2 ...
   To use host binaries, run `chroot /host`
   Pod IP: 10.6.135.11
   If you don't see a command prompt, try pressing enter.
   sh-5.1# chroot /host
   sh-5.1# lsmod|grep irdma
   sh-5.1#

   Copy to Clipboard Toggle word wrap

프로세스

1. 노드의 작업자 노드에서 파일로 MAC 주소 이름을 수집하고 유지해야 하는 인터페이스의 이름을 제공합니다. 이 예에서는 70-persistent-net.rules 파일을 사용하고 세부 정보를 캐시합니다.

   $ cat <<EOF > 70-persistent-net.rules
   SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="b8:3f:d2:3b:51:28",ATTR{type}=="1",NAME="ibs2f0"
   SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="b8:3f:d2:3b:51:29",ATTR{type}=="1",NAME="ens8f0np0"
   SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="b8:3f:d2:f0:36:d0",ATTR{type}=="1",NAME="ibs2f0"
   SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="b8:3f:d2:f0:36:d1",ATTR{type}=="1",NAME="ens8f0np0"
   EOF

   Copy to Clipboard Toggle word wrap

2. 파일을 줄 바꿈 없이 base64 문자열로 변환하고 출력을 변수 PERSIST 로 설정합니다.

   $ PERSIST=`cat 70-persistent-net.rules| base64 -w 0`

   Copy to Clipboard Toggle word wrap

   $ echo $PERSIST

   Copy to Clipboard Toggle word wrap

   U1VCU1lTVEVNPT0ibmV0IixBQ1RJT049PSJhZGQiLEFUVFJ7YWRkcmVzc309PSJiODozZjpkMjozYjo1MToyOCIsQVRUUnt0eXBlfT09IjEiLE5BTUU9ImliczJmMCIKU1VCU1lTVEVNPT0ibmV0IixBQ1RJT049PSJhZGQiLEFUVFJ7YWRkcmVzc309PSJiODozZjpkMjozYjo1MToyOSIsQVRUUnt0eXBlfT09IjEiLE5BTUU9ImVuczhmMG5wMCIKU1VCU1lTVEVNPT0ibmV0IixBQ1RJT049PSJhZGQiLEFUVFJ7YWRkcmVzc309PSJiODozZjpkMjpmMDozNjpkMCIsQVRUUnt0eXBlfT09IjEiLE5BTUU9ImliczJmMCIKU1VCU1lTVEVNPT0ibmV0IixBQ1RJT049PSJhZGQiLEFUVFJ7YWRkcmVzc309PSJiODozZjpkMjpmMDozNjpkMSIsQVRUUnt0eXBlfT09IjEiLE5BTUU9ImVuczhmMG5wMCIK

   Copy to Clipboard Toggle word wrap

3. 다음 명령을 실행하여 머신 구성을 생성하고 사용자 정의 리소스 파일에서 base64 인코딩을 설정합니다.

   $ cat <<EOF > 99-machine-config-udev-network.yaml

   Copy to Clipboard Toggle word wrap

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 99-machine-config-udev-network
   spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          files:
          - contents:
              source: data:text/plain;base64,$PERSIST
            filesystem: root
            mode: 420
            path: /etc/udev/rules.d/70-persistent-net.rules

   Copy to Clipboard Toggle word wrap

4. 다음 명령을 실행하여 클러스터에 머신 구성을 생성합니다. 명령을 실행하면 예상되는 출력에 machineconfig.machineconfiguration.openshift.io/99-machine-config-udev-network가 생성된 것으로 표시됩니다.

   $ oc create -f 99-machine-config-udev-network.yaml

   Copy to Clipboard Toggle word wrap

5. get mcp 명령을 사용하여 머신 구성 상태를 확인합니다.

   $ oc get mcp

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-9adfe851c2c14d9598eea5ec3df6c187   True      False      False      1              1                   1                     0                      6h21m
   worker   rendered-worker-4568f1b174066b4b1a4de794cf538fee   False     True       False      2              0                   0                     0                      6h21m

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 실행하여 openshift-nfd 네임스페이스의 Pod를 보고 Operator가 설치되어 실행 중인지 확인합니다.

   $ oc get pods -n openshift-nfd

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                      READY   STATUS    RESTARTS   AGE
   nfd-controller-manager-8698c88cdd-t8gbc   2/2     Running   0          2m

   Copy to Clipboard Toggle word wrap

2. NFD 컨트롤러가 실행되면 NodeFeatureDiscovery 인스턴스를 생성하여 클러스터에 추가합니다.

   NFD Operator의 ClusterServiceVersion 사양은 Operator 페이로드에 포함된 NFD 피연산자 이미지를 포함하여 기본값을 제공합니다. 다음 명령을 실행하여 값을 검색합니다.

   $ NFD_OPERAND_IMAGE=`echo $(oc get csv -n openshift-nfd -o json | jq -r '.items[0].metadata.annotations["alm-examples"]') | jq -r '.[] | select(.kind == "NodeFeatureDiscovery") | .spec.operand.image'`

   Copy to Clipboard Toggle word wrap

3. 선택 사항: 기본 deviceClassWhiteList 필드에 항목을 추가하여 NVIDIA BlueField DPU와 같은 더 많은 네트워크 어댑터를 지원합니다.

   apiVersion: nfd.openshift.io/v1
   kind: NodeFeatureDiscovery
   metadata:
     name: nfd-instance
     namespace: openshift-nfd
   spec:
     instance: ''
     operand:
       image: '${NFD_OPERAND_IMAGE}'
       servicePort: 12000
     prunerOnDelete: false
     topologyUpdater: false
     workerConfig:
       configData: |
         core:
           sleepInterval: 60s
         sources:
           pci:
             deviceClassWhitelist:
               - "02"
               - "03"
               - "0200"
               - "0207"
               - "12"
             deviceLabelFields:
               - "vendor"

   Copy to Clipboard Toggle word wrap

4. 다음 명령을 실행하여 'NodeFeatureDiscovery' 인스턴스를 생성합니다.

   $ oc create -f nfd-instance.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   nodefeaturediscovery.nfd.openshift.io/nfd-instance created

   Copy to Clipboard Toggle word wrap

5. 다음 명령을 실행하여 openshift-nfd 네임스페이스의 Pod를 보고 인스턴스가 실행 중인지 확인합니다.

   $ oc get pods -n openshift-nfd

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                    READY   STATUS    RESTARTS   AGE
   nfd-controller-manager-7cb6d656-jcnqb   2/2     Running   0          4m
   nfd-gc-7576d64889-s28k9                 1/1     Running   0          21s
   nfd-master-b7bcf5cfd-qnrmz              1/1     Running   0          21s
   nfd-worker-96pfh                        1/1     Running   0          21s
   nfd-worker-b2gkg                        1/1     Running   0          21s
   nfd-worker-bd9bk                        1/1     Running   0          21s
   nfd-worker-cswf4                        1/1     Running   0          21s
   nfd-worker-kp6gg                        1/1     Running   0          21s

   Copy to Clipboard Toggle word wrap

6. 짧은 시간을 기다린 다음 NFD가 노드에 라벨을 추가했는지 확인합니다. NFD 라벨에는 feature.node.kubernetes.io 가 접두사로 지정되어 있으므로 쉽게 필터링할 수 있습니다.

   $ oc get node -o json | jq '.items[0].metadata.labels | with_entries(select(.key | startswith("feature.node.kubernetes.io")))'
   {
     "feature.node.kubernetes.io/cpu-cpuid.ADX": "true",
     "feature.node.kubernetes.io/cpu-cpuid.AESNI": "true",
     "feature.node.kubernetes.io/cpu-cpuid.AVX": "true",
     "feature.node.kubernetes.io/cpu-cpuid.AVX2": "true",
     "feature.node.kubernetes.io/cpu-cpuid.CETSS": "true",
     "feature.node.kubernetes.io/cpu-cpuid.CLZERO": "true",
     "feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8": "true",
     "feature.node.kubernetes.io/cpu-cpuid.CPBOOST": "true",
     "feature.node.kubernetes.io/cpu-cpuid.EFER_LMSLE_UNS": "true",
     "feature.node.kubernetes.io/cpu-cpuid.FMA3": "true",
     "feature.node.kubernetes.io/cpu-cpuid.FP256": "true",
     "feature.node.kubernetes.io/cpu-cpuid.FSRM": "true",
     "feature.node.kubernetes.io/cpu-cpuid.FXSR": "true",
     "feature.node.kubernetes.io/cpu-cpuid.FXSROPT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBPB": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBRS": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBRS_PREFERRED": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBRS_PROVIDES_SMP": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBS": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSBRNTRGT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSFETCHSAM": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSFFV": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSOPCNT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSOPCNTEXT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSOPSAM": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSRDWROPCNT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBSRIPINVALIDCHK": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBS_FETCH_CTLX": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBS_OPFUSE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.IBS_PREVENTHOST": "true",
     "feature.node.kubernetes.io/cpu-cpuid.INT_WBINVD": "true",
     "feature.node.kubernetes.io/cpu-cpuid.INVLPGB": "true",
     "feature.node.kubernetes.io/cpu-cpuid.LAHF": "true",
     "feature.node.kubernetes.io/cpu-cpuid.LBRVIRT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MCAOVERFLOW": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MCOMMIT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MOVBE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MOVU": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MSRIRC": "true",
     "feature.node.kubernetes.io/cpu-cpuid.MSR_PAGEFLUSH": "true",
     "feature.node.kubernetes.io/cpu-cpuid.NRIPS": "true",
     "feature.node.kubernetes.io/cpu-cpuid.OSXSAVE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.PPIN": "true",
     "feature.node.kubernetes.io/cpu-cpuid.PSFD": "true",
     "feature.node.kubernetes.io/cpu-cpuid.RDPRU": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_64BIT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_ALTERNATIVE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_DEBUGSWAP": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_ES": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_RESTRICTED": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SEV_SNP": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SHA": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SME": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SME_COHERENT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SSE4A": "true",
     "feature.node.kubernetes.io/cpu-cpuid.STIBP": "true",
     "feature.node.kubernetes.io/cpu-cpuid.STIBP_ALWAYSON": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SUCCOR": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVM": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVMDA": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVMFBASID": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVML": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVMNP": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVMPF": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SVMPFT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SYSCALL": "true",
     "feature.node.kubernetes.io/cpu-cpuid.SYSEE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.TLB_FLUSH_NESTED": "true",
     "feature.node.kubernetes.io/cpu-cpuid.TOPEXT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.TSCRATEMSR": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VAES": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VMCBCLEAN": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VMPL": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VMSA_REGPROT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ": "true",
     "feature.node.kubernetes.io/cpu-cpuid.VTE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.WBNOINVD": "true",
     "feature.node.kubernetes.io/cpu-cpuid.X87": "true",
     "feature.node.kubernetes.io/cpu-cpuid.XGETBV1": "true",
     "feature.node.kubernetes.io/cpu-cpuid.XSAVE": "true",
     "feature.node.kubernetes.io/cpu-cpuid.XSAVEC": "true",
     "feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT": "true",
     "feature.node.kubernetes.io/cpu-cpuid.XSAVES": "true",
     "feature.node.kubernetes.io/cpu-hardware_multithreading": "false",
     "feature.node.kubernetes.io/cpu-model.family": "25",
     "feature.node.kubernetes.io/cpu-model.id": "1",
     "feature.node.kubernetes.io/cpu-model.vendor_id": "AMD",
     "feature.node.kubernetes.io/kernel-config.NO_HZ": "true",
     "feature.node.kubernetes.io/kernel-config.NO_HZ_FULL": "true",
     "feature.node.kubernetes.io/kernel-selinux.enabled": "true",
     "feature.node.kubernetes.io/kernel-version.full": "5.14.0-427.35.1.el9_4.x86_64",
     "feature.node.kubernetes.io/kernel-version.major": "5",
     "feature.node.kubernetes.io/kernel-version.minor": "14",
     "feature.node.kubernetes.io/kernel-version.revision": "0",
     "feature.node.kubernetes.io/memory-numa": "true",
     "feature.node.kubernetes.io/network-sriov.capable": "true",
     "feature.node.kubernetes.io/pci-102b.present": "true",
     "feature.node.kubernetes.io/pci-10de.present": "true",
     "feature.node.kubernetes.io/pci-10de.sriov.capable": "true",
     "feature.node.kubernetes.io/pci-15b3.present": "true",
     "feature.node.kubernetes.io/pci-15b3.sriov.capable": "true",
     "feature.node.kubernetes.io/rdma.available": "true",
     "feature.node.kubernetes.io/rdma.capable": "true",
     "feature.node.kubernetes.io/storage-nonrotationaldisk": "true",
     "feature.node.kubernetes.io/system-os_release.ID": "rhcos",
     "feature.node.kubernetes.io/system-os_release.OPENSHIFT_VERSION": "4.17",
     "feature.node.kubernetes.io/system-os_release.OSTREE_VERSION": "417.94.202409121747-0",
     "feature.node.kubernetes.io/system-os_release.RHEL_VERSION": "9.4",
     "feature.node.kubernetes.io/system-os_release.VERSION_ID": "4.17",
     "feature.node.kubernetes.io/system-os_release.VERSION_ID.major": "4",
     "feature.node.kubernetes.io/system-os_release.VERSION_ID.minor": "17"
   }

   Copy to Clipboard Toggle word wrap

7. 검색된 네트워크 장치가 있는지 확인합니다.

   $ oc describe node | grep -E 'Roles|pci' | grep pci-15b3
                       feature.node.kubernetes.io/pci-15b3.present=true
                       feature.node.kubernetes.io/pci-15b3.sriov.capable=true
                       feature.node.kubernetes.io/pci-15b3.present=true
                       feature.node.kubernetes.io/pci-15b3.sriov.capable=true

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 실행하여 openshift-sriov-network-operator 네임스페이스에서 Pod를 확인하여 Operator가 설치 및 실행 중인지 확인합니다.

   $ oc get pods -n openshift-sriov-network-operator

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                      READY   STATUS    RESTARTS   AGE
   sriov-network-operator-7cb6c49868-89486   1/1     Running   0          22s

   Copy to Clipboard Toggle word wrap

2. 기본 SriovOperatorConfig CR이 MLNX_OFED 컨테이너에서 작동하도록 하려면 이 명령을 실행하여 다음 값을 업데이트합니다.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovOperatorConfig
   metadata:
     name: default
     namespace: openshift-sriov-network-operator
   spec:
     enableInjector: true
     enableOperatorWebhook: true
     logLevel: 2

   Copy to Clipboard Toggle word wrap

3. 다음 명령을 실행하여 클러스터에 리소스를 생성합니다.

   $ oc create -f sriov-operator-config.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   sriovoperatorconfig.sriovnetwork.openshift.io/default created

   Copy to Clipboard Toggle word wrap

4. 다음 명령을 실행하여 MOFED 컨테이너가 작업할 수 있도록 sriov-operator를 패치합니다.

   $ oc patch sriovoperatorconfig default   --type=merge -n openshift-sriov-network-operator   --patch '{ "spec": { "configDaemonNodeSelector": { "network.nvidia.com/operator.mofed.wait": "false", "node-role.kubernetes.io/worker": "", "feature.node.kubernetes.io/pci-15b3.sriov.capable": "true" } } }'

   Copy to Clipboard Toggle word wrap

   출력 예

   sriovoperatorconfig.sriovnetwork.openshift.io/default patched

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 실행하여 컨트롤러가 nvidia-network-operator 네임스페이스에서 실행되고 있는지 확인하여 네트워크 Operator가 설치 및 실행 중인지 확인합니다.

   $ oc get pods -n nvidia-network-operator

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                                          READY   STATUS             RESTARTS        AGE
   nvidia-network-operator-controller-manager-6f7d6956cd-fw5wg   1/1     Running            0                5m

   Copy to Clipboard Toggle word wrap

2. Operator가 실행되면 NicClusterPolicy 사용자 정의 리소스 파일을 생성합니다. 선택한 장치는 시스템 구성에 따라 다릅니다. 이 예에서 Infiniband 인터페이스 ibs2f0 은 하드 코딩되며 공유 NVIDIA GPUDirect RDMA 장치로 사용됩니다.

   apiVersion: mellanox.com/v1alpha1
   kind: NicClusterPolicy
   metadata:
     name: nic-cluster-policy
   spec:
     nicFeatureDiscovery:
       image: nic-feature-discovery
       repository: ghcr.io/mellanox
       version: v0.0.1
     docaTelemetryService:
       image: doca_telemetry
       repository: nvcr.io/nvidia/doca
       version: 1.16.5-doca2.6.0-host
     rdmaSharedDevicePlugin:
       config: |
         {
           "configList": [
             {
               "resourceName": "rdma_shared_device_ib",
               "rdmaHcaMax": 63,
               "selectors": {
                 "ifNames": ["ibs2f0"]
               }
             },
             {
               "resourceName": "rdma_shared_device_eth",
               "rdmaHcaMax": 63,
               "selectors": {
                 "ifNames": ["ens8f0np0"]
               }
             }
           ]
         }
       image: k8s-rdma-shared-dev-plugin
       repository: ghcr.io/mellanox
       version: v1.5.1
     secondaryNetwork:
       ipoib:
         image: ipoib-cni
         repository: ghcr.io/mellanox
         version: v1.2.0
     nvIpam:
       enableWebhook: false
       image: nvidia-k8s-ipam
       repository: ghcr.io/mellanox
       version: v0.2.0
     ofedDriver:
       readinessProbe:
         initialDelaySeconds: 10
         periodSeconds: 30
       forcePrecompiled: false
       terminationGracePeriodSeconds: 300
       livenessProbe:
         initialDelaySeconds: 30
         periodSeconds: 30
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: true
           enable: true
           force: true
           timeoutSeconds: 300
           podSelector: ''
         maxParallelUpgrades: 1
         safeLoad: false
         waitForCompletion:
           timeoutSeconds: 0
       startupProbe:
         initialDelaySeconds: 10
         periodSeconds: 20
       image: doca-driver
       repository: nvcr.io/nvidia/mellanox
       version: 24.10-0.7.0.0-0
       env:
       - name: UNLOAD_STORAGE_MODULES
         value: "true"
       - name: RESTORE_DRIVER_ON_POD_TERMINATION
         value: "true"
       - name: CREATE_IFNAMES_UDEV
         value: "true"

   Copy to Clipboard Toggle word wrap

3. 다음 명령을 실행하여 클러스터에 NicClusterPolicy 사용자 정의 리소스를 생성합니다.

   $ oc create -f network-sharedrdma-nic-cluster-policy.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   nicclusterpolicy.mellanox.com/nic-cluster-policy created

   Copy to Clipboard Toggle word wrap

4. DOCA/MOFED 컨테이너에서 다음 명령을 실행하여 NicClusterPolicy 를 검증합니다.

   $ oc get pods -n nvidia-network-operator

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                                          READY   STATUS    RESTARTS   AGE
   doca-telemetry-service-hwj65                                  1/1     Running   2          160m
   kube-ipoib-cni-ds-fsn8g                                       1/1     Running   2          160m
   mofed-rhcos4.16-9b5ddf4c6-ds-ct2h5                            2/2     Running   4          160m
   nic-feature-discovery-ds-dtksz                                1/1     Running   2          160m
   nv-ipam-controller-854585f594-c5jpp                           1/1     Running   2          160m
   nv-ipam-controller-854585f594-xrnp5                           1/1     Running   2          160m
   nv-ipam-node-xqttl                                            1/1     Running   2          160m
   nvidia-network-operator-controller-manager-5798b564cd-5cq99   1/1     Running   2         5d23h
   rdma-shared-dp-ds-p9vvg                                       1/1     Running   0          85m

   Copy to Clipboard Toggle word wrap

5. 다음 명령을 실행하여 mofed 컨테이너에 rsh 로 접속하여 상태를 확인하세요.

   $ MOFED_POD=$(oc get pods -n nvidia-network-operator -o name | grep mofed)
   $ oc rsh -n nvidia-network-operator -c mofed-container ${MOFED_POD}
   sh-5.1# ofed_info -s

   Copy to Clipboard Toggle word wrap

   출력 예

   OFED-internal-24.07-0.6.1:

   Copy to Clipboard Toggle word wrap

   sh-5.1# ibdev2netdev -v

   Copy to Clipboard Toggle word wrap

   출력 예

   0000:0d:00.0 mlx5_0 (MT41692 - 900-9D3B4-00EN-EA0) BlueField-3 E-series SuperNIC 400GbE/NDR single port QSFP112, PCIe Gen5.0 x16 FHHL, Crypto Enabled, 16GB DDR5, BMC, Tall Bracket                                                       fw 32.42.1000 port 1 (ACTIVE) ==> ibs2f0 (Up)
   0000:a0:00.0 mlx5_1 (MT41692 - 900-9D3B4-00EN-EA0) BlueField-3 E-series SuperNIC 400GbE/NDR single port QSFP112, PCIe Gen5.0 x16 FHHL, Crypto Enabled, 16GB DDR5, BMC, Tall Bracket                                                       fw 32.42.1000 port 1 (ACTIVE) ==> ens8f0np0 (Up)

   Copy to Clipboard Toggle word wrap

6. IPoIBNetwork 사용자 지정 리소스 파일을 생성합니다.

   apiVersion: mellanox.com/v1alpha1
   kind: IPoIBNetwork
   metadata:
     name: example-ipoibnetwork
   spec:
     ipam: |
       {
         "type": "whereabouts",
         "range": "192.168.6.225/28",
         "exclude": [
          "192.168.6.229/30",
          "192.168.6.236/32"
         ]
       }
     master: ibs2f0
     networkNamespace: default

   Copy to Clipboard Toggle word wrap

7. 다음 명령을 실행하여 클러스터에서 IPoIBNetwork 리소스를 생성합니다.

   $ oc create -f ipoib-network.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   ipoibnetwork.mellanox.com/example-ipoibnetwork created

   Copy to Clipboard Toggle word wrap

8. 다른 인터페이스에 대한 MacvlanNetwork 사용자 지정 리소스 파일을 만듭니다.

   apiVersion: mellanox.com/v1alpha1
   kind: MacvlanNetwork
   metadata:
     name: rdmashared-net
   spec:
     networkNamespace: default
     master: ens8f0np0
     mode: bridge
     mtu: 1500
     ipam: '{"type": "whereabouts", "range": "192.168.2.0/24", "gateway": "192.168.2.1"}'

   Copy to Clipboard Toggle word wrap

9. 다음 명령을 실행하여 클러스터에 리소스를 생성합니다.

   $ oc create -f macvlan-network.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   macvlannetwork.mellanox.com/rdmashared-net created

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 실행하여 Operator Pod가 네임스페이스 아래의 Pod를 확인하도록 실행 중인지 확인합니다.

   $ oc get pods -n nvidia-gpu-operator

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                          READY   STATUS    RESTARTS   AGE
   gpu-operator-b4cb7d74-zxpwq   1/1     Running   0          32s

   Copy to Clipboard Toggle word wrap

2. 다음 예와 유사한 GPU 클러스터 정책 사용자 정의 리소스 파일을 생성합니다.

   apiVersion: nvidia.com/v1
   kind: ClusterPolicy
   metadata:
     name: gpu-cluster-policy
   spec:
     vgpuDeviceManager:
       config:
         default: default
       enabled: true
     migManager:
       config:
         default: all-disabled
         name: default-mig-parted-config
       enabled: true
     operator:
       defaultRuntime: crio
       initContainer: {}
       runtimeClass: nvidia
       use_ocp_driver_toolkit: true
     dcgm:
       enabled: true
     gfd:
       enabled: true
     dcgmExporter:
       config:
         name: ''
       serviceMonitor:
         enabled: true
       enabled: true
     cdi:
       default: false
       enabled: false
     driver:
       licensingConfig:
         nlsEnabled: true
         configMapName: ''
       certConfig:
         name: ''
       rdma:
         enabled: false
       kernelModuleConfig:
         name: ''
       upgradePolicy:
         autoUpgrade: true
         drain:
           deleteEmptyDir: false
           enable: false
           force: false
           timeoutSeconds: 300
         maxParallelUpgrades: 1
         maxUnavailable: 25%
         podDeletion:
           deleteEmptyDir: false
           force: false
           timeoutSeconds: 300
         waitForCompletion:
           timeoutSeconds: 0
       repoConfig:
         configMapName: ''
       virtualTopology:
         config: ''
       enabled: true
       useNvidiaDriverCRD: false
       useOpenKernelModules: true
     devicePlugin:
       config:
         name: ''
         default: ''
       mps:
         root: /run/nvidia/mps
       enabled: true
     gdrcopy:
       enabled: true
     kataManager:
       config:
         artifactsDir: /opt/nvidia-gpu-operator/artifacts/runtimeclasses
     mig:
       strategy: single
     sandboxDevicePlugin:
       enabled: true
     validator:
       plugin:
         env:
           - name: WITH_WORKLOAD
             value: 'false'
     nodeStatusExporter:
       enabled: true
     daemonsets:
       rollingUpdate:
         maxUnavailable: '1'
       updateStrategy: RollingUpdate
     sandboxWorkloads:
       defaultWorkload: container
       enabled: false
     gds:
       enabled: true
       image: nvidia-fs
       version: 2.20.5
       repository: nvcr.io/nvidia/cloud-native
     vgpuManager:
       enabled: false
     vfioManager:
       enabled: true
     toolkit:
       installDir: /usr/local/nvidia
       enabled: true

   Copy to Clipboard Toggle word wrap

3. GPU ClusterPolicy 사용자 정의 리소스가 생성되면 다음 명령을 실행하여 클러스터에 리소스를 생성합니다.

   $ oc create -f gpu-cluster-policy.yaml

   Copy to Clipboard Toggle word wrap

   출력 예

   clusterpolicy.nvidia.com/gpu-cluster-policy created

   Copy to Clipboard Toggle word wrap

4. 다음 명령을 실행하여 Operator가 설치되어 실행 중인지 확인합니다.

   $ oc get pods -n nvidia-gpu-operator

   Copy to Clipboard Toggle word wrap

   출력 예

   NAME                                                  READY   STATUS      RESTARTS   AGE
   gpu-feature-discovery-d5ngn                           1/1     Running     0          3m20s
   gpu-feature-discovery-z42rx                           1/1     Running     0          3m23s
   gpu-operator-6bb4d4b4c5-njh78                         1/1     Running     0          4m35s
   nvidia-container-toolkit-daemonset-bkh8l              1/1     Running     0          3m20s
   nvidia-container-toolkit-daemonset-c4hzm              1/1     Running     0          3m23s
   nvidia-cuda-validator-4blvg                           0/1     Completed   0          106s
   nvidia-cuda-validator-tw8sl                           0/1     Completed   0          112s
   nvidia-dcgm-exporter-rrw4g                            1/1     Running     0          3m20s
   nvidia-dcgm-exporter-xc78t                            1/1     Running     0          3m23s
   nvidia-dcgm-nvxpf                                     1/1     Running     0          3m20s
   nvidia-dcgm-snj4j                                     1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-fk2xz                  1/1     Running     0          3m23s
   nvidia-device-plugin-daemonset-wq87j                  1/1     Running     0          3m20s
   nvidia-driver-daemonset-416.94.202410211619-0-ngrjg   4/4     Running     0          3m58s
   nvidia-driver-daemonset-416.94.202410211619-0-tm4x6   4/4     Running     0          3m58s
   nvidia-node-status-exporter-jlzxh                     1/1     Running     0          3m57s
   nvidia-node-status-exporter-zjffs                     1/1     Running     0          3m57s
   nvidia-operator-validator-l49hx                       1/1     Running     0          3m20s
   nvidia-operator-validator-n44nn                       1/1     Running     0          3m23s

   Copy to Clipboard Toggle word wrap

5. 선택 사항: Pod가 실행 중인지 확인하면 NVIDIA 드라이버 daemonset Pod에 원격 쉘이 로드되었는지 확인하고 NVIDIA 모듈이 로드되었는지 확인합니다. 특히 nvidia_peermem 이 로드되었는지 확인합니다.

   $ oc rsh -n nvidia-gpu-operator $(oc -n nvidia-gpu-operator get pod -o name -l app.kubernetes.io/component=nvidia-driver)
   sh-4.4# lsmod|grep nvidia

   Copy to Clipboard Toggle word wrap

   출력 예

   nvidia_fs             327680  0
   nvidia_peermem         24576  0
   nvidia_modeset       1507328  0
   video                  73728  1 nvidia_modeset
   nvidia_uvm           6889472  8
   nvidia               8810496  43 nvidia_uvm,nvidia_peermem,nvidia_fs,gdrdrv,nvidia_modeset
   ib_uverbs             217088  3 nvidia_peermem,rdma_ucm,mlx5_ib
   drm                   741376  5 drm_kms_helper,drm_shmem_helper,nvidia,mgag200

   Copy to Clipboard Toggle word wrap

6. 선택 사항: nvidia-smi 유틸리티를 실행하여 드라이버 및 하드웨어에 대한 세부 정보를 표시합니다.

1. 드라이버 Pod에 있는 동안 nvidia-smi 명령을 사용하여 GPU 클럭을 최대값으로 설정합니다.

   $ oc rsh -n nvidia-gpu-operator nvidia-driver-daemonset-416.94.202410172137-0-ndhzc
   sh-4.4# nvidia-smi -i 0 -lgc $(nvidia-smi -i 0 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   Copy to Clipboard Toggle word wrap

   출력 예

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:61:00.0
   All done.

   Copy to Clipboard Toggle word wrap

   sh-4.4# nvidia-smi -i 1 -lgc $(nvidia-smi -i 1 --query-supported-clocks=graphics --format=csv,noheader,nounits | sort -h | tail -n 1)

   Copy to Clipboard Toggle word wrap

   출력 예

   GPU clocks set to "(gpuClkMin 1740, gpuClkMax 1740)" for GPU 00000000:E1:00.0
   All done.

   Copy to Clipboard Toggle word wrap

2. 다음 명령을 실행하여 노드 설명 화면에서 리소스를 사용할 수 있는지 확인합니다.

   $ oc describe node -l node-role.kubernetes.io/worker=| grep -E 'Capacity:|Allocatable:' -A9

   Copy to Clipboard Toggle word wrap

   출력 예

   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596712Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445736Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   --
   Capacity:
     cpu:                          128
     ephemeral-storage:            1561525616Ki
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       263596672Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63
   Allocatable:
     cpu:                          127500m
     ephemeral-storage:            1438028263499
     hugepages-1Gi:                0
     hugepages-2Mi:                0
     memory:                       262445696Ki
     nvidia.com/gpu:               2
     pods:                         250
     rdma/rdma_shared_device_eth:  63
     rdma/rdma_shared_device_ib:   63

   Copy to Clipboard Toggle word wrap

프로세스

1. Machineconfig CR을 생성합니다.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 02-worker-container-runtime
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZV0KZGVmYXVsdF91bGltaXRzID0gWwoibWVtbG9jaz0tMTotMSIKXQo=
           mode: 420
           overwrite: true
           path: /etc/crio/crio.conf.d/10-custom

   Copy to Clipboard Toggle word wrap

프로세스

1. 다음 명령을 사용하여 각 rdma-workload-client Pod에 연결합니다.

   $ oc rsh -n default rdma-sriov-32-workload

   Copy to Clipboard Toggle word wrap

   출력 예

   sh-5.1#

   Copy to Clipboard Toggle word wrap

2. 다음 명령을 사용하여 첫 번째 워크로드 Pod에 할당된 IP 주소를 확인합니다. 이 예에서 첫 번째 워크로드 Pod는 RDMA 테스트 서버입니다.

   sh-5.1# ip a

   Copy to Clipboard Toggle word wrap

   출력 예

   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
   2: eth0@if3970: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
       link/ether 0a:58:0a:80:02:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
       inet 10.128.2.167/23 brd 10.128.3.255 scope global eth0
          valid_lft forever preferred_lft forever
       inet6 fe80::858:aff:fe80:2a7/64 scope link
          valid_lft forever preferred_lft forever
   3843: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
       link/ether 26:34:fd:53:a6:ec brd ff:ff:ff:ff:ff:ff
       altname enp55s0f0v5
       inet 192.168.4.225/28 brd 192.168.4.239 scope global net1
          valid_lft forever preferred_lft forever
       inet6 fe80::2434:fdff:fe53:a6ec/64 scope link
          valid_lft forever preferred_lft forever
   sh-5.1#

   Copy to Clipboard Toggle word wrap

   이 Pod에 할당된 RDMA 서버의 IP 주소는 net1 인터페이스입니다. 이 예에서 IP 주소는 192.168.4.225 입니다.

3. ibstatus 명령을 실행하여 각 RDMA 장치 mlx5_x 와 연결된 link_layer 유형, 이더넷 또는 Infiniband를 가져옵니다. 출력에는 ACTIVE 또는 DOWN 이 표시되는 state 필드를 확인하여 모든 RDMA 장치의 상태도 표시됩니다.

   sh-5.1# ibstatus

   Copy to Clipboard Toggle word wrap

   출력 예

   Infiniband device 'mlx5_0' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_1' port 1 status:
   	default gid:	 fe80:0000:0000:0000:e8eb:d303:0072:1415
   	base lid:	 0xc
   	sm lid:		 0x1
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 InfiniBand
   
   Infiniband device 'mlx5_2' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_3' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_4' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_5' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_6' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_7' port 1 status:
   	default gid:	 fe80:0000:0000:0000:2434:fdff:fe53:a6ec
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_8' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_9' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   sh-5.1#

   Copy to Clipboard Toggle word wrap

4. 작업자 노드에서 각 RDMA mlx5 장치에 대한 link_layer 를 가져오려면 ibstat 명령을 실행합니다.

   sh-5.1# ibstat | egrep "Port|Base|Link"

   Copy to Clipboard Toggle word wrap

   출력 예

   Port 1:
   		Physical state: LinkUp
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Physical state: LinkUp
   		Base lid: 12
   		Port GUID: 0xe8ebd30300721415
   		Link layer: InfiniBand
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Physical state: LinkUp
   		Base lid: 0
   		Port GUID: 0x2434fdfffe53a6ec
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   	Port 1:
   		Base lid: 0
   		Port GUID: 0x0000000000000000
   		Link layer: Ethernet
   sh-5.1#

   Copy to Clipboard Toggle word wrap

5. RDMA 공유 장치 또는 호스트 장치 워크로드 Pod의 경우 mlx5_x 라는 RDMA 장치가 이미 알려져 있으며 일반적으로 mlx5_0 또는 mlx5_1 입니다. RDMA Legacy SR-IOV 워크로드 Pod의 경우 어떤 RDMA 장치가 어떤 VF(Virtual Function) 하위 인터페이스와 연결되어 있는지 확인해야 합니다. 다음 명령을 사용하여 이 정보를 제공합니다.

   sh-5.1# rdma link show

   Copy to Clipboard Toggle word wrap

   출력 예

   link mlx5_0/1 state ACTIVE physical_state LINK_UP
   link mlx5_1/1 subnet_prefix fe80:0000:0000:0000 lid 12 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP
   link mlx5_2/1 state DOWN physical_state DISABLED
   link mlx5_3/1 state DOWN physical_state DISABLED
   link mlx5_4/1 state DOWN physical_state DISABLED
   link mlx5_5/1 state DOWN physical_state DISABLED
   link mlx5_6/1 state DOWN physical_state DISABLED
   link mlx5_7/1 state ACTIVE physical_state LINK_UP netdev net1
   link mlx5_8/1 state DOWN physical_state DISABLED
   link mlx5_9/1 state DOWN physical_state DISABLED

   Copy to Clipboard Toggle word wrap

   이 예에서 RDMA 장치 이름 mlx5_7 은 net1 인터페이스와 연결되어 있습니다. 이 출력은 다음 명령에서 RDMA 대역폭 테스트를 수행하는 데 사용되며 작업자 노드 간의 RDMA 연결도 확인합니다.

6. 다음 ib_write_bw RDMA bandwidth test 명령을 실행합니다.

   sh-5.1# /root/perftest/ib_write_bw -R -T 41 -s 65536 -F -x 3 -m 4096 --report_gbits -q 16 -D 60  -d mlx5_7 -p 10000 --source_ip  192.168.4.225 --use_cuda=0 --use_cuda_dmabuf

   Copy to Clipboard Toggle word wrap

   다음과 같습니다.

   • mlx5_7 RDMA 장치는 -d 스위치로 전달됩니다.
   • RDMA 서버를 시작하는 소스 IP 주소는 192.168.4.225 입니다.
   • --use_cuda=0,--use_cuda_dmabuf 스위치는 GPUDirect RDMA를 사용함을 나타냅니다.

   출력 예

   WARNING: BW peak won't be measured in this run.
   Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
   
   ************************************
   * Waiting for client to connect... *
   ************************************

   Copy to Clipboard Toggle word wrap

7. 다른 터미널 창을 열고 RDMA 테스트 클라이언트 포드 역할을 하는 두 번째 워크로드 Pod에서 oc rsh 명령을 실행합니다.

   $ oc rsh -n default rdma-sriov-33-workload

   Copy to Clipboard Toggle word wrap

   출력 예

   sh-5.1#

   Copy to Clipboard Toggle word wrap

8. 다음 명령을 사용하여 net1 인터페이스에서 RDMA 테스트 클라이언트 포드 IP 주소를 가져옵니다.

   sh-5.1# ip a

   Copy to Clipboard Toggle word wrap

   출력 예

   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
   2: eth0@if4139: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
       link/ether 0a:58:0a:83:01:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
       inet 10.131.1.213/23 brd 10.131.1.255 scope global eth0
          valid_lft forever preferred_lft forever
       inet6 fe80::858:aff:fe83:1d5/64 scope link
          valid_lft forever preferred_lft forever
   4076: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
       link/ether 56:6c:59:41:ae:4a brd ff:ff:ff:ff:ff:ff
       altname enp55s0f0v0
       inet 192.168.4.226/28 brd 192.168.4.239 scope global net1
          valid_lft forever preferred_lft forever
       inet6 fe80::546c:59ff:fe41:ae4a/64 scope link
          valid_lft forever preferred_lft forever
   sh-5.1#

   Copy to Clipboard Toggle word wrap

9. 다음 명령을 사용하여 각 RDMA 장치 mlx5_x 와 연결된 link_layer 유형을 가져옵니다.

   sh-5.1# ibstatus

   Copy to Clipboard Toggle word wrap

   출력 예

   Infiniband device 'mlx5_0' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_1' port 1 status:
   	default gid:	 fe80:0000:0000:0000:e8eb:d303:0072:09f5
   	base lid:	 0xd
   	sm lid:		 0x1
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 InfiniBand
   
   Infiniband device 'mlx5_2' port 1 status:
   	default gid:	 fe80:0000:0000:0000:546c:59ff:fe41:ae4a
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 4: ACTIVE
   	phys state:	 5: LinkUp
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_3' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_4' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_5' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_6' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_7' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_8' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet
   
   Infiniband device 'mlx5_9' port 1 status:
   	default gid:	 0000:0000:0000:0000:0000:0000:0000:0000
   	base lid:	 0x0
   	sm lid:		 0x0
   	state:		 1: DOWN
   	phys state:	 3: Disabled
   	rate:		 200 Gb/sec (4X HDR)
   	link_layer:	 Ethernet

   Copy to Clipboard Toggle word wrap

10. 선택 사항: ibstat 명령을 사용하여 Mellanox 카드의 펌웨어 버전을 가져옵니다.

    sh-5.1# ibstat

    Copy to Clipboard Toggle word wrap

    출력 예

    CA 'mlx5_0'
    	CA type: MT4123
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0xe8ebd303007209f4
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Active
    		Physical state: LinkUp
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_1'
    	CA type: MT4123
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0xe8ebd303007209f5
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Active
    		Physical state: LinkUp
    		Rate: 200
    		Base lid: 13
    		LMC: 0
    		SM lid: 1
    		Capability mask: 0xa651e848
    		Port GUID: 0xe8ebd303007209f5
    		Link layer: InfiniBand
    CA 'mlx5_2'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0x566c59fffe41ae4a
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Active
    		Physical state: LinkUp
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x546c59fffe41ae4a
    		Link layer: Ethernet
    CA 'mlx5_3'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0xb2ae4bfffe8f3d02
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_4'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0x2a9967fffe8bf272
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_5'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0x5aff2ffffe2e17e8
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_6'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0x121bf1fffe074419
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_7'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0xb22b16fffed03dd7
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_8'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0x523800fffe16d105
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    CA 'mlx5_9'
    	CA type: MT4124
    	Number of ports: 1
    	Firmware version: 20.43.1014
    	Hardware version: 0
    	Node GUID: 0xd2b4a1fffebdc4a9
    	System image GUID: 0xe8ebd303007209f4
    	Port 1:
    		State: Down
    		Physical state: Disabled
    		Rate: 200
    		Base lid: 0
    		LMC: 0
    		SM lid: 0
    		Capability mask: 0x00010000
    		Port GUID: 0x0000000000000000
    		Link layer: Ethernet
    sh-5.1#

    Copy to Clipboard Toggle word wrap

11. 클라이언트 워크로드 Pod에서 사용하는 가상 기능 하위 인터페이스와 연결된 RDMA 장치를 확인하려면 다음 명령을 실행합니다. 이 예에서 net1 인터페이스는 RDMA 장치 mlx5_2 를 사용하고 있습니다.

    sh-5.1# rdma link show

    Copy to Clipboard Toggle word wrap

    출력 예

    link mlx5_0/1 state ACTIVE physical_state LINK_UP
    link mlx5_1/1 subnet_prefix fe80:0000:0000:0000 lid 13 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP
    link mlx5_2/1 state ACTIVE physical_state LINK_UP netdev net1
    link mlx5_3/1 state DOWN physical_state DISABLED
    link mlx5_4/1 state DOWN physical_state DISABLED
    link mlx5_5/1 state DOWN physical_state DISABLED
    link mlx5_6/1 state DOWN physical_state DISABLED
    link mlx5_7/1 state DOWN physical_state DISABLED
    link mlx5_8/1 state DOWN physical_state DISABLED
    link mlx5_9/1 state DOWN physical_state DISABLED
    sh-5.1#

    Copy to Clipboard Toggle word wrap

12. 다음 ib_write_bw RDMA bandwidth test 명령을 실행합니다.

    sh-5.1# /root/perftest/ib_write_bw -R -T 41 -s 65536 -F -x 3 -m 4096 --report_gbits -q 16 -D 60  -d mlx5_2 -p 10000 --source_ip  192.168.4.226 --use_cuda=0 --use_cuda_dmabuf 192.168.4.225

    Copy to Clipboard Toggle word wrap

    다음과 같습니다.

    • mlx5_2 RDMA 장치는 -d 스위치로 전달됩니다.
    • RDMA 서버 192.168.4.225 의 소스 IP 주소 192.168.4.226 및 대상 IP 주소입니다.

    • --use_cuda=0,--use_cuda_dmabuf 스위치는 GPUDirect RDMA를 사용함을 나타냅니다.

      출력 예

      WARNING: BW peak won't be measured in this run.
      Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
      Requested mtu is higher than active mtu
      Changing to active mtu - 3
      initializing CUDA
      Listing all CUDA devices in system:
      CUDA device 0: PCIe address is 61:00
      
      Picking device No. 0
      [pid = 8909, dev = 0] device name = [NVIDIA A40]
      creating CUDA Ctx
      making it the current CUDA Ctx
      CUDA device integrated: 0
      using DMA-BUF for GPU buffer address at 0x7f8738600000 aligned at 0x7f8738600000 with aligned size 2097152
      allocated GPU buffer of a 2097152 address at 0x23a7420 for type CUDA_MEM_DEVICE
      Calling ibv_reg_dmabuf_mr(offset=0, size=2097152, addr=0x7f8738600000, fd=40) for QP #0
      ---------------------------------------------------------------------------------------
                          RDMA_Write BW Test
       Dual-port       : OFF		Device         : mlx5_2
       Number of qps   : 16		Transport type : IB
       Connection type : RC		Using SRQ      : OFF
       PCIe relax order: ON		Lock-free      : OFF
       ibv_wr* API     : ON		Using DDP      : OFF
       TX depth        : 128
       CQ Moderation   : 1
       CQE Poll Batch  : 16
       Mtu             : 1024[B]
       Link type       : Ethernet
       GID index       : 3
       Max inline data : 0[B]
       rdma_cm QPs	 : ON
       Data ex. method : rdma_cm 	TOS    : 41
      ---------------------------------------------------------------------------------------
       local address: LID 0000 QPN 0x012d PSN 0x3cb6d7
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x012e PSN 0x90e0ac
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x012f PSN 0x153f50
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0130 PSN 0x5e0128
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0131 PSN 0xd89752
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0132 PSN 0xe5fc16
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0133 PSN 0x236787
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0134 PSN 0xd9273e
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0135 PSN 0x37cfd4
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0136 PSN 0x3bff8f
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0137 PSN 0x81f2bd
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0138 PSN 0x575c43
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x0139 PSN 0x6cf53d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x013a PSN 0xcaaf6f
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x013b PSN 0x346437
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       local address: LID 0000 QPN 0x013c PSN 0xcc5865
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x026d PSN 0x359409
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x026e PSN 0xe387bf
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x026f PSN 0x5be79d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0270 PSN 0x1b4b28
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0271 PSN 0x76a61b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0272 PSN 0x3d50e1
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0273 PSN 0x1b572c
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0274 PSN 0x4ae1b5
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0275 PSN 0x5591b5
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0276 PSN 0xfa2593
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0277 PSN 0xd9473b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0278 PSN 0x2116b2
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x0279 PSN 0x9b83b6
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x027a PSN 0xa0822b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x027b PSN 0x6d930d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x027c PSN 0xb1a4d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
      ---------------------------------------------------------------------------------------
       #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
       65536      10329004         0.00               180.47 		     0.344228
      ---------------------------------------------------------------------------------------
      deallocating GPU buffer 00007f8738600000
      destroying current CUDA Ctx
      sh-5.1#

      Copy to Clipboard Toggle word wrap

      긍정적인 테스트는 Mpps에서 예상되는 BW 평균과 MsgRate를 보고 있습니다.

      ib_write_bw 명령이 완료되면 서버 측 출력도 서버 Pod에 표시됩니다. 다음 예제를 참조하십시오.

      출력 예

      WARNING: BW peak won't be measured in this run.
      Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
      
      ************************************
      * Waiting for client to connect... *
      ************************************
      Requested mtu is higher than active mtu
      Changing to active mtu - 3
      initializing CUDA
      Listing all CUDA devices in system:
      CUDA device 0: PCIe address is 61:00
      
      Picking device No. 0
      [pid = 9226, dev = 0] device name = [NVIDIA A40]
      creating CUDA Ctx
      making it the current CUDA Ctx
      CUDA device integrated: 0
      using DMA-BUF for GPU buffer address at 0x7f447a600000 aligned at 0x7f447a600000 with aligned size 2097152
      allocated GPU buffer of a 2097152 address at 0x2406400 for type CUDA_MEM_DEVICE
      Calling ibv_reg_dmabuf_mr(offset=0, size=2097152, addr=0x7f447a600000, fd=40) for QP #0
      ---------------------------------------------------------------------------------------
                          RDMA_Write BW Test
       Dual-port       : OFF		Device         : mlx5_7
       Number of qps   : 16		Transport type : IB
       Connection type : RC		Using SRQ      : OFF
       PCIe relax order: ON		Lock-free      : OFF
       ibv_wr* API     : ON		Using DDP      : OFF
       CQ Moderation   : 1
       CQE Poll Batch  : 16
       Mtu             : 1024[B]
       Link type       : Ethernet
       GID index       : 3
       Max inline data : 0[B]
       rdma_cm QPs	 : ON
       Data ex. method : rdma_cm 	TOS    : 41
      ---------------------------------------------------------------------------------------
       Waiting for client rdma_cm QP to connect
       Please run the same command with the IB/RoCE interface IP
      ---------------------------------------------------------------------------------------
       local address: LID 0000 QPN 0x026d PSN 0x359409
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x026e PSN 0xe387bf
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x026f PSN 0x5be79d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0270 PSN 0x1b4b28
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0271 PSN 0x76a61b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0272 PSN 0x3d50e1
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0273 PSN 0x1b572c
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0274 PSN 0x4ae1b5
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0275 PSN 0x5591b5
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0276 PSN 0xfa2593
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0277 PSN 0xd9473b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0278 PSN 0x2116b2
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x0279 PSN 0x9b83b6
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x027a PSN 0xa0822b
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x027b PSN 0x6d930d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       local address: LID 0000 QPN 0x027c PSN 0xb1a4d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:225
       remote address: LID 0000 QPN 0x012d PSN 0x3cb6d7
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x012e PSN 0x90e0ac
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x012f PSN 0x153f50
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0130 PSN 0x5e0128
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0131 PSN 0xd89752
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0132 PSN 0xe5fc16
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0133 PSN 0x236787
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0134 PSN 0xd9273e
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0135 PSN 0x37cfd4
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0136 PSN 0x3bff8f
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0137 PSN 0x81f2bd
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0138 PSN 0x575c43
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x0139 PSN 0x6cf53d
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x013a PSN 0xcaaf6f
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x013b PSN 0x346437
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
       remote address: LID 0000 QPN 0x013c PSN 0xcc5865
       GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:226
      ---------------------------------------------------------------------------------------
       #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
       65536      10329004         0.00               180.47 		     0.344228
      ---------------------------------------------------------------------------------------
      deallocating GPU buffer 00007f447a600000
      destroying current CUDA Ctx

      Copy to Clipboard Toggle word wrap