第 5 章 为 SR-IOV 配置 RDMA 子系统

流程

1. 创建一个 SriovNetworkPoolConfig CR，并将它保存为 sriov-nw-pool.yaml，如下例所示：

   SriovNetworkPoolConfig CR 示例

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkPoolConfig
   metadata:
     name: worker
     namespace: openshift-sriov-network-operator
   spec:
     maxUnavailable: 1
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker: ""
     rdmaMode: exclusive 

   1

   Copy to Clipboard Toggle word wrap

   1

   将 RDMA 网络命名空间模式设置为 exclusive。

2. 运行以下命令来创建 SriovNetworkPoolConfig 资源：

   $ oc create -f sriov-nw-pool.yaml

   Copy to Clipboard Toggle word wrap

3. 创建一个 SriovNetworkNodePolicy CR，并将它保存为 sriov-node-policy.yaml，如下例所示：

   SriovNetworkNodePolicy CR 示例

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: sriov-nic-pf1
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: netdevice
     isRdma: true 

   1

   
     nicSelector:
       pfNames: ["ens3f0np0"]
     nodeSelector:
       node-role.kubernetes.io/worker: ""
     numVfs: 4
     priority: 99
     resourceName: sriov_nic_pf1

   Copy to Clipboard Toggle word wrap

   1

   激活 RDMA 模式。

4. 运行以下命令来创建 SriovNetworkNodePolicy 资源：

   $ oc create -f sriov-node-policy.yaml

   Copy to Clipboard Toggle word wrap

5. 创建一个 SriovNetwork CR，并将它保存为 sriov-network.yaml，如下例所示：

   SriovNetwork CR 示例

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: sriov-nic-pf1
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: sriov-tests
     resourceName: sriov_nic_pf1
       ipam: |-
     metaPlugins: |
       {
         "type": "rdma" 

   1

   
       }

   Copy to Clipboard Toggle word wrap

   1

   创建 RDMA 插件。

6. 运行以下命令来创建 SriovNetwork 资源：

   $ oc create -f sriov-network.yaml

   Copy to Clipboard Toggle word wrap

验证

1. 创建 Pod CR，并将它保存为 sriov-test-pod.yaml，如下例所示：

   运行时配置示例

   apiVersion: v1
   kind: Pod
   metadata:
     name: sample-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: |-
         [
           {
             "name": "net1",
             "mac": "20:04:0f:f1:88:01",
             "ips": ["192.168.10.1/24", "2001::1/64"]
           }
         ]
   spec:
     containers:
     - name: sample-container
       image: <image>
       imagePullPolicy: IfNotPresent
       command: ["sleep", "infinity"]

   Copy to Clipboard Toggle word wrap

2. 运行以下命令来创建测试 pod：

   $ oc create -f sriov-test-pod.yaml

   Copy to Clipboard Toggle word wrap

3. 运行以下命令登录到测试 pod：

   $ oc rsh testpod1 -n sriov-tests

   Copy to Clipboard Toggle word wrap

4. 运行以下命令，验证 hw-counters 目录的路径是否存在：

   $ ls /sys/bus/pci/devices/${PCIDEVICE_OPENSHIFT_IO_SRIOV_NIC_PF1}/infiniband/*/ports/1/hw_counters/

   Copy to Clipboard Toggle word wrap

   输出示例

   duplicate_request       out_of_buffer req_cqe_flush_error           resp_cqe_flush_error        roce_adp_retrans        roce_slow_restart_trans
   implied_nak_seq_err     out_of_sequence req_remote_access_errors    resp_local_length_error     roce_adp_retrans_to     rx_atomic_requests
   lifespan                packet_seq_err req_remote_invalid_request   resp_remote_access_errors   roce_slow_restart       rx_read_requests
   local_ack_timeout_err  req_cqe_error resp_cqe_error                 rnr_nak_retry_err           roce_slow_restart_cnps  rx_write_requests

   Copy to Clipboard Toggle word wrap