Chapter 8. Using DPDK and RDMA

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the intel-dpdk-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: intel-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: intelnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "8086"
       deviceID: "158b"
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: vfio-pci 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the driver type for the virtual functions to vfio-pci.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f intel-dpdk-node-policy.yaml

   Copy to Clipboard Toggle word wrap

3. Create the following SriovNetwork object, and then save the YAML in the intel-dpdk-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: intel-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |-
   # ... 

   1

   
     vlan: <vlan>
     resourceName: intelnics

   Copy to Clipboard Toggle word wrap

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

4. Create the SriovNetwork object by running the following command:

   $ oc create -f intel-dpdk-network.yaml

   Copy to Clipboard Toggle word wrap

5. Create the following Pod spec, and then save the YAML in the intel-dpdk-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 

   1

   
     annotations:
       k8s.v1.cni.cncf.io/networks: intel-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 

   2

   
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 

   3

   
       volumeMounts:
       - mountPath: /mnt/huge 

   4

   
         name: hugepage
       resources:
         limits:
           openshift.io/intelnics: "1" 

   5

   
           memory: "1Gi"
           cpu: "4" 

   6

   
           hugepages-1Gi: "4Gi" 

   7

   
         requests:
           openshift.io/intelnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   Copy to Clipboard Toggle word wrap

   1

   Specify the same target_namespace where the SriovNetwork object intel-dpdk-network is created. If you would like to create the pod in a different namespace, change target_namespace in both the Pod spec and the SriovNetwork object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount a hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated to DPDK pod. This resource request and limit, if not explicitly specified, will be automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by the SR-IOV Operator. It is enabled by default and can be disabled by setting enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs to be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes. For example, adding kernel arguments default_hugepagesz=1GB, hugepagesz=1G and hugepages=16 will result in 16*1Gi hugepages be allocated during system boot.

6. Create the DPDK pod by running the following command:

   $ oc create -f intel-dpdk-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Save the following SriovNetworkNodePolicy YAML configuration to an mlx-dpdk-node-policy.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 

   1

   
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 

   2

   
     isRdma: true 

   3

   Copy to Clipboard Toggle word wrap

   1

   Specify the device hex code of the SR-IOV network device.

   2

   Specify the driver type for the virtual functions to netdevice. A Mellanox SR-IOV Virtual Function (VF) can work in DPDK mode without using the vfio-pci device type. The VF device appears as a kernel network interface inside a container.

   3

   Enable Remote Direct Memory Access (RDMA) mode. This is required for Mellanox cards to work in DPDK mode.

   Note

   See Configuring an SR-IOV network device for a detailed explanation of each option in the SriovNetworkNodePolicy object.

   When applying the configuration specified in an SriovNetworkNodePolicy object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes. It might take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-dpdk-node-policy.yaml

   Copy to Clipboard Toggle word wrap

3. Save the following SriovNetwork YAML configuration to an mlx-dpdk-network.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 

   1

   
   ...
     vlan: <vlan>
     resourceName: mlxnics

   Copy to Clipboard Toggle word wrap

   1

   Specify a configuration object for the IP Address Management (IPAM) Container Network Interface (CNI) plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See Configuring an SR-IOV network device for a detailed explanation on each option in the SriovNetwork object.

   The app-netutil option library provides several API methods for gathering network information about the parent pod of a container.

4. Create the SriovNetwork object by running the following command:

   $ oc create -f mlx-dpdk-network.yaml

   Copy to Clipboard Toggle word wrap

5. Save the following Pod YAML configuration to an mlx-dpdk-pod.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 

   1

   
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 

   2

   
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 

   3

   
       volumeMounts:
       - mountPath: /mnt/huge 

   4

   
         name: hugepage
       resources:
         limits:
           openshift.io/mlxnics: "1" 

   5

   
           memory: "1Gi"
           cpu: "4" 

   6

   
           hugepages-1Gi: "4Gi" 

   7

   
         requests:
           openshift.io/mlxnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   Copy to Clipboard Toggle word wrap

   1

   Specify the same target_namespace where SriovNetwork object mlx-dpdk-network is created. To create the pod in a different namespace, change target_namespace in both the Pod spec and SriovNetwork object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by the application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated for the DPDK pod. If not explicitly specified, this resource request and limit is automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by SR-IOV Operator. It is enabled by default and can be disabled by setting the enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires that exclusive CPUs be allocated from the kubelet. To do this, set the CPU Manager policy to static and create a pod with Guaranteed Quality of Service (QoS).

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepages requires adding kernel arguments to Nodes.

6. Create the DPDK pod by running the following command:

   $ oc create -f mlx-dpdk-pod.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a file, such as test-namespace.yaml, with content like the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: test-namespace
     labels:
       pod-security.kubernetes.io/enforce: privileged
       pod-security.kubernetes.io/audit: privileged
       pod-security.kubernetes.io/warn: privileged
       security.openshift.io/scc.podSecurityLabelSync: "false"

   Copy to Clipboard Toggle word wrap

2. Create the new Namespace object by running the following command:

   $ oc apply -f test-namespace.yaml

   Copy to Clipboard Toggle word wrap

3. Create a file, such as sriov-node-network-policy.yaml, with content like the following example::

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
    name: sriovnic
    namespace: openshift-sriov-network-operator
   spec:
    deviceType: netdevice 

   1

   
    isRdma: true 

   2

   
    needVhostNet: true 

   3

   
    nicSelector:
      vendor: "15b3" 

   4

   
      deviceID: "101b" 

   5

   
      rootDevices: ["00:05.0"]
    numVfs: 10
    priority: 99
    resourceName: sriovnic
    nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"

   Copy to Clipboard Toggle word wrap

   1

   This indicates that the profile is tailored specifically for Mellanox Network Interface Controllers (NICs).

   2

   Setting isRdma to true is only required for a Mellanox NIC.

   3

   This mounts the /dev/net/tun and /dev/vhost-net devices into the container so the application can create a tap device and connect the tap device to the DPDK workload.

   4

   The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

   5

   The device hexadecimal code of the SR-IOV network device.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f sriov-node-network-policy.yaml

   Copy to Clipboard Toggle word wrap

5. Create the following SriovNetwork object, and then save the YAML in the sriov-network-attachment.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
    name: sriov-network
    namespace: openshift-sriov-network-operator
   spec:
    networkNamespace: test-namespace
    resourceName: sriovnic
    spoofChk: "off"
    trust: "on"

   Copy to Clipboard Toggle word wrap
   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

6. Create the SriovNetwork object by running the following command:

   $ oc create -f sriov-network-attachment.yaml

   Copy to Clipboard Toggle word wrap

7. Create a file, such as tap-example.yaml, that defines a network attachment definition, with content like the following example:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
    name: tap-one
    namespace: test-namespace 

   1

   
   spec:
    config: '{
      "cniVersion": "0.4.0",
      "name": "tap",
      "plugins": [
        {
           "type": "tap",
           "multiQueue": true,
           "selinuxcontext": "system_u:system_r:container_t:s0"
        },
        {
          "type":"tuning",
          "capabilities":{
            "mac":true
          }
        }
      ]
    }'

   Copy to Clipboard Toggle word wrap

   1

   Specify the same target_namespace where the SriovNetwork object is created.

8. Create the NetworkAttachmentDefinition object by running the following command:

   $ oc apply -f tap-example.yaml

   Copy to Clipboard Toggle word wrap

9. Create a file, such as dpdk-pod-rootless.yaml, with content like the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: test-namespace 

   1

   
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {"name": "sriov-network", "namespace": "test-namespace"},
         {"name": "tap-one", "interface": "ext0", "namespace": "test-namespace"}]'
   spec:
     nodeSelector:
       kubernetes.io/hostname: "worker-0"
     securityContext:
         fsGroup: 1001 

   2

   
         runAsGroup: 1001 

   3

   
         seccompProfile:
           type: RuntimeDefault
     containers:
     - name: testpmd
       image: <DPDK_image> 

   4

   
       securityContext:
         capabilities:
           drop: ["ALL"] 

   5

   
           add: 

   6

   
             - IPC_LOCK
             - NET_RAW #for mlx only 

   7

   
         runAsUser: 1001 

   8

   
         privileged: false 

   9

   
         allowPrivilegeEscalation: true 

   10

   
         runAsNonRoot: true 

   11

   
       volumeMounts:
       - mountPath: /mnt/huge 

   12

   
         name: hugepages
       resources:
         limits:
           openshift.io/sriovnic: "1" 

   13

   
           memory: "1Gi"
           cpu: "4" 

   14

   
           hugepages-1Gi: "4Gi" 

   15

   
         requests:
           openshift.io/sriovnic: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     runtimeClassName: performance-cnf-performanceprofile 

   16

   
     volumes:
     - name: hugepages
       emptyDir:
         medium: HugePages

   Copy to Clipboard Toggle word wrap

   1

   Specify the same target_namespace in which the SriovNetwork object is created. If you want to create the pod in a different namespace, change target_namespace in both the Pod spec and the SriovNetwork object.

   2

   Sets the group ownership of volume-mounted directories and files created in those volumes.

   3

   Specify the primary group ID used for running the container.

   4

   Specify the DPDK image that contains your application and the DPDK library used by application.

   5

   Removing all capabilities (ALL) from the container’s securityContext means that the container has no special privileges beyond what is necessary for normal operation.

   6

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access. These capabilities must also be set in the binary file by using the setcap command.

   7

   Mellanox network interface controller (NIC) requires the NET_RAW capability.

   8

   Specify the user ID used for running the container.

   9

   This setting indicates that the container or containers within the pod should not be granted privileged access to the host system.

   10

   This setting allows a container to escalate its privileges beyond the initial non-root privileges it might have been assigned.

   11

   This setting ensures that the container runs with a non-root user. This helps enforce the principle of least privilege, limiting the potential impact of compromising the container and reducing the attack surface.

   12

   Mount a hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   13

   Optional: Specify the number of DPDK devices allocated for the DPDK pod. If not explicitly specified, this resource request and limit is automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by SR-IOV Operator. It is enabled by default and can be disabled by setting the enableInjector option to false in the default SriovOperatorConfig CR.

   14

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs to be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   15

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes. For example, adding kernel arguments default_hugepagesz=1GB, hugepagesz=1G and hugepages=16 will result in 16*1Gi hugepages be allocated during system boot.

   16

   If your performance profile is not named cnf-performance profile, replace that string with the correct performance profile name.

10. Create the DPDK pod by running the following command:

    $ oc create -f dpdk-pod-rootless.yaml

    Copy to Clipboard Toggle word wrap

Procedure

1. Create a PerformanceProfile object based on the following example:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     globallyDisableIrqLoadBalancing: true
     cpu:
       isolated: 21-51,73-103 

   1

   
       reserved: 0-20,52-72 

   2

   
     hugepages:
       defaultHugepagesSize: 1G 

   3

   
       pages:
         - count: 32
           size: 1G
     net:
       userLevelNetworking: true
     numa:
       topologyPolicy: "single-numa-node"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

   Copy to Clipboard Toggle word wrap

   1

   If hyperthreading is enabled on the system, allocate the relevant symbolic links to the isolated and reserved CPU groups. If the system contains multiple non-uniform memory access nodes (NUMAs), allocate CPUs from both NUMAs to both groups. You can also use the Performance Profile Creator for this task. For more information, see Creating a performance profile.

   2

   You can also specify a list of devices that will have their queues set to the reserved CPU count. For more information, see Reducing NIC queues using the Node Tuning Operator.

   3

   Allocate the number and size of hugepages needed. You can specify the NUMA configuration for the hugepages. By default, the system allocates an even number to every NUMA node on the system. If needed, you can request the use of a realtime kernel for the nodes. See Provisioning a worker with real-time capabilities for more information.

2. Save the yaml file as mlx-dpdk-perfprofile-policy.yaml.

3. Apply the performance profile using the following command:

   $ oc create -f mlx-dpdk-perfprofile-policy.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-rdma-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-rdma-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 

   1

   
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 

   2

   
     isRdma: true 

   3

   Copy to Clipboard Toggle word wrap

   1

   Specify the device hex code of the SR-IOV network device.

   2

   Specify the driver type for the virtual functions to netdevice.

   3

   Enable RDMA mode.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-node-policy.yaml

   Copy to Clipboard Toggle word wrap

3. Create the following SriovNetwork object, and then save the YAML in the mlx-rdma-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-rdma-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 

   1

   
   # ...
     vlan: <vlan>
     resourceName: mlxnics

   Copy to Clipboard Toggle word wrap

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-network.yaml

   Copy to Clipboard Toggle word wrap

5. Create the following Pod spec, and then save the YAML in the mlx-rdma-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: rdma-app
     namespace: <target_namespace> 

   1

   
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-rdma-network
   spec:
     containers:
     - name: testpmd
       image: <RDMA_image> 

   2

   
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 

   3

   
       volumeMounts:
       - mountPath: /mnt/huge 

   4

   
         name: hugepage
       resources:
         limits:
           memory: "1Gi"
           cpu: "4" 

   5

   
           hugepages-1Gi: "4Gi" 

   6

   
         requests:
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   Copy to Clipboard Toggle word wrap

   1

   Specify the same target_namespace where SriovNetwork object mlx-rdma-network is created. If you would like to create the pod in a different namespace, change target_namespace in both Pod spec and SriovNetwork object.

   2

   Specify the RDMA image which includes your application and RDMA library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to RDMA pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Specify number of CPUs. The RDMA pod usually requires exclusive CPUs be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and create pod with Guaranteed QoS.

   6

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the RDMA pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes.

6. Create the RDMA pod by running the following command:

   $ oc create -f mlx-rdma-pod.yaml

   Copy to Clipboard Toggle word wrap