Chapter 23. Hardware networks

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.
2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".

3. Create the SriovNetworkNodePolicy object:

   $ oc create -f <name>-sriov-node-network.yaml

   where <name> specifies the name for this configuration.

   After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Procedure

1. Create a SriovNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: attach1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: net1
     networkNamespace: project2
     ipam: |-
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "gateway": "10.56.217.1"
       }

2. To create the object, enter the following command:

   $ oc create -f <name>.yaml

   where <name> specifies the name of the additional network.

3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Create a SriovIBNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovIBNetwork
   metadata:
     name: attach1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: net1
     networkNamespace: project2
     ipam: |-
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "gateway": "10.56.217.1"
       }

2. To create the object, enter the following command:

   $ oc create -f <name>.yaml

   where <name> specifies the name of the additional network.

3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovIBNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovIBNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Add an annotation to the Pod object. Only one of the following annotation formats can be used:

   1. To attach an additional network without any customization, add an annotation with the following format. Replace <network> with the name of the additional network to associate with the pod:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: <network>[,<network>,...] 1

      1

      To specify more than one additional network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same additional network multiple times, that pod will have multiple network interfaces attached to that network.

   2. To attach an additional network with customizations, add an annotation with the following format:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "<network>", 1
                "namespace": "<namespace>", 2
                "default-route": ["<default-route>"] 3
              }
            ]

      1

      Specify the name of the additional network defined by a NetworkAttachmentDefinition object.

      2

      Specify the namespace where the NetworkAttachmentDefinition object is defined.

      3

      Optional: Specify an override for the default route, such as 192.168.17.1.

2. To create the pod, enter the following command. Replace <name> with the name of the pod.

   $ oc create -f <name>.yaml

3. Optional: To Confirm that the annotation exists in the Pod CR, enter the following command, replacing <name> with the name of the pod.

   $ oc get pod <name> -o yaml

   In the following example, the example-pod pod is attached to the net1 additional network:

   $ oc get pod example-pod -o yaml
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-bridge
       k8s.v1.cni.cncf.io/network-status: |- 1
         [{
             "name": "openshift-sdn",
             "interface": "eth0",
             "ips": [
                 "10.128.2.14"
             ],
             "default": true,
             "dns": {}
         },{
             "name": "macvlan-bridge",
             "interface": "net1",
             "ips": [
                 "20.2.2.100"
             ],
             "mac": "22:2f:60:a5:f8:00",
             "dns": {}
         }]
     name: example-pod
     namespace: default
   spec:
     ...
   status:
     ...

   1

   The k8s.v1.cni.cncf.io/network-status parameter is a JSON array of objects. Each object describes the status of an additional network attached to the pod. The annotation value is stored as a plain text value.

Procedure

1. Create the following SR-IOV pod spec, and then save the YAML in the <name>-sriov-pod.yaml file. Replace <name> with a name for this pod.

   The following example shows an SR-IOV pod spec:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sample-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: <name> 1
   spec:
     containers:
     - name: sample-container
       image: <image> 2
       command: ["sleep", "infinity"]
       resources:
         limits:
           memory: "1Gi" 3
           cpu: "2" 4
         requests:
           memory: "1Gi"
           cpu: "2"

   1

   Replace <name> with the name of the SR-IOV network attachment definition CR.

   2

   Replace <image> with the name of the sample-pod image.

   3

   To create the SR-IOV pod with guaranteed QoS, set memory limits equal to memory requests.

   4

   To create the SR-IOV pod with guaranteed QoS, set cpu limits equals to cpu requests.

2. Create the sample SR-IOV pod by running the following command:

   $ oc create -f <filename> 1

   1

   Replace <filename> with the name of the file you created in the previous step.

3. Confirm that the sample-pod is configured with guaranteed QoS.

   $ oc describe pod sample-pod

4. Confirm that the sample-pod is allocated with exclusive CPUs.

   $ oc exec sample-pod -- cat /sys/fs/cgroup/cpuset/cpuset.cpus

5. Confirm that the SR-IOV device and CPUs that are allocated for the sample-pod are on the same NUMA node.

   $ oc exec sample-pod -- cat /sys/fs/cgroup/cpuset/cpuset.cpus

1. Install the Node Feature Discovery (NFD) Operator. NFD detects the presence of SR-IOV enabled NICs and labels the nodes with node.alpha.kubernetes-incubator.io/nfd-network-sriov.capable = true.

2. Examine the SriovNetworkNodeState CR for each node. The interfaces stanza includes a list of all of the SR-IOV devices discovered by the SR-IOV Network Operator on the worker node. Label each node with feature.node.kubernetes.io/network-sriov.capable: "true" by using the following command:

   $ oc label node <node_name> feature.node.kubernetes.io/network-sriov.capable="true"

   Note

   You can label the nodes with whatever name you want.

Procedure

1. Create a file named sriovnetpolicy-810-sriov-node-network.yaml by using the following content:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: sriovnetpolicy-810
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: netdevice
     nicSelector:
       pfNames:
         - ens5f0#0-9
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numVfs: 10
     priority: 99
     resourceName: resource810

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f sriovnetpolicy-810-sriov-node-network.yaml

3. Open a separate terminal window and monitor the synchronization status of the SR-IOV network node state for the node specified in the openshift-sriov-network-operator namespace by running the following command:

   $ watch -n 1 'oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath="{.status.syncStatus}"'

   The synchronization status indicates a change from InProgress to Succeeded.

4. Create a SriovNetwork object, and set the outer VLAN called the S-tag, or Service Tag, as it belongs to the infrastructure.

   Important

   You must configure the VLAN on the trunk interface of the switch. In addition, you might need to further configure some switches to support QinQ tagging.

   1. Create a file named nad-sriovnetwork-1ad-810.yaml by using the following content:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: sriovnetwork-1ad-810
        namespace: openshift-sriov-network-operator
      spec:
        ipam: '{}'
        vlan: 171 1
        vlanProto: "802.1ad" 2
        networkNamespace: default
        resourceName: resource810

      1

      Sets the S-tag VLAN tag to 171.

      2

      Specifies the VLAN protocol to assign to the virtual function (VF). Supported values are 802.1ad and 802.1q. The default value is 802.1q.

   2. Create the object by running the following command:

      $ oc create -f nad-sriovnetwork-1ad-810.yaml

5. Create a NetworkAttachmentDefinition object with an inner VLAN. The inner VLAN is often referred to as the C-tag, or Customer Tag, as it belongs to the Network Function:

   1. Create a file named nad-cvlan100.yaml by using the following content:

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: nad-cvlan100
        namespace: default
      spec:
        config: '{
          "name": "vlan-100",
          "cniVersion": "0.3.1",
          "type": "vlan",
          "linkInContainer": true,
          "master": "net1", 1
          "vlanId": 100,
          "ipam": {"type": "static"}
        }'

      1

      Specifies the VF interface inside the pod. The default name is net1 as the name is not set in the pod annotation.

   2. Apply the YAML file by running the following command:

      $ oc apply -f nad-cvlan100.yaml

Procedure

1. Create a SriovNetworkNodePolicy object:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: policy-example
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: example
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     numVfs: 4
     nicSelector:
       vendor: "8086"
       pfNames: ['ens803f0']
       rootDevices: ['0000:86:00.0']

2. Create a SriovNetwork object:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: net-example
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: default
     ipam: | 1
       {
         "type": "host-local", 2
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "routes": [
           {"dst": "224.0.0.0/5"},
           {"dst": "232.0.0.0/5"}
         ],
         "gateway": "10.56.217.1"
       }
     resourceName: example

   1 2

   If you choose to configure DHCP as IPAM, ensure that you provision the following default routes through your DHCP server: 224.0.0.0/5 and 232.0.0.0/5. This is to override the static multicast route set by the default network provider.

3. Create a pod with multicast application:

   apiVersion: v1
   kind: Pod
   metadata:
     name: testpmd
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: nic1
   spec:
     containers:
     - name: example
       image: rhel7:latest
       securityContext:
         capabilities:
           add: ["NET_ADMIN"] 1
       command: [ "sleep", "infinity"]

   1

   The NET_ADMIN capability is required only if your application needs to assign the multicast IP address to the SR-IOV interface. Otherwise, it can be omitted.

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the intel-dpdk-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: intel-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: intelnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "8086"
       deviceID: "158b"
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: vfio-pci 1

   1

   Specify the driver type for the virtual functions to vfio-pci.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f intel-dpdk-node-policy.yaml

3. Create the following SriovNetwork object, and then save the YAML in the intel-dpdk-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: intel-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |-
   # ... 1
     vlan: <vlan>
     resourceName: intelnics

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

4. Create the SriovNetwork object by running the following command:

   $ oc create -f intel-dpdk-network.yaml

5. Create the following Pod spec, and then save the YAML in the intel-dpdk-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: intel-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /mnt/huge 4
         name: hugepage
       resources:
         limits:
           openshift.io/intelnics: "1" 5
           memory: "1Gi"
           cpu: "4" 6
           hugepages-1Gi: "4Gi" 7
         requests:
           openshift.io/intelnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where the SriovNetwork object intel-dpdk-network is created. If you would like to create the pod in a different namespace, change target_namespace in both the Pod spec and the SriovNetwork object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount a hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated to DPDK pod. This resource request and limit, if not explicitly specified, will be automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by the SR-IOV Operator. It is enabled by default and can be disabled by setting enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs to be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes. For example, adding kernel arguments default_hugepagesz=1GB, hugepagesz=1G and hugepages=16 will result in 16*1Gi hugepages be allocated during system boot.

6. Create the DPDK pod by running the following command:

   $ oc create -f intel-dpdk-pod.yaml

Procedure

1. Save the following SriovNetworkNodePolicy YAML configuration to an mlx-dpdk-node-policy.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 1
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 2
     isRdma: true 3

   1

   Specify the device hex code of the SR-IOV network device.

   2

   Specify the driver type for the virtual functions to netdevice. A Mellanox SR-IOV Virtual Function (VF) can work in DPDK mode without using the vfio-pci device type. The VF device appears as a kernel network interface inside a container.

   3

   Enable Remote Direct Memory Access (RDMA) mode. This is required for Mellanox cards to work in DPDK mode.

   Note

   See Configuring an SR-IOV network device for a detailed explanation of each option in the SriovNetworkNodePolicy object.

   When applying the configuration specified in an SriovNetworkNodePolicy object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes. It might take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-dpdk-node-policy.yaml

3. Save the following SriovNetwork YAML configuration to an mlx-dpdk-network.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 1
   ...
     vlan: <vlan>
     resourceName: mlxnics

   1

   Specify a configuration object for the IP Address Management (IPAM) Container Network Interface (CNI) plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See Configuring an SR-IOV network device for a detailed explanation on each option in the SriovNetwork object.

   The app-netutil option library provides several API methods for gathering network information about the parent pod of a container.

4. Create the SriovNetwork object by running the following command:

   $ oc create -f mlx-dpdk-network.yaml

5. Save the following Pod YAML configuration to an mlx-dpdk-pod.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /mnt/huge 4
         name: hugepage
       resources:
         limits:
           openshift.io/mlxnics: "1" 5
           memory: "1Gi"
           cpu: "4" 6
           hugepages-1Gi: "4Gi" 7
         requests:
           openshift.io/mlxnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where SriovNetwork object mlx-dpdk-network is created. To create the pod in a different namespace, change target_namespace in both the Pod spec and SriovNetwork object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by the application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated for the DPDK pod. If not explicitly specified, this resource request and limit is automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by SR-IOV Operator. It is enabled by default and can be disabled by setting the enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires that exclusive CPUs be allocated from the kubelet. To do this, set the CPU Manager policy to static and create a pod with Guaranteed Quality of Service (QoS).

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepages requires adding kernel arguments to Nodes.

6. Create the DPDK pod by running the following command:

   $ oc create -f mlx-dpdk-pod.yaml

Procedure

1. Create a file, such as test-namespace.yaml, with content like the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: test-namespace
     labels:
       pod-security.kubernetes.io/enforce: privileged
       pod-security.kubernetes.io/audit: privileged
       pod-security.kubernetes.io/warn: privileged
       security.openshift.io/scc.podSecurityLabelSync: "false"

2. Create the new Namespace object by running the following command:

   $ oc apply -f test-namespace.yaml

3. Create a file, such as sriov-node-network-policy.yaml, with content like the following example::

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
    name: sriovnic
    namespace: openshift-sriov-network-operator
   spec:
    deviceType: netdevice 1
    isRdma: true 2
    needVhostNet: true 3
    nicSelector:
      vendor: "15b3" 4
      deviceID: "101b" 5
      rootDevices: ["00:05.0"]
    numVfs: 10
    priority: 99
    resourceName: sriovnic
    nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"

   1

   This indicates that the profile is tailored specifically for Mellanox Network Interface Controllers (NICs).

   2

   Setting isRdma to true is only required for a Mellanox NIC.

   3

   This mounts the /dev/net/tun and /dev/vhost-net devices into the container so the application can create a tap device and connect the tap device to the DPDK workload.

   4

   The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

   5

   The device hexadecimal code of the SR-IOV network device.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f sriov-node-network-policy.yaml

5. Create the following SriovNetwork object, and then save the YAML in the sriov-network-attachment.yaml file:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
    name: sriov-network
    namespace: openshift-sriov-network-operator
   spec:
    networkNamespace: test-namespace
    resourceName: sriovnic
    spoofChk: "off"
    trust: "on"

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

6. Create the SriovNetwork object by running the following command:

   $ oc create -f sriov-network-attachment.yaml

7. Create a file, such as tap-example.yaml, that defines a network attachment definition, with content like the following example:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
    name: tap-one
    namespace: test-namespace 1
   spec:
    config: '{
      "cniVersion": "0.4.0",
      "name": "tap",
      "plugins": [
        {
           "type": "tap",
           "multiQueue": true,
           "selinuxcontext": "system_u:system_r:container_t:s0"
        },
        {
          "type":"tuning",
          "capabilities":{
            "mac":true
          }
        }
      ]
    }'

   1

   Specify the same target_namespace where the SriovNetwork object is created.

8. Create the NetworkAttachmentDefinition object by running the following command:

   $ oc apply -f tap-example.yaml

9. Create a file, such as dpdk-pod-rootless.yaml, with content like the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: test-namespace 1
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {"name": "sriov-network", "namespace": "test-namespace"},
         {"name": "tap-one", "interface": "ext0", "namespace": "test-namespace"}]'
   spec:
     nodeSelector:
       kubernetes.io/hostname: "worker-0"
     securityContext:
         fsGroup: 1001 2
         runAsGroup: 1001 3
         seccompProfile:
           type: RuntimeDefault
     containers:
     - name: testpmd
       image: <DPDK_image> 4
       securityContext:
         capabilities:
           drop: ["ALL"] 5
           add: 6
             - IPC_LOCK
             - NET_RAW #for mlx only 7
         runAsUser: 1001 8
         privileged: false 9
         allowPrivilegeEscalation: true 10
         runAsNonRoot: true 11
       volumeMounts:
       - mountPath: /mnt/huge 12
         name: hugepages
       resources:
         limits:
           openshift.io/sriovnic: "1" 13
           memory: "1Gi"
           cpu: "4" 14
           hugepages-1Gi: "4Gi" 15
         requests:
           openshift.io/sriovnic: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     runtimeClassName: performance-cnf-performanceprofile 16
     volumes:
     - name: hugepages
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace in which the SriovNetwork object is created. If you want to create the pod in a different namespace, change target_namespace in both the Pod spec and the SriovNetwork object.

   2

   Sets the group ownership of volume-mounted directories and files created in those volumes.

   3

   Specify the primary group ID used for running the container.

   4

   Specify the DPDK image that contains your application and the DPDK library used by application.

   5

   Removing all capabilities (ALL) from the container’s securityContext means that the container has no special privileges beyond what is necessary for normal operation.

   6

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access. These capabilities must also be set in the binary file by using the setcap command.

   7

   Mellanox network interface controller (NIC) requires the NET_RAW capability.

   8

   Specify the user ID used for running the container.

   9

   This setting indicates that the container or containers within the pod should not be granted privileged access to the host system.

   10

   This setting allows a container to escalate its privileges beyond the initial non-root privileges it might have been assigned.

   11

   This setting ensures that the container runs with a non-root user. This helps enforce the principle of least privilege, limiting the potential impact of compromising the container and reducing the attack surface.

   12

   Mount a hugepage volume to the DPDK pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   13

   Optional: Specify the number of DPDK devices allocated for the DPDK pod. If not explicitly specified, this resource request and limit is automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by SR-IOV Operator. It is enabled by default and can be disabled by setting the enableInjector option to false in the default SriovOperatorConfig CR.

   14

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs to be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   15

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes. For example, adding kernel arguments default_hugepagesz=1GB, hugepagesz=1G and hugepages=16 will result in 16*1Gi hugepages be allocated during system boot.

   16

   If your performance profile is not named cnf-performance profile, replace that string with the correct performance profile name.

10. Create the DPDK pod by running the following command:

    $ oc create -f dpdk-pod-rootless.yaml

Procedure

1. Create a PerformanceProfile object based on the following example:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     globallyDisableIrqLoadBalancing: true
     cpu:
       isolated: 21-51,73-103 1
       reserved: 0-20,52-72 2
     hugepages:
       defaultHugepagesSize: 1G 3
       pages:
         - count: 32
           size: 1G
     net:
       userLevelNetworking: true
     numa:
       topologyPolicy: "single-numa-node"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

   1

   If hyperthreading is enabled on the system, allocate the relevant symbolic links to the isolated and reserved CPU groups. If the system contains multiple non-uniform memory access nodes (NUMAs), allocate CPUs from both NUMAs to both groups. You can also use the Performance Profile Creator for this task. For more information, see Creating a performance profile.

   2

   You can also specify a list of devices that will have their queues set to the reserved CPU count. For more information, see Reducing NIC queues using the Node Tuning Operator.

   3

   Allocate the number and size of hugepages needed. You can specify the NUMA configuration for the hugepages. By default, the system allocates an even number to every NUMA node on the system. If needed, you can request the use of a realtime kernel for the nodes. See Provisioning a worker with real-time capabilities for more information.

2. Save the yaml file as mlx-dpdk-perfprofile-policy.yaml.

3. Apply the performance profile using the following command:

   $ oc create -f mlx-dpdk-perfprofile-policy.yaml

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-rdma-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-rdma-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 1
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 2
     isRdma: true 3

   1

   Specify the device hex code of the SR-IOV network device.

   2

   Specify the driver type for the virtual functions to netdevice.

   3

   Enable RDMA mode.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-node-policy.yaml

3. Create the following SriovNetwork object, and then save the YAML in the mlx-rdma-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-rdma-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 1
   # ...
     vlan: <vlan>
     resourceName: mlxnics

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

   An optional library, app-netutil, provides several API methods for gathering network information about a container’s parent pod.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-network.yaml

5. Create the following Pod spec, and then save the YAML in the mlx-rdma-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: rdma-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-rdma-network
   spec:
     containers:
     - name: testpmd
       image: <RDMA_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /mnt/huge 4
         name: hugepage
       resources:
         limits:
           memory: "1Gi"
           cpu: "4" 5
           hugepages-1Gi: "4Gi" 6
         requests:
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where SriovNetwork object mlx-rdma-network is created. If you would like to create the pod in a different namespace, change target_namespace in both Pod spec and SriovNetwork object.

   2

   Specify the RDMA image which includes your application and RDMA library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to RDMA pod under /mnt/huge. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Specify number of CPUs. The RDMA pod usually requires exclusive CPUs be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and create pod with Guaranteed QoS.

   6

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the RDMA pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes.

6. Create the RDMA pod by running the following command:

   $ oc create -f mlx-rdma-pod.yaml

Procedure

1. Create a SriovOperatorConfig custom resource (CR) to deploy all the SR-IOV Operator components:

   1. Create a file named sriovOperatorConfig.yaml that contains the following YAML:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovOperatorConfig
      metadata:
        name: default 1
        namespace: openshift-sriov-network-operator
      spec:
        enableInjector: true
        enableOperatorWebhook: true
        configurationMode: "systemd" 2
        logLevel: 2

      1

      The only valid name for the SriovOperatorConfig resource is default and it must be in the namespace where the Operator is deployed.

      2

      Setting the SR-IOV Network Operator into systemd mode is only relevant for Open vSwitch hardware offloading.

   2. Create the resource by running the following command:

      $ oc apply -f sriovOperatorConfig.yaml

Prerequisites

1. SR-IOV Network Operator installed and set into systemd mode.

Procedure

1. Create a machine config pool for machines you want to use hardware offloading on.

   1. Create a file, such as mcp-offloading.yaml, with content like the following example:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: mcp-offloading 1
      spec:
        machineConfigSelector:
          matchExpressions:
            - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,mcp-offloading]} 2
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/mcp-offloading: "" 3

      1 2

      The name of your machine config pool for hardware offloading.

      3

      This node role label is used to add nodes to the machine config pool.

   2. Apply the configuration for the machine config pool:

      $ oc create -f mcp-offloading.yaml

2. Add nodes to the machine config pool. Label each node with the node role label of your pool:

   $ oc label node worker-2 node-role.kubernetes.io/mcp-offloading=""

3. Optional: To verify that the new pool is created, run the following command:

   $ oc get nodes

   Example output

   NAME       STATUS   ROLES                   AGE   VERSION
   master-0   Ready    master                  2d    v1.29.4
   master-1   Ready    master                  2d    v1.29.4
   master-2   Ready    master                  2d    v1.29.4
   worker-0   Ready    worker                  2d    v1.29.4
   worker-1   Ready    worker                  2d    v1.29.4
   worker-2   Ready    mcp-offloading,worker   47h   v1.29.4
   worker-3   Ready    mcp-offloading,worker   47h   v1.29.4

4. Add this machine config pool to the SriovNetworkPoolConfig custom resource:

   1. Create a file, such as sriov-pool-config.yaml, with content like the following example:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkPoolConfig
      metadata:
        name: sriovnetworkpoolconfig-offload
        namespace: openshift-sriov-network-operator
      spec:
        ovsHardwareOffloadConfig:
          name: mcp-offloading 1

      1

      The name of your machine config pool for hardware offloading.

   2. Apply the configuration:

      $ oc create -f <SriovNetworkPoolConfig_name>.yaml

      Note

      When you apply the configuration specified in a SriovNetworkPoolConfig object, the SR-IOV Operator drains and restarts the nodes in the machine config pool.

      It might take several minutes for a configuration changes to apply.

Procedure

1. Create a file, such as sriov-node-policy.yaml, with content like the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: sriov-node-policy 1
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: netdevice 2
     eSwitchMode: "switchdev" 3
     nicSelector:
       deviceID: "1019"
       rootDevices:
       - 0000:d8:00.0
       vendor: "15b3"
       pfNames:
       - ens8f0
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     numVfs: 6
     priority: 5
     resourceName: mlxnics

   1

   The name for the custom resource object.

   2

   Required. Hardware offloading is not supported with vfio-pci.

   3

   Required.

2. Apply the configuration for the policy:

   $ oc create -f sriov-node-policy.yaml

   Note

   When you apply the configuration specified in a SriovNetworkPoolConfig object, the SR-IOV Operator drains and restarts the nodes in the machine config pool.

   It might take several minutes for a configuration change to apply.

Procedure

1. Add the network.operator.openshift.io/smart-nic label to each worker node with a SmartNIC present by running the following command:

   $ oc label node <node-name> network.operator.openshift.io/smart-nic=

   Use the oc get nodes command to get a list of the available nodes.

2. Create a policy named sriov-node-mgmt-vf-policy.yaml for the management port with content such as the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: sriov-node-mgmt-vf-policy
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: netdevice
     eSwitchMode: "switchdev"
     nicSelector:
       deviceID: "1019"
       rootDevices:
       - 0000:d8:00.0
       vendor: "15b3"
       pfNames:
       - ens8f0#0-0 1
     nodeSelector:
       network.operator.openshift.io/smart-nic: ""
     numVfs: 6 2
     priority: 5
     resourceName: mgmtvf

   1

   Replace this device with the appropriate network device for your use case. The #0-0 part of the pfNames value reserves a single virtual function used by OVN-Kubernetes.

   2

   The value provided here is an example. Replace this value with one that meets your requirements. For more information, see SR-IOV network node configuration object in the Additional resources section.

3. Create a policy named sriov-node-policy.yaml with content such as the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: sriov-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: netdevice
     eSwitchMode: "switchdev"
     nicSelector:
       deviceID: "1019"
       rootDevices:
       - 0000:d8:00.0
       vendor: "15b3"
       pfNames:
       - ens8f0#1-5 1
     nodeSelector:
       network.operator.openshift.io/smart-nic: ""
     numVfs: 6 2
     priority: 5
     resourceName: mlxnics

   1

   Replace this device with the appropriate network device for your use case.

   2

   The value provided here is an example. Replace this value with the value specified in the sriov-node-mgmt-vf-policy.yaml file. For more information, see SR-IOV network node configuration object in the Additional resources section.

   Note

   The sriov-node-mgmt-vf-policy.yaml file has different values for the pfNames and resourceName keys than the sriov-node-policy.yaml file.

4. Apply the configuration for both policies:

   $ oc create -f sriov-node-policy.yaml

   $ oc create -f sriov-node-mgmt-vf-policy.yaml

5. Create a Cluster Network Operator (CNO) ConfigMap in the cluster for the management configuration:

   1. Create a ConfigMap named hardware-offload-config.yaml with the following contents:

      apiVersion: v1
      kind: ConfigMap
      metadata:
          name: hardware-offload-config
          namespace: openshift-network-operator
      data:
          mgmt-port-resource-name: openshift.io/mgmtvf

   2. Apply the configuration for the ConfigMap:

      $ oc create -f hardware-offload-config.yaml

Procedure

1. Create a file, such as net-attach-def.yaml, with content like the following example:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: net-attach-def 1
     namespace: net-attach-def 2
     annotations:
       k8s.v1.cni.cncf.io/resourceName: openshift.io/mlxnics 3
   spec:
     config: '{"cniVersion":"0.3.1","name":"ovn-kubernetes","type":"ovn-k8s-cni-overlay","ipam":{},"dns":{}}'

   1

   The name for your network attachment definition.

   2

   The namespace for your network attachment definition.

   3

   This is the value of the spec.resourceName field you specified in the SriovNetworkNodePolicy object.

2. Apply the configuration for the network attachment definition:

   $ oc create -f net-attach-def.yaml

Procedure

1. Add the following labels to each of your worker nodes by entering the following commands:

   $ oc label node <example_node_name_one> node-role.kubernetes.io/sriov=

   $ oc label node <example_node_name_two> node-role.kubernetes.io/sriov=

2. Create a machine config pool for the SR-IOV Network Operator, for example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: sriov
   spec:
     machineConfigSelector:
       matchExpressions:
       - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,sriov]}
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/sriov: ""

3. Apply the following machineconfig.yaml file to the worker nodes:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: sriov
     name: 99-bf2-dpu
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,ZmluZF9jb250YWluZXIoKSB7CiAgY3JpY3RsIHBzIC1vIGpzb24gfCBqcSAtciAnLmNvbnRhaW5lcnNbXSB8IHNlbGVjdCgubWV0YWRhdGEubmFtZT09InNyaW92LW5ldHdvcmstY29uZmlnLWRhZW1vbiIpIHwgLmlkJwp9CnVudGlsIG91dHB1dD0kKGZpbmRfY29udGFpbmVyKTsgW1sgLW4gIiRvdXRwdXQiIF1dOyBkbwogIGVjaG8gIndhaXRpbmcgZm9yIGNvbnRhaW5lciB0byBjb21lIHVwIgogIHNsZWVwIDE7CmRvbmUKISBzdWRvIGNyaWN0bCBleGVjICRvdXRwdXQgL2JpbmRhdGEvc2NyaXB0cy9iZjItc3dpdGNoLW1vZGUuc2ggIiRAIgo=
           mode: 0755
           overwrite: true
           path: /etc/default/switch_in_sriov_config_daemon.sh
       systemd:
         units:
         - name: dpu-switch.service
           enabled: true
           contents: |
             [Unit]
             Description=Switch BlueField2 card to NIC/DPU mode
             RequiresMountsFor=%t/containers
             Wants=network.target
             After=network-online.target kubelet.service
             [Service]
             SuccessExitStatus=0 120
             RemainAfterExit=True
             ExecStart=/bin/bash -c '/etc/default/switch_in_sriov_config_daemon.sh nic || shutdown -r now' 1
             Type=oneshot
             [Install]
             WantedBy=multi-user.target

   1

   Optional: The PCI address of a specific card can optionally be specified, for example ExecStart=/bin/bash -c '/etc/default/switch_in_sriov_config_daemon.sh nic 0000:5e:00.0 || echo done'. By default, the first device is selected. If there is more than one device, you must specify which PCI address to be used. The PCI address must be the same on all nodes that are switching Bluefield-2 from DPU mode to NIC mode.

4. Wait for the worker nodes to restart. After restarting, the Bluefield-2 network device on the worker nodes is switched into NIC mode.
5. Optional: You might need to restart the host hardware because most recent Bluefield-2 firmware releases require a hardware restart to switch into NIC mode.

Procedure

1. Delete all SR-IOV custom resources (CRs):

   $ oc delete sriovnetwork -n openshift-sriov-network-operator --all

   $ oc delete sriovnetworknodepolicy -n openshift-sriov-network-operator --all

   $ oc delete sriovibnetwork -n openshift-sriov-network-operator --all

2. Follow the instructions in the "Deleting Operators from a cluster" section to remove the SR-IOV Network Operator from your cluster.

3. Delete the SR-IOV custom resource definitions that remain in the cluster after the SR-IOV Network Operator is uninstalled:

   $ oc delete crd sriovibnetworks.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworknodepolicies.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworknodestates.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworkpoolconfigs.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworks.sriovnetwork.openshift.io

   $ oc delete crd sriovoperatorconfigs.sriovnetwork.openshift.io

4. Delete the SR-IOV webhooks:

   $ oc delete mutatingwebhookconfigurations network-resources-injector-config

   $ oc delete MutatingWebhookConfiguration sriov-operator-webhook-config

   $ oc delete ValidatingWebhookConfiguration sriov-operator-webhook-config

5. Delete the SR-IOV Network Operator namespace:

   $ oc delete namespace openshift-sriov-network-operator