Chapter 14. Hardware networks

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.
2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".

3. Create the SriovNetworkNodePolicy object:

   $ oc create -f <name>-sriov-node-network.yaml

   where <name> specifies the name for this configuration.

   After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Procedure

1. Create a SriovNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: attach1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: net1
     networkNamespace: project2
     ipam: |-
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "gateway": "10.56.217.1"
       }

2. To create the object, enter the following command:

   $ oc create -f <name>.yaml

   where <name> specifies the name of the additional network.

3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Create a SriovIBNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovIBNetwork
   metadata:
     name: attach1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: net1
     networkNamespace: project2
     ipam: |-
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "gateway": "10.56.217.1"
       }

2. To create the object, enter the following command:

   $ oc create -f <name>.yaml

   where <name> specifies the name of the additional network.

3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovIBNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovIBNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Add an annotation to the Pod object. Only one of the following annotation formats can be used:

   1. To attach an additional network without any customization, add an annotation with the following format. Replace <network> with the name of the additional network to associate with the pod:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: <network>[,<network>,...] 1

      1

      To specify more than one additional network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same additional network multiple times, that pod will have multiple network interfaces attached to that network.

   2. To attach an additional network with customizations, add an annotation with the following format:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "<network>", 1
                "namespace": "<namespace>", 2
                "default-route": ["<default-route>"] 3
              }
            ]

      1

      Specify the name of the additional network defined by a NetworkAttachmentDefinition object.

      2

      Specify the namespace where the NetworkAttachmentDefinition object is defined.

      3

      Optional: Specify an override for the default route, such as 192.168.17.1.

2. To create the pod, enter the following command. Replace <name> with the name of the pod.

   $ oc create -f <name>.yaml

3. Optional: To Confirm that the annotation exists in the Pod CR, enter the following command, replacing <name> with the name of the pod.

   $ oc get pod <name> -o yaml

   In the following example, the example-pod pod is attached to the net1 additional network:

   $ oc get pod example-pod -o yaml
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-bridge
       k8s.v1.cni.cncf.io/networks-status: |- 1
         [{
             "name": "openshift-sdn",
             "interface": "eth0",
             "ips": [
                 "10.128.2.14"
             ],
             "default": true,
             "dns": {}
         },{
             "name": "macvlan-bridge",
             "interface": "net1",
             "ips": [
                 "20.2.2.100"
             ],
             "mac": "22:2f:60:a5:f8:00",
             "dns": {}
         }]
     name: example-pod
     namespace: default
   spec:
     ...
   status:
     ...

   1

   The k8s.v1.cni.cncf.io/networks-status parameter is a JSON array of objects. Each object describes the status of an additional network attached to the pod. The annotation value is stored as a plain text value.

Procedure

1. Create the following SR-IOV pod spec, and then save the YAML in the <name>-sriov-pod.yaml file. Replace <name> with a name for this pod.

   The following example shows an SR-IOV pod spec:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sample-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: <name> 1
   spec:
     containers:
     - name: sample-container
       image: <image> 2
       command: ["sleep", "infinity"]
       resources:
         limits:
           memory: "1Gi" 3
           cpu: "2" 4
         requests:
           memory: "1Gi"
           cpu: "2"

   1

   Replace <name> with the name of the SR-IOV network attachment definition CR.

   2

   Replace <image> with the name of the sample-pod image.

   3

   To create the SR-IOV pod with guaranteed QoS, set memory limits equal to memory requests.

   4

   To create the SR-IOV pod with guaranteed QoS, set cpu limits equals to cpu requests.

2. Create the sample SR-IOV pod by running the following command:

   $ oc create -f <filename> 1

   1

   Replace <filename> with the name of the file you created in the previous step.

3. Confirm that the sample-pod is configured with guaranteed QoS.

   $ oc describe pod sample-pod

4. Confirm that the sample-pod is allocated with exclusive CPUs.

   $ oc exec sample-pod -- cat /sys/fs/cgroup/cpuset/cpuset.cpus

5. Confirm that the SR-IOV device and CPUs that are allocated for the sample-pod are on the same NUMA node.

   $ oc exec sample-pod -- cat /sys/fs/cgroup/cpuset/cpuset.cpus

Procedure

1. Create a SriovNetworkNodePolicy object:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: policy-example
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: example
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     numVfs: 4
     nicSelector:
       vendor: "8086"
       pfNames: ['ens803f0']
       rootDevices: ['0000:86:00.0']

2. Create a SriovNetwork object:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: net-example
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: default
     ipam: | 1
       {
         "type": "host-local", 2
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "routes": [
           {"dst": "224.0.0.0/5"},
           {"dst": "232.0.0.0/5"}
         ],
         "gateway": "10.56.217.1"
       }
     resourceName: example

   1 2

   If you choose to configure DHCP as IPAM, ensure that you provision the following default routes through your DHCP server: 224.0.0.0/5 and 232.0.0.0/5. This is to override the static multicast route set by the default network provider.

3. Create a pod with multicast application:

   apiVersion: v1
   kind: Pod
   metadata:
     name: testpmd
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: nic1
   spec:
     containers:
     - name: example
       image: rhel7:latest
       securityContext:
         capabilities:
           add: ["NET_ADMIN"] 1
       command: [ "sleep", "infinity"]

   1

   The NET_ADMIN capability is required only if your application needs to assign the multicast IP address to the SR-IOV interface. Otherwise, it can be omitted.

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the intel-dpdk-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: intel-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: intelnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "8086"
       deviceID: "158b"
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: vfio-pci 1

   1

   Specify the driver type for the virtual functions to vfio-pci.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f intel-dpdk-node-policy.yaml

3. Create the following SriovNetwork object, and then save the YAML in the intel-dpdk-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: intel-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: "{}" 1
     vlan: <vlan>
     resourceName: intelnics

   1

   Specify an empty object "{}" for the ipam CNI plugin. DPDK works in userspace mode and does not require an IP address.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

4. Create the SriovNetwork object by running the following command:

   $ oc create -f intel-dpdk-network.yaml

5. Create the following Pod spec, and then save the YAML in the intel-dpdk-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: intel-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /dev/hugepages 4
         name: hugepage
       resources:
         limits:
           openshift.io/intelnics: "1" 5
           memory: "1Gi"
           cpu: "4" 6
           hugepages-1Gi: "4Gi" 7
         requests:
           openshift.io/intelnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where the SriovNetwork object intel-dpdk-network is created. If you would like to create the pod in a different namespace, change target_namespace in both the Pod spec and the SriovNetowrk object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount a hugepage volume to the DPDK pod under /dev/hugepages. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated to DPDK pod. This resource request and limit, if not explicitly specified, will be automatically added by the SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by the SR-IOV Operator. It is enabled by default and can be disabled by setting enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs to be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes. For example, adding kernel arguments default_hugepagesz=1GB, hugepagesz=1G and hugepages=16 will result in 16*1Gi hugepages be allocated during system boot.

6. Create the DPDK pod by running the following command:

   $ oc create -f intel-dpdk-pod.yaml

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-dpdk-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-dpdk-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 1
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 2
     isRdma: true 3

   1

   Specify the device hex code of the SR-IOV network device. The only allowed values for Mellanox cards are 1015, 1017.

   2

   Specify the driver type for the virtual functions to netdevice. Mellanox SR-IOV VF can work in DPDK mode without using the vfio-pci device type. VF device appears as a kernel network interface inside a container.

   3

   Enable RDMA mode. This is required by Mellanox cards to work in DPDK mode.

   Note

   See the Configuring SR-IOV network devices section for detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-dpdk-node-policy.yaml

3. Create the following SriovNetwork object, and then save the YAML in the mlx-dpdk-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-dpdk-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 1
       ...
     vlan: <vlan>
     resourceName: mlxnics

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-dpdk-network.yaml

5. Create the following Pod spec, and then save the YAML in the mlx-dpdk-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: dpdk-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-dpdk-network
   spec:
     containers:
     - name: testpmd
       image: <DPDK_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /dev/hugepages 4
         name: hugepage
       resources:
         limits:
           openshift.io/mlxnics: "1" 5
           memory: "1Gi"
           cpu: "4" 6
           hugepages-1Gi: "4Gi" 7
         requests:
           openshift.io/mlxnics: "1"
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where SriovNetwork object mlx-dpdk-network is created. If you would like to create the pod in a different namespace, change target_namespace in both Pod spec and SriovNetowrk object.

   2

   Specify the DPDK image which includes your application and the DPDK library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to the DPDK pod under /dev/hugepages. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Optional: Specify the number of DPDK devices allocated to the DPDK pod. This resource request and limit, if not explicitly specified, will be automatically added by SR-IOV network resource injector. The SR-IOV network resource injector is an admission controller component managed by SR-IOV Operator. It is enabled by default and can be disabled by setting the enableInjector option to false in the default SriovOperatorConfig CR.

   6

   Specify the number of CPUs. The DPDK pod usually requires exclusive CPUs be allocated from kubelet. This is achieved by setting CPU Manager policy to static and creating a pod with Guaranteed QoS.

   7

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to DPDK pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes.

6. Create the DPDK pod by running the following command:

   $ oc create -f mlx-dpdk-pod.yaml

Procedure

1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-rdma-node-policy.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: mlx-rdma-node-policy
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: mlxnics
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true"
     priority: <priority>
     numVfs: <num>
     nicSelector:
       vendor: "15b3"
       deviceID: "1015" 1
       pfNames: ["<pf_name>", ...]
       rootDevices: ["<pci_bus_id>", "..."]
     deviceType: netdevice 2
     isRdma: true 3

   1

   Specify the device hex code of SR-IOV network device. The only allowed values for Mellanox cards are 1015, 1017.

   2

   Specify the driver type for the virtual functions to netdevice.

   3

   Enable RDMA mode.

   Note

   See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

   When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

   After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

2. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-node-policy.yaml

3. Create the following SriovNetwork object, and then save the YAML in the mlx-rdma-network.yaml file.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: mlx-rdma-network
     namespace: openshift-sriov-network-operator
   spec:
     networkNamespace: <target_namespace>
     ipam: |- 1
       ...
     vlan: <vlan>
     resourceName: mlxnics

   1

   Specify a configuration object for the ipam CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.

   Note

   See the "Configuring SR-IOV additional network" section for a detailed explanation on each option in SriovNetwork.

4. Create the SriovNetworkNodePolicy object by running the following command:

   $ oc create -f mlx-rdma-network.yaml

5. Create the following Pod spec, and then save the YAML in the mlx-rdma-pod.yaml file.

   apiVersion: v1
   kind: Pod
   metadata:
     name: rdma-app
     namespace: <target_namespace> 1
     annotations:
       k8s.v1.cni.cncf.io/networks: mlx-rdma-network
   spec:
     containers:
     - name: testpmd
       image: <RDMA_image> 2
       securityContext:
         runAsUser: 0
         capabilities:
           add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] 3
       volumeMounts:
       - mountPath: /dev/hugepages 4
         name: hugepage
       resources:
         limits:
           memory: "1Gi"
           cpu: "4" 5
           hugepages-1Gi: "4Gi" 6
         requests:
           memory: "1Gi"
           cpu: "4"
           hugepages-1Gi: "4Gi"
       command: ["sleep", "infinity"]
     volumes:
     - name: hugepage
       emptyDir:
         medium: HugePages

   1

   Specify the same target_namespace where SriovNetwork object mlx-rdma-network is created. If you would like to create the pod in a different namespace, change target_namespace in both Pod spec and SriovNetowrk object.

   2

   Specify the RDMA image which includes your application and RDMA library used by application.

   3

   Specify additional capabilities required by the application inside the container for hugepage allocation, system resource allocation, and network interface access.

   4

   Mount the hugepage volume to RDMA pod under /dev/hugepages. The hugepage volume is backed by the emptyDir volume type with the medium being Hugepages.

   5

   Specify number of CPUs. The RDMA pod usually requires exclusive CPUs be allocated from the kubelet. This is achieved by setting CPU Manager policy to static and create pod with Guaranteed QoS.

   6

   Specify hugepage size hugepages-1Gi or hugepages-2Mi and the quantity of hugepages that will be allocated to the RDMA pod. Configure 2Mi and 1Gi hugepages separately. Configuring 1Gi hugepage requires adding kernel arguments to Nodes.

6. Create the RDMA pod by running the following command:

   $ oc create -f mlx-rdma-pod.yaml

Procedure

1. Delete all SR-IOV custom resources (CRs):

   $ oc delete sriovnetwork -n openshift-sriov-network-operator --all

   $ oc delete sriovnetworknodepolicy -n openshift-sriov-network-operator --all

   $ oc delete sriovibnetwork -n openshift-sriov-network-operator --all

2. Follow the instructions in the "Deleting Operators from a cluster" section to remove the SR-IOV Network Operator from your cluster.

3. Delete the SR-IOV custom resource definitions that remain in the cluster after the SR-IOV Network Operator is uninstalled:

   $ oc delete crd sriovibnetworks.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworknodepolicies.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworknodestates.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworkpoolconfigs.sriovnetwork.openshift.io

   $ oc delete crd sriovnetworks.sriovnetwork.openshift.io

   $ oc delete crd sriovoperatorconfigs.sriovnetwork.openshift.io

4. Delete the SR-IOV webhooks:

   $ oc delete mutatingwebhookconfigurations network-resources-injector-config

   $ oc delete MutatingWebhookConfiguration sriov-operator-webhook-config

   $ oc delete ValidatingWebhookConfiguration sriov-operator-webhook-config

5. Delete the SR-IOV Network Operator namespace:

   $ oc delete namespace openshift-sriov-network-operator