Chapter 3. Configuring an SR-IOV Ethernet network attachment

Procedure

1. Create a SriovNetwork object, and then save the YAML in the <name>.yaml file, where <name> is a name for this additional network. The object specification might resemble the following example:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: attach1
     namespace: openshift-sriov-network-operator
   spec:
     resourceName: net1
     networkNamespace: project2
     ipam: |-
       {
         "type": "host-local",
         "subnet": "10.56.217.0/24",
         "rangeStart": "10.56.217.171",
         "rangeEnd": "10.56.217.181",
         "gateway": "10.56.217.1"
       }

   Copy to Clipboard Toggle word wrap

2. To create the object, enter the following command:

   $ oc create -f <name>.yaml

   Copy to Clipboard Toggle word wrap

   where <name> specifies the name of the additional network.

3. Optional: To confirm that the NetworkAttachmentDefinition object that is associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the networkNamespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Add an annotation to the Pod object. Only one of the following annotation formats can be used:

   1. To attach a secondary network without any customization, add an annotation with the following format. Replace <network> with the name of the secondary network to associate with the pod:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: <network>[,<network>,...] 

      1

      Copy to Clipboard Toggle word wrap

      1

      To specify more than one secondary network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same secondary network multiple times, that pod will have multiple network interfaces attached to that network.

   2. To attach a secondary network with customizations, add an annotation with the following format:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "<network>", 

      1

      
                "namespace": "<namespace>", 

      2

      
                "default-route": ["<default-route>"] 

      3

      
              }
            ]

      Copy to Clipboard Toggle word wrap

      1

      Specify the name of the secondary network defined by a NetworkAttachmentDefinition object.

      2

      Specify the namespace where the NetworkAttachmentDefinition object is defined.

      3

      Optional: Specify an override for the default route, such as 192.168.17.1.

2. To create the pod, enter the following command. Replace <name> with the name of the pod.

   $ oc create -f <name>.yaml

   Copy to Clipboard Toggle word wrap

3. Optional: To Confirm that the annotation exists in the Pod CR, enter the following command, replacing <name> with the name of the pod.

   $ oc get pod <name> -o yaml

   Copy to Clipboard Toggle word wrap

   In the following example, the example-pod pod is attached to the net1 secondary network:

   $ oc get pod example-pod -o yaml
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-bridge
       k8s.v1.cni.cncf.io/network-status: |- 

   1

   
         [{
             "name": "ovn-kubernetes",
             "interface": "eth0",
             "ips": [
                 "10.128.2.14"
             ],
             "default": true,
             "dns": {}
         },{
             "name": "macvlan-bridge",
             "interface": "net1",
             "ips": [
                 "20.2.2.100"
             ],
             "mac": "22:2f:60:a5:f8:00",
             "dns": {}
         }]
     name: example-pod
     namespace: default
   spec:
     ...
   status:
     ...

   Copy to Clipboard Toggle word wrap

   1

   The k8s.v1.cni.cncf.io/network-status parameter is a JSON array of objects. Each object describes the status of a secondary network attached to the pod. The annotation value is stored as a plain text value.

Procedure

1. Create a SriovNetworkPoolConfig resource:

   1. Create a YAML file that defines the SriovNetworkPoolConfig resource:

      Example sriov-nw-pool.yaml file

      apiVersion: v1
      kind: SriovNetworkPoolConfig
      metadata:
        name: pool-1 

      1

      
        namespace: openshift-sriov-network-operator 

      2

      
      spec:
        maxUnavailable: 2 

      3

      
        nodeSelector: 

      4

      
          matchLabels:
            node-role.kubernetes.io/worker: ""

      Copy to Clipboard Toggle word wrap

      1

      Specify the name of the SriovNetworkPoolConfig object.

      2

      Specify namespace where the SR-IOV Network Operator is installed.

      3

      Specify an integer number, or percentage value, for nodes that can be unavailable in the pool during an update. For example, if you have 10 nodes and you set the maximum unavailable to 2, then only 2 nodes can be drained in parallel at any time, leaving 8 nodes for handling workloads.

      4

      Specify the nodes to add the pool by using the node selector. This example adds all nodes with the worker role to the pool.

   2. Create the SriovNetworkPoolConfig resource by running the following command:

      $ oc create -f sriov-nw-pool.yaml

      Copy to Clipboard Toggle word wrap

2. Create the sriov-test namespace by running the following comand:

   $ oc create namespace sriov-test

   Copy to Clipboard Toggle word wrap

3. Create a SriovNetworkNodePolicy resource:

   1. Create a YAML file that defines the SriovNetworkNodePolicy resource:

      Example sriov-node-policy.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: sriov-nic-1
        namespace: openshift-sriov-network-operator
      spec:
        deviceType: netdevice
        nicSelector:
          pfNames: ["ens1"]
        nodeSelector:
          node-role.kubernetes.io/worker: ""
        numVfs: 5
        priority: 99
        resourceName: sriov_nic_1

      Copy to Clipboard Toggle word wrap

   2. Create the SriovNetworkNodePolicy resource by running the following command:

      $ oc create -f sriov-node-policy.yaml

      Copy to Clipboard Toggle word wrap

4. Create a SriovNetwork resource:

   1. Create a YAML file that defines the SriovNetwork resource:

      Example sriov-network.yaml file

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: sriov-nic-1
        namespace: openshift-sriov-network-operator
      spec:
        linkState: auto
        networkNamespace: sriov-test
        resourceName: sriov_nic_1
        capabilities: '{ "mac": true, "ips": true }'
        ipam: '{ "type": "static" }'

      Copy to Clipboard Toggle word wrap

   2. Create the SriovNetwork resource by running the following command:

      $ oc create -f sriov-network.yaml

      Copy to Clipboard Toggle word wrap

1. Update the number of virtual functions in the SriovNetworkNodePolicy resource to trigger workload draining in the cluster:

   $ oc patch SriovNetworkNodePolicy sriov-nic-1 -n openshift-sriov-network-operator --type merge -p '{"spec": {"numVfs": 4}}'

   Copy to Clipboard Toggle word wrap

2. Monitor the draining status on the target cluster by running the following command:

   $ oc get sriovNetworkNodeState -n openshift-sriov-network-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE                          NAME       SYNC STATUS   DESIRED SYNC STATE   CURRENT SYNC STATE   AGE
   openshift-sriov-network-operator   worker-0   InProgress    Drain_Required       DrainComplete        3d10h
   openshift-sriov-network-operator   worker-1   InProgress    Drain_Required       DrainComplete        3d10h

   Copy to Clipboard Toggle word wrap

   When the draining process is complete, the SYNC STATUS changes to Succeeded, and the DESIRED SYNC STATE and CURRENT SYNC STATE values return to IDLE.

Procedure

1. Create the SriovNetworkNodePolicy CR:

   1. Save the following YAML in the sriov-network-node-policy.yaml file, replacing values in the YAML to match your environment:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: <policy_name>
        namespace: openshift-sriov-network-operator
      spec:
        resourceName: sriovnuma0 

      1

      
        nodeSelector:
          kubernetes.io/hostname: <node_name>
        numVfs: <number_of_Vfs>
        nicSelector: 

      2

      
          vendor: "<vendor_ID>"
          deviceID: "<device_ID>"
        deviceType: netdevice
        excludeTopology: true 

      3

      Copy to Clipboard Toggle word wrap

      1

      The resource name of the SR-IOV network device plugin. This YAML uses a sample resourceName value.

      2

      Identify the device for the Operator to configure by using the NIC selector.

      3

      To exclude advertising the NUMA node for the SR-IOV network resource to the Topology Manager, set the value to true. The default value is false.

      Note

      If multiple SriovNetworkNodePolicy resources target the same SR-IOV network resource, the SriovNetworkNodePolicy resources must have the same value as the excludeTopology specification. Otherwise, the conflicting policy is rejected.

   2. Create the SriovNetworkNodePolicy resource by running the following command. Successful output lists the name of the SriovNetworkNodePolicy resource and the created status.

      $ oc create -f sriov-network-node-policy.yaml

      Copy to Clipboard Toggle word wrap

2. Create the SriovNetwork CR:

   1. Save the following YAML in the sriov-network.yaml file, replacing values in the YAML to match your environment:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: sriov-numa-0-network 

      1

      
        namespace: openshift-sriov-network-operator
      spec:
        resourceName: sriovnuma0 

      2

      
        networkNamespace: <namespace> 

      3

      
        ipam: |- 

      4

      
          {
            "type": "<ipam_type>",
          }

      Copy to Clipboard Toggle word wrap

      1

      Replace sriov-numa-0-network with the name for the SR-IOV network resource.

      2

      Specify the resource name for the SriovNetworkNodePolicy CR from the previous step. This YAML uses a sample resourceName value.

      3

      Enter the namespace for your SR-IOV network resource.

      4

      Enter the IP address management configuration for the SR-IOV network.

   2. Create the SriovNetwork resource by running the following command. Successful output lists the name of the SriovNetwork resource and the created status.

      $ oc create -f sriov-network.yaml

      Copy to Clipboard Toggle word wrap

3. Create a pod and assign the SR-IOV network resource from the previous step:

   1. Save the following YAML in the sriov-network-pod.yaml file, replacing values in the YAML to match your environment:

      apiVersion: v1
      kind: Pod
      metadata:
        name: <pod_name>
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "sriov-numa-0-network", 

      1

      
              }
            ]
      spec:
        containers:
        - name: <container_name>
          image: <image>
          imagePullPolicy: IfNotPresent
          command: ["sleep", "infinity"]

      Copy to Clipboard Toggle word wrap

      1

      This is the name of the SriovNetwork resource that uses the SriovNetworkNodePolicy resource.

   2. Create the Pod resource by running the following command. The expected output shows the name of the Pod resource and the created status.

      $ oc create -f sriov-network-pod.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. Verify the status of the pod by running the following command, replacing <pod_name> with the name of the pod:

   $ oc get pod <pod_name>

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                     READY   STATUS    RESTARTS   AGE
   test-deployment-sriov-76cbbf4756-k9v72   1/1     Running   0          45h

   Copy to Clipboard Toggle word wrap

2. Open a debug session with the target pod to verify that the SR-IOV network resources are deployed to a different node than the memory and CPU resources.

   1. Open a debug session with the pod by running the following command, replacing <pod_name> with the target pod name.

      $ oc debug pod/<pod_name>

      Copy to Clipboard Toggle word wrap

   2. Set /host as the root directory within the debug shell. The debug pod mounts the root file system from the host in /host within the pod. By changing the root directory to /host, you can run binaries from the host file system:

      $ chroot /host

      Copy to Clipboard Toggle word wrap

   3. View information about the CPU allocation by running the following commands:

      $ lscpu | grep NUMA

      Copy to Clipboard Toggle word wrap

      Example output

      NUMA node(s):                    2
      NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,...
      NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,...

      Copy to Clipboard Toggle word wrap

      $ cat /proc/self/status | grep Cpus

      Copy to Clipboard Toggle word wrap

      Example output

      Cpus_allowed:	aa
      Cpus_allowed_list:	1,3,5,7

      Copy to Clipboard Toggle word wrap

      $ cat  /sys/class/net/net1/device/numa_node

      Copy to Clipboard Toggle word wrap

      Example output

      0

      Copy to Clipboard Toggle word wrap

      In this example, CPUs 1,3,5, and 7 are allocated to NUMA node1 but the SR-IOV network resource can use the NIC in NUMA node0.