Chapter 4. Using the SR-IOV Network Operator

Procedure

1. If provided as an optional RPM, install the required microshift-sriov RPM package.
2. Restart the MicroShift service to deploy the SR-IOV resources in the sriov-network-operator namespace.

3. To specify the required VF configuration based on the available hardware, create an SriovNetworkNodePolicy custom resource (CR). For example, save the following YAML as the file policyoneflag-sriov-node-network.yaml:

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: policyoneflag
     namespace: sriov-network-operator
   spec:
     resourceName: policyoneflag
     nodeSelector:
       node.kubernetes.io/instance-type: rhde
     priority: 10
     numVfs: 5
     nicSelector:
       pfNames: ["ens5"]
     deviceType: "netdevice"
     isRdma: false

   Copy to Clipboard Toggle word wrap
   • The metadata.name parameter specifies the name for the custom resource object.
   • The metadata.namespace parameter specifies the namespace where the SR-IOV Network Operator is installed.
   • The spec.resourceName parameter specifies the resource name of the SR-IOV network device plugin. You can create multiple SR-IOV network node policies for a resource name.
   • (Optional) The spec.priority parameter specifies a priority value. The priority is an integer value between 0 and 99. A smaller value receives higher priority. For example, a priority of 10 is a higher priority than 99. The default value is 99.
   • The spec.numVfs parameter specifies the number of the virtual functions (VFs) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 127.
   • The spec.nicSelector parameter identifies the device for the Operator to configure. You do not have to specify values for all the parameters. It is recommended to identify the network device with enough precision to avoid selecting a device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they refer to the same device. If you specify a value for netFilter, then you do not need to specify any other parameter because a network ID is unique.
   • (Optional) The spec.nicSelector.pfNames parameter specifies an array of one or more physical function (PF) names for the device.
   • (Optional) The spec.deviceType parameter specifies the driver type for the virtual functions. The only allowed value is netdevice.

   • (Optional) The spec.isRDMA parameter configures whether to enable remote direct memory access (RDMA) mode. The default value is false. If the spec.isRdma parameter is set to true, you can continue to use the RDMA-enabled VF as a normal network device. A device can be used in either mode. to configure a Mellanox NIC for use with Fast Datapath DPDK applications, set spec.isRdma to true and additionally set the needVhostNet parameter to true

     Note

     The vfio-pci driver type is not supported.

4. Create the SriovNetworkNodePolicy object by entering the following command:

   $ oc create -f policyoneflag-sriov-node-network.yaml

   Copy to Clipboard Toggle word wrap

   After applying the configuration update, the workload contains the required resources and dependencies for VF access.

5. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured. Expected output shows Succeeded.

   $ oc get sriovnetworknodestates -n sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

   Copy to Clipboard Toggle word wrap

6. Deploy an SriovNetwork custom resource (CR) which references the SriovNetworkNodePolicy CR and insert the metaPlugins configuration, as in the following example CR. The Operator generates a NetworkAttachmentDefinition CR and the VFs become available to the pods. Save the YAML as the file sriov-network-interface-sysctl.yaml.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: onevalidflag
     namespace: sriov-network-operator
   spec:
     resourceName: policyoneflag
     networkNamespace: sysctl-tuning-test
     ipam: '{ "type": "static" }'
     capabilities: '{ "mac": true, "ips": true }'
     metaPlugins : |
       {
         "type": "tuning",
         "capabilities":{
           "mac":true
         },
         "sysctl":{
            "net.ipv4.conf.IFNAME.accept_redirects": "1"
         }
       }

   Copy to Clipboard Toggle word wrap
   • The metadata.name parameter specifies the name for the object. The SR-IOV Network Operator creates a NetworkAttachmentDefinition object with the same name.
   • The metadata.namespace parameter specifies the namespace where the SR-IOV Network Operator is installed.
   • The spec.resourceName parameter displays the value from the SriovNetworkNodePolicy object that defines the SR-IOV hardware for this additional network.
   • The spec.networkNamespace parameter specifies the target namespace for the SriovNetwork object. Only pods in the target namespace can attach to the additional network.
   • The spec.ipam parameter specifies a configuration object for the IPAM CNI plugin as a YAML block scalar. The plugin manages IP address assignment for the attachment definition.
   • (Optional) The spec.capabilities parameter sets capabilities for the additional network. You can specify "{ "ips": true }" to enable IP address support or "{ "mac": true }" to enable MAC address support.
   • (Optional) The spec.metaPlugins parameter is used to add additional capabilities to the device. In this use case set the type field to tuning. Specify the interface-level network sysctl you want to set in the sysctl field.

7. Create the SriovNetwork resource by entering the following command:

   $ oc create -f sriov-network-interface-sysctl.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Confirm that the SR-IOV Network Operator created the NetworkAttachmentDefinition CR by running the following command:

   $ oc get network-attachment-definitions -n <namespace>

   Copy to Clipboard Toggle word wrap

   Replace <namespace> with the value for networkNamespace parameter that you specified in the SriovNetwork object, for example, sysctl-tuning-test. The expected output shows the name of the NAD CRD and the creation age in minutes.

   Note

   There might be a delay before the SR-IOV Network Operator creates the CR.

2. Verify that the tuning CNI is correctly configured and the additional SR-IOV network is attached:

   1. Create a Pod CR. Save the following YAML as the file examplepod.yaml:

      apiVersion: v1
      kind: Pod
      metadata:
        name: tunepod
        namespace: sysctl-tuning-test
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "onevalidflag",
                "mac": "0a:56:0a:83:04:0c",
                "ips": ["10.100.100.200/24"]
             }
            ]
      spec:
        containers:
        - name: podexample
          image: centos
          command: ["/bin/bash", "-c", "sleep INF"]
          securityContext:
            runAsUser: 2000
            runAsGroup: 3000
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault

      Copy to Clipboard Toggle word wrap
      • The annotations.name parameter specifies the name of the SR-IOV network attachment definition CR.
      • (Optional) The annotations.mac parameter specifies the MAC address for the SR-IOV device that is allocated from the resource type defined in the SR-IOV network attachment definition CR. To use this feature, you also must specify { "mac": true } in the SriovNetwork object.
      • (Optional) The annotations.ips parameter specifies the IP addresses for the SR-IOV device that are allocated from the resource type defined in the SR-IOV network attachment definition CR. Both IPv4 and IPv6 addresses are supported. To use this feature, you also must specify { "ips": true } in the SriovNetwork object.

   2. Create the Pod CR by entering the following command:

      $ oc apply -f examplepod.yaml

      Copy to Clipboard Toggle word wrap

   3. Verify that the pod is created by running the following command:

      $ oc get pod -n sysctl-tuning-test

      Copy to Clipboard Toggle word wrap

      Example output:

      NAME      READY   STATUS    RESTARTS   AGE
      tunepod   1/1     Running   0          47s

      Copy to Clipboard Toggle word wrap

   4. Log in to the pod by running the following command:

      $ oc rsh -n sysctl-tuning-test tunepod

      Copy to Clipboard Toggle word wrap

   5. Verify the values of the configured sysctl flag. Find the value net.ipv4.conf.IFNAME.accept_redirects by running the following command:

      $ sysctl net.ipv4.conf.net1.accept_redirects

      Copy to Clipboard Toggle word wrap