Chapter 3. Secondary networks

Procedure

1. Create a new YAML file named, such as setsebool-container-use-devices.yaml, with the following details:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-setsebool
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - enabled: true
           name: setsebool.service
           contents: |
             [Unit]
             Description=Set SELinux boolean for the TAP CNI plugin
             Before=kubelet.service
   
             [Service]
             Type=oneshot
             ExecStart=/usr/sbin/setsebool container_use_devices=on
             RemainAfterExit=true
   
             [Install]
             WantedBy=multi-user.target graphical.target

   Copy to Clipboard Toggle word wrap

2. Create the new MachineConfig object by running the following command:

   $ oc apply -f setsebool-container-use-devices.yaml

   Copy to Clipboard Toggle word wrap
   Note

   Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. This update can take some time to be applied.

3. Verify the change is applied by running the following command:

   $ oc get machineconfigpools

   Copy to Clipboard Toggle word wrap

   Expected output

   NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
   worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d

   Copy to Clipboard Toggle word wrap

   Note

   All nodes should be in the updated and ready state.

1. Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit.

   $ oc edit pod <name>

   Copy to Clipboard Toggle word wrap

2. In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties.

   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]'
   # ...

   Copy to Clipboard Toggle word wrap

   where:

   <network>

   Replace with a JSON object as shown in the following examples. The single quotes are required.

   In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter.

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
       {
         "name": "net1"
       },
       {
         "name": "net2", 

   1

   
         "default-route": ["192.0.2.1"] 

   2

   
       }]'
   spec:
     containers:
     - name: example-pod
       command: ["/bin/bash", "-c", "sleep 2000000000000"]
       image: centos/tools

   Copy to Clipboard Toggle word wrap

   where:

   name

   The name key is the name of the secondary network to associate with the pod.

   default-route

   The default-route key specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

   The default route will cause any traffic that is not specified in other routes to be routed to the gateway.

   Important

   Setting the default route to an interface other than the default network interface for OpenShift Container Platform may cause traffic that is anticipated for pod-to-pod traffic to be routed over another interface.

   To verify the routing properties of a pod, the oc command may be used to execute the ip command within a pod.

   $ oc exec -it <pod_name> -- ip route

   Copy to Clipboard Toggle word wrap
   Note

   You may also reference the pod’s k8s.v1.cni.cncf.io/network-status to see which secondary network has been assigned the default route, by the presence of the default-route key in the JSON-formatted list of objects.

   To set a static IP address or MAC address for a pod you can use the JSON formatted annotations. This requires you create networks that specifically allow for this functionality. This can be specified in a rawCNIConfig for the CNO.

3. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

   Copy to Clipboard Toggle word wrap

   The following YAML describes the configuration parameters for the CNO:

   Cluster Network Operator YAML configuration

   name: <name> 

   1

   
   namespace: <namespace> 

   2

   
   rawCNIConfig: '{ 

   3

   
     ...
   }'
   type: Raw

   Copy to Clipboard Toggle word wrap

   where:

   name

   Specify a name for the secondary network attachment that you are creating. The name must be unique within the specified namespace.

   namespace

   Specify the namespace to create the network attachment in. If you do not specify a value, then the default namespace is used.

   rawCNIConfig

   Specify the CNI plugin configuration in JSON format, which is based on the following template.

   The following object describes the configuration parameters for utilizing static MAC address and IP address using the macvlan CNI plugin:

   macvlan CNI plugin JSON configuration object using static IP and MAC address

   {
     "cniVersion": "0.3.1",
     "name": "<name>", 

   1

   
     "plugins": [{ 

   2

   
         "type": "macvlan",
         "capabilities": { "ips": true }, 

   3

   
         "master": "eth0", 

   4

   
         "mode": "bridge",
         "ipam": {
           "type": "static"
         }
       }, {
         "capabilities": { "mac": true }, 

   5

   
         "type": "tuning"
       }]
   }

   Copy to Clipboard Toggle word wrap

   where:

   name

   Specifies the name for the secondary network attachment to create. The name must be unique within the specified namespace.

   plugins

   Specifies an array of CNI plugin configurations. The first object specifies a macvlan plugin configuration and the second object specifies a tuning plugin configuration.

   ips

   Specifies that a request is made to enable the static IP address functionality of the CNI plugin runtime configuration capabilities.

   master

   Specifies the interface that the macvlan plugin uses.

   mac

   Specifies that a request is made to enable the static MAC address functionality of a CNI plugin.

   The above network attachment can be referenced in a JSON formatted annotation, along with keys to specify which static IP and MAC address will be assigned to a given pod.

   Edit the pod with:

   $ oc edit pod <name>

   Copy to Clipboard Toggle word wrap

   macvlan CNI plugin JSON configuration object using static IP and MAC address

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "<name>", 

   1

   
           "ips": [ "192.0.2.205/24" ], 

   2

   
           "mac": "CA:FE:C0:FF:EE:00" 

   3

   
         }
       ]'

   Copy to Clipboard Toggle word wrap

   1 1 1 1

   Use the <name> as provided when creating the rawCNIConfig above.

   2 2 2 2

   Provide an IP address including the subnet mask.

   3 3 3

   Provide the MAC address.

   Note

   Static IP addresses and MAC addresses do not have to be used at the same time, you may use them individually, or together.

4. To verify the IP address and MAC properties of a pod with secondary networks, use the oc command to execute the ip command within a pod.

   $ oc exec -it <pod_name> -- ip a

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

   $ oc edit network.operator.openshift.io cluster

   Copy to Clipboard Toggle word wrap

2. Include the additionalNetworks section shown in this example YAML extract within the spec definition of the custom resource (CR):

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   # ...
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       rawCNIConfig: |-
         {
          "name": "whereabouts-shim",
          "cniVersion": "0.3.1",
          "type": "bridge",
          "ipam": {
            "type": "whereabouts"
          }
         }
       type: Raw
   # ...

   Copy to Clipboard Toggle word wrap
3. Save the file and exit the text editor.

4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Copy to Clipboard Toggle word wrap

   Example output

   pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
   pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
   daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

   Copy to Clipboard Toggle word wrap

Procedure

1. Run the following command to create a ConfigMap object named whereabouts-config in the openshift-multus namespace with a specific cron expression for the IP reconciler:

   $ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"

   Copy to Clipboard Toggle word wrap

   This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

   Note

   The whereabouts-reconciler daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

2. Retrieve information about resources related to the whereabouts-reconciler daemon set and pods within the openshift-multus namespace by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Copy to Clipboard Toggle word wrap

   Example output

   pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
   daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s

   Copy to Clipboard Toggle word wrap

3. Run the following command to verify that the whereabouts-reconciler pod runs the IP reconciler with the configured interval:

   $ oc -n openshift-multus logs whereabouts-reconciler-2p7hw

   Copy to Clipboard Toggle word wrap

   Example output

   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
   2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
   2024-02-02T16:45:00Z [verbose] starting reconciler run
   2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
   2024-02-02T16:45:00Z [debug] listing IP pools
   2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
   2024-02-02T16:45:00Z [verbose] reconciler success

   Copy to Clipboard Toggle word wrap

Procedure

1. Set type to whereabouts.

2. Use ipRanges to allocate IP addresses as shown in the following example:

   cniVersion: operator.openshift.io/v1
   kind: Network
   =metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       type: Raw
       rawCNIConfig: |-
         {
          "name": "whereabouts-dual-stack",
          "cniVersion": "0.3.1,
          "type": "bridge",
          "ipam": {
            "type": "whereabouts",
            "ipRanges": [
                     {"range": "192.168.10.0/24"},
                     {"range": "2001:db8::/64"}
                 ]
          }
         }

   Copy to Clipboard Toggle word wrap
3. Attach network to a pod. For more information, see "Adding a pod to a secondary network".
4. Verify that all IP addresses are assigned.

5. Run the following command to ensure the IP addresses are assigned as metadata.

   $ oc exec -it mypod -- ip a

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

   $ oc new-project test-namespace

   Copy to Clipboard Toggle word wrap

2. Create an SR-IOV node policy:

   1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" 

      1

      
         deviceID: "101b" 

      2

      
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      Copy to Clipboard Toggle word wrap
      Note

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1

      The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

      2

      The device hexadecimal code of the SR-IOV network device.

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-node-network-policy.yaml

      Copy to Clipboard Toggle word wrap
      Note

      Applying this might take some time due to the node requiring a reboot.

3. Create an SR-IOV network:

   1. Create the SriovNetwork custom resource (CR) for the additional secondary SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"

      Copy to Clipboard Toggle word wrap

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-network-attachment.yaml

      Copy to Clipboard Toggle word wrap

4. Create the VLAN secondary network:

   1. Using the following YAML example, create a file named vlan100-additional-network-configuration.yaml:

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: vlan-100
        namespace: test-namespace
      spec:
        config: |
          {
            "cniVersion": "0.4.0",
            "name": "vlan-100",
            "plugins": [
              {
                "type": "vlan",
                "master": "ext0", 

      1

      
                "mtu": 1500,
                "vlanId": 100,
                "linkInContainer": true, 

      2

      
                "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
              }
            ]
          }

      Copy to Clipboard Toggle word wrap

      1

      The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.

      2

      The linkInContainer parameter must be specified.

   2. Apply the YAML file by running the following command:

      $ oc apply -f vlan100-additional-network-configuration.yaml

      Copy to Clipboard Toggle word wrap

5. Create a pod definition by using the earlier specified networks:

   1. Using the following YAML example, create a file named pod-a.yaml file:

      Note

      The manifest below includes 2 resources:

      • Namespace with security labels
      • Pod definition with appropriate network annotation

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-namespace
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx-pod
        namespace: test-namespace
        annotations:
          k8s.v1.cni.cncf.io/networks: '[
            {
              "name": "sriov-network",
              "namespace": "test-namespace",
              "interface": "ext0" 

      1

      
            },
            {
              "name": "vlan-100",
              "namespace": "test-namespace",
              "interface": "ext0.100"
            }
          ]'
      spec:
        securityContext:
          runAsNonRoot: true
        containers:
          - name: nginx-container
            image: nginxinc/nginx-unprivileged:latest
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            ports:
              - containerPort: 80
            seccompProfile:
              type: "RuntimeDefault"

      Copy to Clipboard Toggle word wrap

      1

      The name to be used as the master for the VLAN interface.

   2. Apply the YAML file by running the following command:

      $ oc apply -f pod-a.yaml

      Copy to Clipboard Toggle word wrap

6. Get detailed information about the nginx-pod within the test-namespace by running the following command:

   $ oc describe pods nginx-pod -n test-namespace

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         nginx-pod
   Namespace:    test-namespace
   Priority:     0
   Node:         worker-1/10.46.186.105
   Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
   Labels:       <none>
   Annotations:  k8s.ovn.org/pod-networks:
                   {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                 k8s.v1.cni.cncf.io/network-status:
                   [{
                       "name": "ovn-kubernetes",
                       "interface": "eth0",
                       "ips": [
                           "10.131.0.26"
                       ],
                       "mac": "0a:58:0a:83:00:1a",
                       "default": true,
                       "dns": {}
                   },{
                       "name": "test-namespace/sriov-network",
                       "interface": "ext0",
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {},
                       "device-info": {
                           "type": "pci",
                           "version": "1.0.0",
                           "pci": {
                               "pci-address": "0000:d8:00.2"
                           }
                       }
                   },{
                       "name": "test-namespace/vlan-100",
                       "interface": "ext0.100",
                       "ips": [
                           "1.1.1.1"
                       ],
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {}
                   }]
                 k8s.v1.cni.cncf.io/networks:
                   [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                 openshift.io/scc: privileged
   Status:       Running
   IP:           10.131.0.26
   IPs:
     IP:  10.131.0.26

   Copy to Clipboard Toggle word wrap