Chapter 21. Multiple networks

Procedure

1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

   $ oc edit network.operator.openshift.io cluster

2. Include the additionalNetworks section shown in this example YAML extract within the spec definition of the custom resource (CR):

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   # ...
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       rawCNIConfig: |-
         {
          "name": "whereabouts-shim",
          "cniVersion": "0.3.1",
          "type": "bridge",
          "ipam": {
            "type": "whereabouts"
          }
         }
       type: Raw
   # ...

3. Save the file and exit the text editor.

4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
   pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
   pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
   pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
   pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
   pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
   daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Procedure

1. Run the following command to create a ConfigMap object named whereabouts-config in the openshift-multus namespace with a specific cron expression for the IP reconciler:

   $ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"

   This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

   Note

   The whereabouts-reconciler daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

2. Retrieve information about resources related to the whereabouts-reconciler daemon set and pods within the openshift-multus namespace by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-94zw6                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-mfh68                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-pgshz                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-xn5xz                   1/1     Running   0             4m14s
   daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s

3. Run the following command to verify that the whereabouts-reconciler pod runs the IP reconciler with the configured interval:

   $ oc -n openshift-multus logs whereabouts-reconciler-2p7hw

   Example output

   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
   2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
   2024-02-02T16:45:00Z [verbose] starting reconciler run
   2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
   2024-02-02T16:45:00Z [debug] listing IP pools
   2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
   2024-02-02T16:45:00Z [verbose] reconciler success

Procedure

1. Set type to whereabouts.

2. Use ipRanges to allocate IP addresses as shown in the following example:

   cniVersion: operator.openshift.io/v1
   kind: Network
   =metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       type: Raw
       rawCNIConfig: |-
         {
          "name": "whereabouts-dual-stack",
          "cniVersion": "0.3.1,
          "type": "bridge",
          "ipam": {
            "type": "whereabouts",
            "ipRanges": [
                     {"range": "192.168.10.0/24"},
                     {"range": "2001:db8::/64"}
                 ]
          }
         }

3. Attach network to a pod. For more information, see "Adding a pod to an additional network".
4. Verify that all IP addresses are assigned.

5. Run the following command to ensure the IP addresses are assigned as metadata.

   $ oc exec -it mypod -- ip a

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

   $ oc new-project test-namespace

2. Create an SR-IOV node policy:

   1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" 1
         deviceID: "101b" 2
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      Note

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1

      The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

      2

      The device hexadecimal code of the SR-IOV network device.

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-node-network-policy.yaml

      Note

      Applying this might take some time due to the node requiring a reboot.

3. Create an SR-IOV network:

   1. Create the SriovNetwork custom resource (CR) for the additional SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-network-attachment.yaml

4. Create the VLAN additional network:

   1. Using the following YAML example, create a file named vlan100-additional-network-configuration.yaml:

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: vlan-100
        namespace: test-namespace
      spec:
        config: |
          {
            "cniVersion": "0.4.0",
            "name": "vlan-100",
            "plugins": [
              {
                "type": "vlan",
                "master": "ext0", 1
                "mtu": 1500,
                "vlanId": 100,
                "linkInContainer": true, 2
                "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
              }
            ]
          }

      1

      The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.

      2

      The linkInContainer parameter must be specified.

   2. Apply the YAML file by running the following command:

      $ oc apply -f vlan100-additional-network-configuration.yaml

5. Create a pod definition by using the earlier specified networks:

   1. Using the following YAML example, create a file named pod-a.yaml file:

      Note

      The manifest below includes 2 resources:

      • Namespace with security labels
      • Pod definition with appropriate network annotation

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-namespace
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx-pod
        namespace: test-namespace
        annotations:
          k8s.v1.cni.cncf.io/networks: '[
            {
              "name": "sriov-network",
              "namespace": "test-namespace",
              "interface": "ext0" 1
            },
            {
              "name": "vlan-100",
              "namespace": "test-namespace",
              "interface": "ext0.100"
            }
          ]'
      spec:
        securityContext:
          runAsNonRoot: true
        containers:
          - name: nginx-container
            image: nginxinc/nginx-unprivileged:latest
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            ports:
              - containerPort: 80
            seccompProfile:
              type: "RuntimeDefault"

      1

      The name to be used as the master for the VLAN interface.

   2. Apply the YAML file by running the following command:

      $ oc apply -f pod-a.yaml

6. Get detailed information about the nginx-pod within the test-namespace by running the following command:

   $ oc describe pods nginx-pod -n test-namespace

   Example output

   Name:         nginx-pod
   Namespace:    test-namespace
   Priority:     0
   Node:         worker-1/10.46.186.105
   Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
   Labels:       <none>
   Annotations:  k8s.ovn.org/pod-networks:
                   {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                 k8s.v1.cni.cncf.io/network-status:
                   [{
                       "name": "ovn-kubernetes",
                       "interface": "eth0",
                       "ips": [
                           "10.131.0.26"
                       ],
                       "mac": "0a:58:0a:83:00:1a",
                       "default": true,
                       "dns": {}
                   },{
                       "name": "test-namespace/sriov-network",
                       "interface": "ext0",
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {},
                       "device-info": {
                           "type": "pci",
                           "version": "1.0.0",
                           "pci": {
                               "pci-address": "0000:d8:00.2"
                           }
                       }
                   },{
                       "name": "test-namespace/vlan-100",
                       "interface": "ext0.100",
                       "ips": [
                           "1.1.1.1"
                       ],
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {}
                   }]
                 k8s.v1.cni.cncf.io/networks:
                   [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                 openshift.io/scc: privileged
   Status:       Running
   IP:           10.131.0.26
   IPs:
     IP:  10.131.0.26

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by running the following command:

   $ oc new-project test-namespace

2. Using the following YAML example, create a bridge NetworkAttachmentDefinition custom resource (CR) file named bridge-nad.yaml:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-network
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "bridge-network",
       "type": "bridge",
       "bridge": "br-001",
       "isGateway": true,
       "ipMasq": true,
       "hairpinMode": true,
       "ipam": {
         "type": "host-local",
         "subnet": "10.0.0.0/24",
         "routes": [{"dst": "0.0.0.0/0"}]
       }
     }'

3. Run the following command to apply the NetworkAttachmentDefinition CR to your OpenShift Container Platform cluster:

   $ oc apply -f bridge-nad.yaml

4. Verify that the NetworkAttachmentDefinition CR has been created successfully by running the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   15s

5. Using the following YAML example, create a file named ipvlan-additional-network-configuration.yaml for the IPVLAN additional network configuration:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ipvlan-net
     namespace: test-namespace
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "ipvlan-net",
       "type": "ipvlan",
       "master": "ext0", 1
       "mode": "l3",
       "linkInContainer": true, 2
       "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]}
     }'

   1

   Specifies the ethernet interface to associate with the network attachment. This is subsequently configured in the pod networks annotation.

   2

   Specifies that the master interface is in the container network namespace.

6. Apply the YAML file by running the following command:

   $ oc apply -f ipvlan-additional-network-configuration.yaml

7. Verify that the NetworkAttachmentDefinition CR has been created successfully by running the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   87s
   ipvlan-net       9s

8. Using the following YAML example, create a file named pod-a.yaml for the pod definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-a
     namespace: test-namespace
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "bridge-network",
           "interface": "ext0" 1
         },
         {
           "name": "ipvlan-net",
           "interface": "ext1"
         }
       ]'
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: test-pod
       image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]

   1

   Specifies the name to be used as the master for the IPVLAN interface.

9. Apply the YAML file by running the following command:

   $ oc apply -f pod-a.yaml

10. Verify that the pod is running by using the following command:

    $ oc get pod -n test-namespace

    Example output

    NAME    READY   STATUS    RESTARTS   AGE
    pod-a   1/1     Running   0          2m36s

11. Show network interface information about the pod-a resource within the test-namespace by running the following command:

    $ oc exec -n test-namespace pod-a -- ip a

    Example output

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
        link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::488b:91ff:fe84:a94b/64 scope link
           valid_lft forever preferred_lft forever
    4: ext0@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.0.0.2/24 brd 10.0.0.255 scope global ext0
           valid_lft forever preferred_lft forever
        inet6 fe80::bcda:bdff:fe7e:f437/64 scope link
           valid_lft forever preferred_lft forever
    5: ext1@ext0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.1/24 brd 10.0.0.255 scope global ext1
           valid_lft forever preferred_lft forever
        inet6 fe80::beda:bd00:17e:f437/64 scope link
           valid_lft forever preferred_lft forever

    This output shows that the network interface ext1 is associated with the physical interface ext0.

1. Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit.

   $ oc edit pod <name>

2. In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties.

   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]' 1

   1

   Replace <network> with a JSON object as shown in the following examples. The single quotes are required.

3. In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter.

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-pod
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
       {
         "name": "net1"
       },
       {
         "name": "net2", 1
         "default-route": ["192.0.2.1"] 2
       }]'
   spec:
     containers:
     - name: example-pod
       command: ["/bin/bash", "-c", "sleep 2000000000000"]
       image: centos/tools

   1

   The name key is the name of the additional network to associate with the pod.

   2

   The default-route key specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster