Chapter 22. Multiple networks

Procedure

1. Edit the

   Network.operator.openshift.io

   custom resource (CR) by running the following command:

   $ oc edit network.operator.openshift.io cluster

2. Include the

   additionalNetworks

   section shown in this example YAML extract within the

   spec

   definition of the custom resource (CR):

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   # ...
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       rawCNIConfig: |-
         {
          "name": "whereabouts-shim",
          "cniVersion": "0.3.1",
          "type": "bridge",
          "ipam": {
            "type": "whereabouts"
          }
         }
       type: Raw
   # ...

3. Save the file and exit the text editor.

4. Verify that the

   whereabouts-reconciler

   daemon set deployed successfully by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
   pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
   pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
   pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
   pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
   pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
   daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Procedure

1. Run the following command to create a

   ConfigMap

   object named

   whereabouts-config

   in the

   openshift-multus

   namespace with a specific cron expression for the IP reconciler:

   $ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"

   This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

   Note

   The

   whereabouts-reconciler

   daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

2. Retrieve information about resources related to the

   whereabouts-reconciler

   daemon set and pods within the

   openshift-multus

   namespace by running the following command:

   $ oc get all -n openshift-multus | grep whereabouts-reconciler

   Example output

   pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-94zw6                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-mfh68                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-pgshz                   1/1     Running   0             4m14s
   pod/whereabouts-reconciler-xn5xz                   1/1     Running   0             4m14s
   daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s

3. Run the following command to verify that the

   whereabouts-reconciler

   pod runs the IP reconciler with the configured interval:

   $ oc -n openshift-multus logs whereabouts-reconciler-2p7hw

   Example output

   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
   2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
   2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
   2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
   2024-02-02T16:45:00Z [verbose] starting reconciler run
   2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
   2024-02-02T16:45:00Z [debug] listing IP pools
   2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
   2024-02-02T16:45:00Z [verbose] reconciler success

Procedure

1. Set

   type

   to

   whereabouts

   .

2. Use

   ipRanges

   to allocate IP addresses as shown in the following example:

   cniVersion: operator.openshift.io/v1
   kind: Network
   =metadata:
     name: cluster
   spec:
     additionalNetworks:
     - name: whereabouts-shim
       namespace: default
       type: Raw
       rawCNIConfig: |-
         {
          "name": "whereabouts-dual-stack",
          "cniVersion": "0.3.1,
          "type": "bridge",
          "ipam": {
            "type": "whereabouts",
            "ipRanges": [
                     {"range": "192.168.10.0/24"},
                     {"range": "2001:db8::/64"}
                 ]
          }
         }

3. Attach network to a pod. For more information, see "Adding a pod to an additional network".
4. Verify that all IP addresses are assigned.

5. Run the following command to ensure the IP addresses are assigned as metadata.

   $ oc exec -it mypod -- ip a

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

   $ oc new-project test-namespace

2. Create an SR-IOV node policy:

   1. Create an

      SriovNetworkNodePolicy

      object, and then save the YAML in the

      sriov-node-network-policy.yaml

      file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" 

      1

      
         deviceID: "101b" 

      2

      
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      Note

      The SR-IOV network node policy configuration example, with the setting

      deviceType: netdevice

      , is tailored specifically for Mellanox Network Interface Cards (NICs).

      1

      The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.

      2

      The device hexadecimal code of the SR-IOV network device.

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-node-network-policy.yaml

      Note

      Applying this might take some time due to the node requiring a reboot.

3. Create an SR-IOV network:

   1. Create the

      SriovNetwork

      custom resource (CR) for the additional SR-IOV network attachment as in the following example CR. Save the YAML as the file

      sriov-network-attachment.yaml

      :

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"

   2. Apply the YAML by running the following command:

      $ oc apply -f sriov-network-attachment.yaml

4. Create the VLAN additional network:

   1. Using the following YAML example, create a file named

      vlan100-additional-network-configuration.yaml

      :

      apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: vlan-100
        namespace: test-namespace
      spec:
        config: |
          {
            "cniVersion": "0.4.0",
            "name": "vlan-100",
            "plugins": [
              {
                "type": "vlan",
                "master": "ext0", 

      1

      
                "mtu": 1500,
                "vlanId": 100,
                "linkInContainer": true, 

      2

      
                "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
              }
            ]
          }

      1

      The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.

      2

      The linkInContainer parameter must be specified.

   2. Apply the YAML file by running the following command:

      $ oc apply -f vlan100-additional-network-configuration.yaml

5. Create a pod definition by using the earlier specified networks:

   1. Using the following YAML example, create a file named

      pod-a.yaml

      file:
      Note

      The manifest below includes 2 resources:

      • Namespace with security labels
      • Pod definition with appropriate network annotation

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-namespace
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
          security.openshift.io/scc.podSecurityLabelSync: "false"
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx-pod
        namespace: test-namespace
        annotations:
          k8s.v1.cni.cncf.io/networks: '[
            {
              "name": "sriov-network",
              "namespace": "test-namespace",
              "interface": "ext0" 

      1

      
            },
            {
              "name": "vlan-100",
              "namespace": "test-namespace",
              "interface": "ext0.100"
            }
          ]'
      spec:
        securityContext:
          runAsNonRoot: true
        containers:
          - name: nginx-container
            image: nginxinc/nginx-unprivileged:latest
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            ports:
              - containerPort: 80
            seccompProfile:
              type: "RuntimeDefault"

      1

      The name to be used as the master for the VLAN interface.

   2. Apply the YAML file by running the following command:

      $ oc apply -f pod-a.yaml

6. Get detailed information about the

   nginx-pod

   within the

   test-namespace

   by running the following command:

   $ oc describe pods nginx-pod -n test-namespace

   Example output

   Name:         nginx-pod
   Namespace:    test-namespace
   Priority:     0
   Node:         worker-1/10.46.186.105
   Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
   Labels:       <none>
   Annotations:  k8s.ovn.org/pod-networks:
                   {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                 k8s.v1.cni.cncf.io/network-status:
                   [{
                       "name": "ovn-kubernetes",
                       "interface": "eth0",
                       "ips": [
                           "10.131.0.26"
                       ],
                       "mac": "0a:58:0a:83:00:1a",
                       "default": true,
                       "dns": {}
                   },{
                       "name": "test-namespace/sriov-network",
                       "interface": "ext0",
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {},
                       "device-info": {
                           "type": "pci",
                           "version": "1.0.0",
                           "pci": {
                               "pci-address": "0000:d8:00.2"
                           }
                       }
                   },{
                       "name": "test-namespace/vlan-100",
                       "interface": "ext0.100",
                       "ips": [
                           "1.1.1.1"
                       ],
                       "mac": "6e:a7:5e:3f:49:1b",
                       "dns": {}
                   }]
                 k8s.v1.cni.cncf.io/networks:
                   [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                 openshift.io/scc: privileged
   Status:       Running
   IP:           10.131.0.26
   IPs:
     IP:  10.131.0.26

Procedure

1. Create a dedicated container namespace where you want to deploy your pod by entering the following command:

   $ oc new-project test-namespace

2. Using the following YAML example, create a bridge

   NetworkAttachmentDefinition

   custom resource definition (CRD) file named

   bridge-nad.yaml

   :

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-network
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "bridge-network",
       "type": "bridge",
       "bridge": "br-001",
       "isGateway": true,
       "ipMasq": true,
       "hairpinMode": true,
       "ipam": {
         "type": "host-local",
         "subnet": "10.0.0.0/24",
         "routes": [{"dst": "0.0.0.0/0"}]
       }
     }'

3. Run the following command to apply the

   NetworkAttachmentDefinition

   CRD to your OpenShift Container Platform cluster:

   $ oc apply -f bridge-nad.yaml

4. Verify that you successfully created a

   NetworkAttachmentDefinition

   CRD by entering the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   15s

5. Using the following YAML example, create a file named

   ipvlan-additional-network-configuration.yaml

   for the IPVLAN additional network configuration:

   apiVersion: k8s.cni.cncf.io/v1
   kind: NetworkAttachmentDefinition
   metadata:
     name: ipvlan-net
     namespace: test-namespace
   spec:
     config: '{
       "cniVersion": "0.3.1",
       "name": "ipvlan-net",
       "type": "ipvlan",
       "master": "ext0", 

   1

   
       "mode": "l3",
       "linkInContainer": true, 

   2

   
       "ipam": {"type": "whereabouts", "ipRanges": [{"range": "10.0.0.0/24"}]}
     }'

   1

   Specifies the ethernet interface to associate with the network attachment. This is subsequently configured in the pod networks annotation.

   2

   Specifies that the master interface is in the container network namespace.

6. Apply the YAML file by running the following command:

   $ oc apply -f ipvlan-additional-network-configuration.yaml

7. Verify that the

   NetworkAttachmentDefinition

   CRD has been created successfully by running the following command:

   $ oc get network-attachment-definitions

   Example output

   NAME             AGE
   bridge-network   87s
   ipvlan-net       9s

8. Using the following YAML example, create a file named

   pod-a.yaml

   for the pod definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-a
     namespace: test-namespace
     annotations:
       k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "bridge-network",
           "interface": "ext0" 

   1

   
         },
         {
           "name": "ipvlan-net",
           "interface": "ext1"
         }
       ]'
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
     - name: test-pod
       image: quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop: [ALL]

   1

   Specifies the name to be used as the master for the IPVLAN interface.

9. Apply the YAML file by running the following command:

   $ oc apply -f pod-a.yaml

10. Verify that the pod is running by using the following command:

    $ oc get pod -n test-namespace

    Example output

    NAME    READY   STATUS    RESTARTS   AGE
    pod-a   1/1     Running   0          2m36s

11. Show network interface information about the

    pod-a

    resource within the

    test-namespace

    by running the following command:

    $ oc exec -n test-namespace pod-a -- ip a

    Example output

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    3: eth0@if105: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
        link/ether 0a:58:0a:d9:00:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.217.0.93/23 brd 10.217.1.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::488b:91ff:fe84:a94b/64 scope link
           valid_lft forever preferred_lft forever
    4: ext0@if107: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 10.0.0.2/24 brd 10.0.0.255 scope global ext0
           valid_lft forever preferred_lft forever
        inet6 fe80::bcda:bdff:fe7e:f437/64 scope link
           valid_lft forever preferred_lft forever
    5: ext1@ext0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
        link/ether be:da:bd:7e:f4:37 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.1/24 brd 10.0.0.255 scope global ext1
           valid_lft forever preferred_lft forever
        inet6 fe80::beda:bd00:17e:f437/64 scope link
           valid_lft forever preferred_lft forever

    This output shows that the network interface

    ext1

    is associated with the physical interface

    ext0

    .