Chapter 5. Multiple networks

Prerequisites

1. You have root access to the host.

Procedure

1. Install the Multus RPM package by running the following command:

   $ sudo dnf install microshift-multus

   Tip

   If you create your custom resources (CRs) for additional networks now, you can complete your installation and apply configurations with one restart.

2. To apply the package manifest to an active cluster, restart the host by running the following command:

   $ sudo systemctl restart

Verification

1. After restarting, ensure that the Multus CNI plugin components are created by running the following command:

   $ oc get pod -A | grep multus

   Example output

   openshift-multus      dhcp-daemon-ktzqf     1/1   Running   0     45h
   openshift-multus      multus-4frf4          1/1   Running   0     45h

Next steps

1. If you have not done so, configure and apply the additional networks you want to use.
2. Deploy your applications that use the created CRs.

Procedure

1. Optional: Verify that the MicroShift cluster is running with the Multus CNI by running the following command:

   $ oc get pods -n openshift-multus

   Example output

   NAME                READY   STATUS    RESTARTS   AGE
   dhcp-daemon-dfbzw   1/1     Running   0          5h
   multus-rz8xc        1/1     Running   0          5h

2. Create a NetworkAttachmentDefinition configuration file by running the following command and using the following example file for reference:

   $ oc apply -f network-attachment-definition.yaml

   Example NetworkAttachmentDefinition file

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: bridge-conf
   spec:
     config: '{
         "cniVersion": "0.4.0",
         "type": "bridge", 1
         "bridge": "br-test", 2
         "mode": "bridge",
         "ipam": {
           "type": "host-local", 3
           "ranges": [
             [
               {
                 "subnet": "10.10.0.0/24",
                 "rangeStart": "10.10.0.20",
                 "rangeEnd": "10.10.0.50",
                 "gateway": "10.10.0.254"
                }
             ],
             [
               {
                 "subnet": "fd00:IJKL:MNOP:10::0/64", 4
                 "rangeStart": "fd00:IJKL:MNOP:10::1",
                 "rangeEnd": "fd00:IJKL:MNOP:10::9"
           "dataDir": "/var/lib/cni/br-test"
         }
       }'

   1

   The type value specifies a name of the CNI plugin. This example uses the bridge type.

   2

   The bridge value is name of the bridge on the MicroShift host that is used. The additional interface of the pod is connected to that bridge. If the interface does not exist on the host, the Bridge CNI creates it. If the interface already exists, it is reused. In this example, the name of the interface is br-test.

   3

   The IPAM type.

   4

   IPv6 addresses can be added to the secondary interface.

   Note

   Using the name of the bridge is specific to the bridge type of plugin. Other plugins use different fields in their NetworkAttachmentDefinitions. For example, the macvlan and ipvlan configurations use master to specify the host interface to attach.

Procedure

1. Add an annotation to a Pod YAML file. Only one of the following annotation formats can be used:

   1. To attach an additional network without any customization, add an annotation with the following format. Replace <network> with the name of the additional network to associate with the pod:

      apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: <network>[,<network>,...] 1
      # ...

      1

      Replace <network> with the name of the additional network to associate with the pod. To specify more than one additional network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same additional network multiple times, that pod will have multiple network interfaces attached to that network.

      Example annotation for a bridge-type additional network

      apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: bridge-conf
      # ...

   2. To attach an additional network with customizations, add an annotation with the following format:

      apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "<network>", 1
                "namespace": "<namespace>", 2
                "default-route": ["<default-route>"] 3
              }
            ]
      # ...

      1

      Specify the name of the additional network defined by a NetworkAttachmentDefinition object.

      2

      Specify the namespace where the NetworkAttachmentDefinition object is defined.

      3

      Optional: Specify an override for the default route, such as 192.168.17.1.

2. To create a Pod YAML file and add the NetworkAttachmentDefinition annotation for an additional network, run the following command and use the example YAML:

   $ oc apply -f ./<test-bridge>.yaml 1

   1

   Replace <test-bridge> with the pod name that you want to use.

   Example output

   pod/test-bridge created

   Example test-bridge pod YAML

   apiVersion: v1
   kind: Pod
   metadata:
     name: test-bridge
     annotations:
       k8s.v1.cni.cncf.io/networks: bridge-conf
     labels:
       app: test-bridge
   spec:
     terminationGracePeriodSeconds: 0
     containers:
     - name: hello-microshift
       image: quay.io/microshift/busybox:1.36
       command: ["/bin/sh"]
       args: ["-c", "while true; do echo -ne \"HTTP/1.0 200 OK\r\nContent-Length: 16\r\n\r\nHello MicroShift\" | nc -l -p 8080 ; done"]
       ports:
       - containerPort: 8080
         protocol: TCP
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop:
           - ALL
         runAsNonRoot: true
         runAsUser: 1001
         runAsGroup: 1001
         seccompProfile:
           type: RuntimeDefault

3. Make sure that the NetworkAttachmentDefinition annotation is correct:

   Example NetworkAttachmentDefinition annotation

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: bridge-conf
   # ...

4. Optional: To confirm that the NetworkAttachmentDefinition annotation exists in a Pod YAML, run the following command, replacing <name> with the name of the pod.

   $ oc get pod <name> -o yaml 1

   1

   Replace <name> with the pod name you want to use. In the following example, test-bridge is used.

   In the following example, the test-bridge is attached to the net1 additional network:

   $ oc get pod test-bridge -o yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: bridge-conf
       k8s.v1.cni.cncf.io/network-status: |- 1
         [{
             "name": "ovn-kubernetes",
             "interface": "eth0",
             "ips": [
                 "10.42.0.18"
             ],
             "default": true,
             "dns": {}
         },{
             "name": "bridge-conf",
             "interface": "net1",
             "ips": [
                 "20.2.2.100"
             ],
             "mac": "22:2f:60:a5:f8:00",
             "dns": {}
         }]
     name: pod
     namespace: default
   spec:
   # ...
   status:
   # ...

   1

   The k8s.v1.cni.cncf.io/network-status parameter is a JSON array of objects. Each object describes the status of an additional network attached to the pod. The annotation value is stored as a plain text value.

5. Verify that the pod is running by running the following command:

   $ oc get pod

   Example output

   NAME          READY   STATUS    RESTARTS   AGE
   test-bridge   1/1     Running   0          81s

Prerequisite

1. You created and applied the NetworkAttachmentDefinition object configuration.

Procedure

1. Verify that the bridge was created on the host by running the following command:

   $ ip a show br-test

   Example output

   22: br-test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
       link/ether 96:bf:ca:be:1d:15 brd ff:ff:ff:ff:ff:ff
       inet6 fe80::34e2:bbff:fed2:31f2/64 scope link
          valid_lft forever preferred_lft forever

2. Configure an IP address for the bridge by running the following command:

   $ sudo ip addr add 10.10.0.10/24 dev br-test

3. Verify that the IP address configuration is added to the bridge by running the following command:

   $ ip a show br-test

   Example output

   22: br-test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
       link/ether 96:bf:ca:be:1d:15 brd ff:ff:ff:ff:ff:ff
       inet 10.10.0.10/24 scope global br-test 1
          valid_lft forever preferred_lft forever
       inet6 fe80::34e2:bbff:fed2:31f2/64 scope link
          valid_lft forever preferred_lft forever

   1

   The IP address is configured as expected.

4. Verify the IP address of the pod by running the following command:

   $ oc get pod test-bridge --output=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'

   Example output

   [{
       "name": "ovn-kubernetes",
       "interface": "eth0",
       "ips": [
           "10.42.0.17"
       ],
       "mac": "0a:58:0a:2a:00:11",
       "default": true,
       "dns": {}
   },{
       "name": "default/bridge-conf", 1
       "interface": "net1",
       "ips": [
           "10.10.0.20"
       ],
       "mac": "82:01:98:e5:0c:b7",
       "dns": {}

   1

   The bridge additional network is attached as expected.

5. Optional: You can use oc exec to access the pod and confirm its interfaces by using the ip command:

   $ oc exec -ti test-bridge -- ip a

   Example output

   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
   2: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
       link/ether 0a:58:0a:2a:00:11 brd ff:ff:ff:ff:ff:ff
       inet 10.42.0.17/24 brd 10.42.0.255 scope global eth0
          valid_lft forever preferred_lft forever
       inet6 fe80::858:aff:fe2a:11/64 scope link
          valid_lft forever preferred_lft forever
   3: net1@if23: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
       link/ether 82:01:98:e5:0c:b7 brd ff:ff:ff:ff:ff:ff
       inet 10.10.0.20/24 brd 10.10.0.255 scope global net1 1
          valid_lft forever preferred_lft forever
       inet6 fe80::8001:98ff:fee5:cb7/64 scope link
          valid_lft forever preferred_lft forever

   1

   Pod is attached to the 10.10.0.20 IP address on the net1 interface as expected.

6. Confirm that the connection is working as expected by accessing the HTTP server in the pod from the MicroShift host. Use the following command:

   $ curl 10.10.0.20:8080

   Example output

   Hello MicroShift