Chapter 8. Network requirements

In order for Ceph-CSI to communicate with a Multus-enabled CephCluster, some setup is required for Kubernetes hosts.

These prerequisites require an understanding of how Multus networks are configured and how Rook uses them. This section will help clarify questions that could arise.

Two basic requirements must be met:

These two requirements can be broken down further as follows:

Node configurations and pod configurations are interrelated and tightly coupled. Both must be planned at the same time, and OpenShift Data Foundation cannot support Multus public networks without both.

The “public-network-interface” must be the same for both. Generally, the connection technology (Macvlan) should also be the same for both. IP range(s) in the NetworkAttachmentDefinition must be encoded as routes on nodes, and, in mirror, IP ranges for nodes must be encoded as routes in the NetworkAttachmentDefinition.

Some installations might not want to use the same public network IP address range for both pods and nodes. In the case where there are different ranges for pods and nodes, additional steps must be taken to ensure each range routes to the other so that they act as a single, contiguous network.These requirements require careful planning. See Multus examples to help understand and implement these requirements.

OpenShift Container Platform recommends using the NMState operator’s NodeNetworkConfigurationPolicies as a good method of configuring hosts to meet host requirements. Other methods can be used as well if needed.

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: multus-public-test
 namespace: openshift-storage
 labels:
   app: multus-public-test
spec:
 selector:
   matchLabels:
     app: multus-public-test
 template:
   metadata:
     labels:
       app: multus-public-test
     annotations:
       k8s.v1.cni.cncf.io/networks: openshift-storage/public-net #
   spec:
     containers:
       - name: test
         image: quay.io/ceph/ceph:v18 # image known to have ‘ping’ installed
         command:
           - sleep
           - infinity
         resources: {}

$ oc -n openshift-storage describe pod -l app=multus-public-test | grep -o -E 'Add .* from .*'
Add eth0 [10.128.2.86/23] from ovn-kubernetes
Add net1 [192.168.20.22/24] from default/public-net
Add eth0 [10.129.2.173/23] from ovn-kubernetes
Add net1 [192.168.20.29/24] from default/public-net
Add eth0 [10.131.0.108/23] from ovn-kubernetes
Add net1 [192.168.20.23/24] from default/public-net

$ oc debug node/NODE
Starting pod/NODE-debug ...
To use host binaries, run `chroot /host`
Pod IP: ****
If you don't see a command prompt, try pressing enter.


sh-5.1# chroot /host


sh-5.1# ping 192.168.20.22
PING 192.168.20.22 (192.168.20.22) 56(84) bytes of data.
64 bytes from 192.168.20.22: icmp_seq=1 ttl=64 time=0.093 ms
64 bytes from 192.168.20.22: icmp_seq=2 ttl=64 time=0.056 ms
^C
--- 192.168.20.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1046ms
rtt min/avg/max/mdev = 0.056/0.074/0.093/0.018 ms


sh-5.1# ping 192.168.20.29
PING 192.168.20.29 (192.168.20.29) 56(84) bytes of data.
64 bytes from 192.168.20.29: icmp_seq=1 ttl=64 time=0.403 ms
64 bytes from 192.168.20.29: icmp_seq=2 ttl=64 time=0.181 ms
^C
--- 192.168.20.29 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.181/0.292/0.403/0.111 ms


sh-5.1# ping 192.168.20.23
PING 192.168.20.23 (192.168.20.23) 56(84) bytes of data.
64 bytes from 192.168.20.23: icmp_seq=1 ttl=64 time=0.329 ms
64 bytes from 192.168.20.23: icmp_seq=2 ttl=64 time=0.227 ms
^C
--- 192.168.20.23 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.227/0.278/0.329/0.051 ms

The relevant network plan for this cluster is as follows:

Nodes’ network policies must be configured to route to pods on the Multus public network.

Because pods will be connecting via Macvlan, and because Macvlan does not allow hosts and pods to route between each other, the host must also be connected via Macvlan. Generally speaking, the host must connect to the Multus public network using the same technology that pods do. Pod connections are configured in the Network Attachment Definition.

Because the host IP range is a subset of the whole range, hosts are not able to route to pods simply by IP assignment. A route must be added to hosts to allow them to route to the whole 192.168.0.0/16 range.

NodeNetworkConfigurationPolicy desiredState specs will look like the following:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: ceph-public-net-shim-compute-0
 namespace: openshift-storage
spec:
 nodeSelector:
   node-role.kubernetes.io/worker: ""
   kubernetes.io/hostname: compute-0
 desiredState:
   interfaces:
     - name: odf-pub-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: eth0
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.252.1 # STATIC IP FOR compute-0
             prefix-length: 22
   routes:
     config:
       - destination: 192.168.0.0/16
         next-hop-interface: odf-pub-shim
---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: ceph-public-net-shim-compute-1
 namespace: openshift-storage
spec:
 nodeSelector:
   node-role.kubernetes.io/worker: ""
   kubernetes.io/hostname: compute-1
 desiredState:
   interfaces:
     - name: odf-pub-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan
       state: up
       mac-vlan:
         base-iface: eth0
         mode: bridge
         promiscuous: true
       ipv4:
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.252.1 # STATIC IP FOR compute-1
             prefix-length: 22
   routes:
     config:
       - destination: 192.168.0.0/16
         next-hop-interface: odf-pub-shim
---
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
 name: ceph-public-net-shim-compute-2 # [1]
 namespace: openshift-storage
spec:
 nodeSelector:
   node-role.kubernetes.io/worker: ""
   kubernetes.io/hostname: compute-2 # [2]
 desiredState:
   Interfaces: [3]
     - name: odf-pub-shim
       description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
       type: mac-vlan # [4]
       state: up
       mac-vlan:
         base-iface: eth0 # [5]
         mode: bridge
         promiscuous: true
       ipv4: # [6]
         enabled: true
         dhcp: false
         address:
           - ip: 192.168.252.2 # STATIC IP FOR compute-2 # [7]
             prefix-length: 22
   routes: # [8]
     config:
       - destination: 192.168.0.0/16 # [9]
         next-hop-interface: odf-pub-shim

The NetworkAttachmentDefinition for the public network would look like the following, using Whereabouts' exclude option to simplify the range request. The Whereabouts routes[].dst option ensures pods route to hosts via the Multus public network.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: public-net
 namespace: openshift-storage
spec:
 config: '{
     "cniVersion": "0.3.1",
     "type": "macvlan", # [1]
     "master": "eth0", # [2]
     "mode": "bridge",
     "ipam": {
       "type": "whereabouts", # [3]
       "range": "192.168.0.0/16", # [4]
       "exclude": [
          "192.168.252.0/22" # [5]
       ],
       "routes": [              # [6]
         {"dst": "192.168.252.0/22"} # [7]
       ]
     }
   }'

Due to the recurring maintenance impact of holder pods during upgrade (holder pods are present when Multus is enabled), holder pods are deprecated in the ODF v4.18 release and targeted for removal in the ODF v4.18 release. This deprecation requires completing additional network configuration actions before removing the holder pods. In ODF v4.16, clusters with Multus enabled are upgraded to v4.17 following standard upgrade procedures. After the ODF cluster (with Multus enabled) is successfully upgraded to v4.17, administrators must then complete the procedure documented in the article Disabling Multus holder pods to disable and remove holder pods. Be aware that this disabling procedure is time consuming; however, it is not critical to complete the entire process immediately after upgrading to v4.17. It is critical to complete the process before ODF is upgraded to v4.18.

By default, Red Hat OpenShift Data Foundation is configured to use the Red Hat OpenShift Software Defined Network (SDN). The default SDN carries the following types of traffic:

There are three ways to segregate OpenShift Data Foundation from OpenShift default network:

Use Multus for OpenShift Data Foundation when you need the following:

Improved latency - Multus with ODF always improves latency. Use host interfaces at near-host network speeds and bypass OpenShift’s software-defined Pod network. You can also perform Linux per interface level tuning for each interface.

Improved bandwidth - Dedicated interfaces for OpenShift Data Foundation client data traffic and internal data traffic. These dedicated interfaces reserve full bandwidth.

Improved security - Multus isolates storage network traffic from application network traffic for added security. Bandwidth or performance might not be isolated when networks share an interface, however, you can use QoS or traffic shaping to prioritize bandwidth on shared interfaces.

To use Multus, you must create network attachment definitions (NADs) before deploying the OpenShift Data Foundation cluster, which is later attached to the cluster. For more information, see Creating network attachment definitions.

To attach additional network interfaces to a pod, you must create configurations that define how the interfaces are attached. You specify each interface by using a NetworkAttachmentDefinition custom resource (CR). A Container Network Interface (CNI) configuration inside each of these CRs defines how that interface is created.

OpenShift Data Foundation supports the macvlan driver, which includes the following features:

OpenShift Data Foundation supports the following two types IP address management:

See Creating Multus networks for the necessary steps to configure a Multus based configuration on bare metal.