Chapter 6. Multus architecture for OpenShift Data Foundation

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using the Multus public network:

Figure 6.2. OpenShift Data Foundation deployment using Multus public network

This architecture achieves all the three benefits of Multus networking:

However, if the public network has less throughput than the Pod network, storage traffic might not be faster than a traditional deployment.

Public network is the recommended architecture for the majority of OpenShift Data Foundation cluster deployments. In this architecture, all storage traffic uses the dedicated public network, which includes I/O traffic from applications, data replication traffic within the storage system, and data recovery traffic also within the storage system.

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using both Multus public network and Multus cluster network:

Figure 6.3. Cluster network architecture using both Multus public network and Multus cluster network

When both public and cluster networks are used simultaneously, all three benefits of Multus are achieved. In this case, each storage network performs a specific role in the OpenShift Data Foundation cluster. The public network handles only the I/O traffic from applications and the cluster network handles storage replication and recovery traffic.

The cluster network handles data replication traffic on a regular basis. Assuming OpenShift Data Foundation’s default 3x replication, the cluster network handles 2x the application I/O traffic for each data write to bring the total number of storage data copies up to 3x. During a storage node or zone failure event, the cluster network also handles recovering data in the cluster.

In this architecture, the size of the cluster network is recommended to be 2-3x larger than the public network. Larger sizing keeps the failover times short, but it also results in a large amount of bandwidth being wasted during normal cluster operations. Overall storage bandwidth during a storage failure event might be limited by either the public or cluster network depending on the size of either network.

This architecture is more complicated than the recommended public-only architecture and has many more factors that could affect user SLOs. Hence, as an alternative, OpenShift Data Foundation suggests bonding all storage network interfaces together to create a single, larger public network instead.

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using Multus cluster network:

Figure 6.4. Cluster network architecture

In the cluster network architecture, Multus benefits are partially achieved. The cluster network takes the replication and recovery roles. The OpenShift Pod network takes the role of handling storage I/O from applications. This ensures noisy-neighbor interference is minimized, but application I/O can still be affected by other application networking, and vice versa.

Application I/O to the storage system is expected to have higher latency than other Multus architectures due to the software-defined Pod network. Similarly, the storage network is not isolated from applications. For example, a privileged Pod could still snoop storage traffic.

The general requirements for Multus networking are as follows:

OpenShift Data Foundation with Multus creates many virtual MAC addresses on each storage network host interface. This requires network cards and switch hardware to support promiscuous mode. Some smart switches have security processes that inspect each packet, and these switches can bog down significantly due to the high number of MAC addresses for OpenShift Data Foundation. It is best to disable smart switch security features that require advanced processing.

Similarly, virtual or software switches might struggle to process the large number of MAC addresses. Some are not able to disable this functionality. Therefore, do not use virtual switches without a support exception.

For OpenShift Data Foundation Persistent Volume Claims (PVCs) to work properly, the Ceph-CSI driver pods deployed to each OpenShift node (workers and storage) must use host networking. Ceph-CSI drivers must also be able to reach OpenShift Data Foundation Ceph cluster through the Multus public network. This means that each OpenShift node must have a connection to the public network, and vice versa.

To make this as simple as possible without risk of IP overlap issues, select a different subnet (IP range) for node connections to the public network and Pod connections to the public network. Set up routing on the node public subnet and Pod public subnet such that they route to each other. The final result is that the node public subnet and Pod public subnet will act together as a single contiguous public network.

If the cluster network is used, do not route the Pod public network or node public network to the cluster network. The cluster network should be independent and isolated. The diagram below illustrates subnet routing requirements:

Figure 6.5. Subnet routing requirements

OpenShift Data Foundation with Multus requires an IP address for each OpenShift Data Foundation Ceph storage daemon. Networks must have enough IP addresses to account for the number of storage pods that gets attached to the network, and some additional space to account for failover events. Expanding networks in the future might require downtime, so allocating space for the largest foreseeable cluster size is recommended.

For Multus public network, the following ranges must be planned:

For Multus cluster network, the following range must be planned:

Red Hat Ceph underpins OpenShift Data Foundation. Red Hat Ceph networking recommendations are useful for understanding more about OpenShift Data Foundation Multus networking and for finding specific recommendations. For more information, see link: Network considerations for Red Hat Ceph Storage that also applies to OpenShift Data Foundation networking.

The following table recommends IP ranges for each Multus network or subnetwork:

This should suffice for most organizations. The recommendation uses the last 6.25% (1/16th) of the reserved private address space (192.168.0.0/16). Approximate maximum ODF and OpenShift sizing is noted, which accounts for 25% overhead to gracefully handle failure events.

The following diagram shows example interfaces and connections that might be planned for each node type:

Figure 6.6. Example interfaces and connections

This example includes cluster network for completeness.

In this example, vlan220 (public) and vlan221 (cluster) are the interfaces to use for OpenShift Data Foundation configuration. If VLANs are unused in the deployed environment, bond1 and bond2 would be used instead. If no bonds or VLANs are used, host interfaces (for example, eth2, eth4) would be used instead. When using Whereabouts IPAM, IP addresses are not required on Multus parent interfaces such as NICs, bonds, or VLANs (for example, eth2–eth4, bond1–bond2, vlan220–vlan221).

For an OpenShift Data Foundation Multus architecture using only the public network (recommended for most users), the same nodes from the example above can be used. eth0, eth1, eth2, and eth4 would be bonded as bond1 and tagged as vlan220. This architecture ensures that storage nodes have additional available bandwidth for replication and recovery needed by OpenShift Data Foundation Ceph OSDs. vlan220 is be used for OpenShift Data Foundation configuration. However, many variations of valid environments exist.

Figure 6.7. Recommended host interface connection

There is no requirement to dedicate extra public network bandwidth to storage nodes, though this is beneficial. eth4 and eth5 can be omitted from storage nodes entirely to dedicate the same bandwidth to both worker and storage nodes. This design has been commonly seen in customer environments.

Alternatively, eth2 and eth3 can be higher-throughput interfaces on storage nodes (for example, 10Gbps on worker nodes and 25Gbps on storage nodes). This allows for more storage node bandwidth while still using only two interfaces on storage nodes.

The recommended Pod public network configuration for the example is as follow. Omit this if the Multus public network is not used. Make sure to include the routes section that allows Pods on the public network to reach nodes through the public network.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: public-net
  namespace: openshift-storage
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "vlan220", # host public network interface
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.240.0/21", # pod public network range
        "routes": [
          {"dst": "192.168.252.0/23"} # node public network range
        ]
      }
    }

The recommended Pod cluster network configuration for the example is as follows. Omit this if the Multus cluster network is not used (recommended for most users). This configuration should not have a routes section.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: cluster-net
  namespace: openshift-storage
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "vlan221", # host cluster network interface
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.248.0/22" # pod cluster network range
      }
    }

OpenShift worker and storage nodes must be configured to route host traffic to the Pods on the public network through the host public network interface.

The recommended way to configure nodes is using OpenShift NodeNetworkConfigurationPolicy objects. This method can be supported for all OpenShift users regardless of deployment strategy. This method requires the NMState Operator to be installed and enabled. For more information, see link: Kubernetes NMState Operator.

Each node must obtain an IP address on the ODF public network in the node public network address range. Static IP address management is the only IPAM method that can be supported for any OpenShift cluster. Thus, static management is OpenShift Data Foundation supports only the static management method. This requires one NodeNetworkConfigurationPolicy object per host. The template that can be used to configure a host is shown below.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ceph-public-net-shim-{{NODE_NAME}}
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
    kubernetes.io/hostname: {{NODE_NAME}}
  desiredState:
    interfaces:
      - name: odf-pub-shim
        description: Shim interface to connect to ODF public network
        type: mac-vlan
        state: up
        mac-vlan:
          base-iface: vlan220 # host public network interface
          mode: bridge
          promiscuous: true
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 192.168.252.1 # static IP in node public network range
              prefix-length: 23 # node public network range mask
    routes:
      config:
        - destination: 192.168.240.0/21 # pod public network range
          next-hop-interface: odf-pub-shim

First, follow comments in the template to update the base template for the environment being deployed. Then, for each node, copy the template, and fill in {{NODE_NAME}} and a unique static IP for each node.

In addition to the OpenShift Data Foundation health checking, ensure that Multus is configured as expected by performing the following: