Red Hat OpenShift Data Foundation architecture

The odf-operator has a dependency on the ocs-operator package. It also manages the Subscription of the mcg-operator. In addition, the odf-operator bundle defines a second Deployment for the OpenShift Data Foundation external plugin for the OpenShift Console. This defines an nginx-based Pod that serves the necessary files to register and integrate OpenShift Data Foundation dashboards directly into the OpenShift Container Platform Console.

This diagram illustrates how odf-operator is integrated with the OpenShift Container Platform.

Figure 3.1. OpenShift Data Foundation Operator

The odf-operator creates the Operator Lifecycle Manager Resources CR, which creates a Subscription for the operator.

The odf-operator does not provide any data storage or services itself. It exists as an integration and management layer for other storage systems.

High availability is not a primary requirement for the odf-operator Pod similar to most of the other operators. In general, there are no operations that require or benefit from process distribution. OpenShift Container Platform quickly spins up a replacement Pod whenever the current Pod becomes unavailable or is deleted.

The odf-operator comes with a ConfigMap of variables that can be used to modify the behavior of the operator.

To get an understanding of the OpenShift Data Foundation and troubleshoot issues, you can look at the following:

The odf-operator is required to be present as long as the OpenShift Data Foundation bundle remains installed. This is managed as part of OLM’s reconciliation of the OpenShift Data Foundation CSV. At least one instance of the pod should be in Ready state.

The operator operands such as CRDs should not affect the lifecycle of the operator. The creation and deletion of StorageClusters is an operation outside the operator’s control and must be initiated by the administrator or automated with the appropriate application programming interface (API) calls.

The ocs-operator does not have any dependent components. However, the operator has a dependency on the existence of all the custom resource definitions (CRDs) from other operators, which are defined in the ClusterServiceVersion (CSV).

This diagram illustrates how OpenShift Container Storage is integrated with the OpenShift Container Platform.

Figure 3.2. OpenShift Container Storage Operator

The two ocs-operator CRDs are:

OCSInitialization is a singleton CRD used for encapsulating operations that apply at the operator level. The operator takes care of ensuring that one instance always exists. The CR triggers the following:

The StorageCluster CRD represents the system that provides the full functionality of OpenShift Container Storage. It triggers the operator to ensure the generation and reconciliation of Rook-Ceph and NooBaa CRDs. The ocs-operator algorithmically generates the CephCluster and NooBaa CRDs based on the configuration in the StorageCluster spec. The operator also creates additional CRs, such as CephBlockPools, Routes, and so on. These resources are required for enabling different features of OpenShift Container Storage. Currently, only one StorageCluster CR per OpenShift Container Platform cluster is supported.

The ocs-operator creates the following CRs in response to the spec of the CRDs it defines . The configuration of some of these resources can be overridden, allowing for changes to the generated spec or not creating them altogether.

The ocs-operator neither deploys nor reconciles the other Pods of OpenShift Data Foundation. The ocs-operator CSV defines the top-level components such as operator Deployments and the Operator Lifecycle Manager (OLM) reconciles the specified component.

High availability is not a primary requirement for the ocs-operator Pod similar to most of the other operators. In general, there are no operations that require or benefit from process distribution. OpenShift Container Platform quickly spins up a replacement Pod whenever the current Pod becomes unavailable or is deleted.

The ocs-operator configuration is entirely specified by the CSV and is not modifiable without a custom build of the CSV.

To get an understanding of the OpenShift Container Storage and troubleshoot issues, you can look at the following:

The ocs-operator is required to be present as long as the OpenShift Container Storage bundle remains installed. This is managed as part of OLM’s reconciliation of the OpenShift Container Storage CSV. At least one instance of the pod should be in Ready state.

The operator operands such as CRDs should not affect the lifecycle of the operator. An OCSInitialization CR should always exist. The operator creates one if it does not exist. The creation and deletion of StorageClusters is an operation outside the operator’s control and must be initiated by the administrator or automated with the appropriate API calls.

The Rook-Ceph operator manages a number of components as part of the OpenShift Data Foundation deployment.

The following image illustrates how Ceph Rook integrates with OpenShift Container Platform.

Figure 3.3. Rook-Ceph Operator

With Ceph running in the OpenShift Container Platform cluster, OpenShift Container Platform applications can mount block devices and filesystems managed by Rook-Ceph, or can use the S3/Swift API for object storage.

The Rook-Ceph operator is a container that bootstraps and monitors the storage cluster. It performs the following functions:

The Rook-Ceph operator image includes all required tools to manage the cluster. There is no change to the data path. However, the operator does not expose all Ceph configurations. Many of the Ceph features like placement groups and crush maps are hidden from the users and are provided with a better user experience in terms of physical resources, pools, volumes, filesystems, and buckets.

Rook-Ceph operator adds owner references to all the resources it creates in the openshift-storage namespace. When the cluster is uninstalled, the owner references ensure that the resources are all cleaned up. This includes OpenShift Container Platform resources such as configmaps, secrets, services, deployments, daemonsets, and so on.

The Rook-Ceph operator watches CRs to configure the settings determined by OpenShift Data Foundation, which includes CephCluster, CephObjectStore, CephFilesystem, and CephBlockPool.

Rook-Ceph operator manages the lifecycle of the following pods in the Ceph cluster:

The MCG operator does not have sub-components. However, it consists of a reconcile loop for the different resources that are controlled by it.

The MCG operator has a command-line interface (CLI) and is available as a part of OpenShift Data Foundation. It enables the creation, deletion, and querying of various resources. This CLI adds a layer of input sanitation and status validation before the configurations are applied unlike applying a YAML file directly.

The MCG operator reconciles and is responsible for the custom resource definitions (CRDs) and OpenShift Container Platform entities.

A default backing store is created as part of the deployment depending on the platform that the OpenShift Container Platform is running on. For example, when OpenShift Container Platform or OpenShift Data Foundation is deployed on Amazon Web Services (AWS), it results in a default backing store which is an AWS::S3 bucket. Similarly, for Microsoft Azure, the default backing store is a blob container and so on.

The default backing stores are created using CRDs for the cloud credential operator, which comes with OpenShift Container Platform. There is no limit on the amount of the backing stores that can be added to MCG. The backing stores are used in the bucket class CRD to define the different policies of the bucket. Refer the documentation of the specific OpenShift Data Foundation version to identify the types of services or resources supported as backing stores.

A default bucket class is created during deployment and it is set with a placement policy that uses the default backing store. There is no limit to the number of bucket class that can be added.

Refer to the documentation of the specific OpenShift Data Foundation version to identify the types of policies that are supported.

As an operator, the only high availability provided is that the OpenShift Container Platform reschedules a failed pod.

To troubleshoot issues with the NooBaa operator, you can look at the following:

The MCG operator runs and reconciles after OpenShift Data Foundation is deployed and until it is uninstalled.

Both internal and internal-attached storage clusters have the same setup process as follows:

For external storage clusters, ocs-operator follows a slightly different setup process. The ocs-operator looks for the existence of the rook-ceph-external-cluster-details ConfigMap, which must be created by someone else, either the administrator or the Console. For information about how to create the ConfigMap, see Creating an OpenShift Data Foundation Cluster for external mode. The ocs-operator then creates some or all of the following resources, as specified in the ConfigMap:

After creating the resources specified in the ConfigMap, the StorageCluster creation process proceeds as follows:

In this mode, no CephCluster is created. Instead a NooBaa system CR is created using default values to take advantage of pre-existing StorageClasses in the OpenShift Container Platform. dashboards.

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using the Multus public network:

Figure 6.2. OpenShift Data Foundation deployment using Multus public network

This architecture achieves all the three benefits of Multus networking:

However, if the public network has less throughput than the Pod network, storage traffic might not be faster than a traditional deployment.

Public network is the recommended architecture for the majority of OpenShift Data Foundation cluster deployments. In this architecture, all storage traffic uses the dedicated public network, which includes I/O traffic from applications, data replication traffic within the storage system, and data recovery traffic also within the storage system.

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using both Multus public network and Multus cluster network:

Figure 6.3. Cluster network architecture using both Multus public network and Multus cluster network

When both public and cluster networks are used simultaneously, all three benefits of Multus are achieved. In this case, each storage network performs a specific role in the OpenShift Data Foundation cluster. The public network handles only the I/O traffic from applications and the cluster network handles storage replication and recovery traffic.

The cluster network handles data replication traffic on a regular basis. Assuming OpenShift Data Foundation’s default 3x replication, the cluster network handles 2x the application I/O traffic for each data write to bring the total number of storage data copies up to 3x. During a storage node or zone failure event, the cluster network also handles recovering data in the cluster.

In this architecture, the size of the cluster network is recommended to be 2-3x larger than the public network. Larger sizing keeps the failover times short, but it also results in a large amount of bandwidth being wasted during normal cluster operations. Overall storage bandwidth during a storage failure event might be limited by either the public or cluster network depending on the size of either network.

This architecture is more complicated than the recommended public-only architecture and has many more factors that could affect user SLOs. Hence, as an alternative, OpenShift Data Foundation suggests bonding all storage network interfaces together to create a single, larger public network instead.

The following diagram shows the cluster network architecture for an OpenShift Data Foundation deployment using Multus cluster network:

Figure 6.4. Cluster network architecture

In the cluster network architecture, Multus benefits are partially achieved. The cluster network takes the replication and recovery roles. The OpenShift Pod network takes the role of handling storage I/O from applications. This ensures noisy-neighbor interference is minimized, but application I/O can still be affected by other application networking, and vice versa.

Application I/O to the storage system is expected to have higher latency than other Multus architectures due to the software-defined Pod network. Similarly, the storage network is not isolated from applications. For example, a privileged Pod could still snoop storage traffic.

The general requirements for Multus networking are as follows:

OpenShift Data Foundation with Multus creates many virtual MAC addresses on each storage network host interface. This requires network cards and switch hardware to support promiscuous mode. Some smart switches have security processes that inspect each packet, and these switches can bog down significantly due to the high number of MAC addresses for OpenShift Data Foundation. It is best to disable smart switch security features that require advanced processing.

Similarly, virtual or software switches might struggle to process the large number of MAC addresses. Some are not able to disable this functionality. Therefore, do not use virtual switches without a support exception.

For OpenShift Data Foundation Persistent Volume Claims (PVCs) to work properly, the Ceph-CSI driver pods deployed to each OpenShift node (workers and storage) must use host networking. Ceph-CSI drivers must also be able to reach OpenShift Data Foundation Ceph cluster through the Multus public network. This means that each OpenShift node must have a connection to the public network, and vice versa.

To make this as simple as possible without risk of IP overlap issues, select a different subnet (IP range) for node connections to the public network and Pod connections to the public network. Set up routing on the node public subnet and Pod public subnet such that they route to each other. The final result is that the node public subnet and Pod public subnet will act together as a single contiguous public network.

If the cluster network is used, do not route the Pod public network or node public network to the cluster network. The cluster network should be independent and isolated. The diagram below illustrates subnet routing requirements:

Figure 6.5. Subnet routing requirements

OpenShift Data Foundation with Multus requires an IP address for each OpenShift Data Foundation Ceph storage daemon. Networks must have enough IP addresses to account for the number of storage pods that gets attached to the network, and some additional space to account for failover events. Expanding networks in the future might require downtime, so allocating space for the largest foreseeable cluster size is recommended.

For Multus public network, the following ranges must be planned:

For Multus cluster network, the following range must be planned:

Red Hat Ceph underpins OpenShift Data Foundation. Red Hat Ceph networking recommendations are useful for understanding more about OpenShift Data Foundation Multus networking and for finding specific recommendations. For more information, see link: Network considerations for Red Hat Ceph Storage that also applies to OpenShift Data Foundation networking.

The following table recommends IP ranges for each Multus network or subnetwork:

This should suffice for most organizations. The recommendation uses the last 6.25% (1/16th) of the reserved private address space (192.168.0.0/16). Approximate maximum ODF and OpenShift sizing is noted, which accounts for 25% overhead to gracefully handle failure events.

The following diagram shows example interfaces and connections that might be planned for each node type:

Figure 6.6. Example interfaces and connections

This example includes cluster network for completeness.

In this example, vlan220 (public) and vlan221 (cluster) are the interfaces to use for OpenShift Data Foundation configuration. If VLANs are unused in the deployed environment, bond1 and bond2 would be used instead. If no bonds or VLANs are used, host interfaces (for example, eth2, eth4) would be used instead.OpenShift Data Foundation

For an OpenShift Data Foundation Multus architecture using only the public network (recommended for most users), the same nodes from the example above can be used. eth0, eth1, eth2, and eth4 would be bonded as bond1 and tagged as vlan220. This architecture ensures that storage nodes have additional available bandwidth for replication and recovery needed by OpenShift Data Foundation Ceph OSDs. vlan220 is be used for OpenShift Data Foundation configuration. However, many variations of valid environments exist.

Figure 6.7. Recommended host interface connection

There is no requirement to dedicate extra public network bandwidth to storage nodes, though this is beneficial. eth4 and eth5 can be omitted from storage nodes entirely to dedicate the same bandwidth to both worker and storage nodes. This design has been commonly seen in customer environments.

Alternatively, eth2 and eth3 can be higher-throughput interfaces on storage nodes (for example, 10Gbps on worker nodes and 25Gbps on storage nodes). This allows for more storage node bandwidth while still using only two interfaces on storage nodes.

The recommended Pod public network configuration for the example is as follow. Omit this if the Multus public network is not used. Make sure to include the routes section that allows Pods on the public network to reach nodes through the public network.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: public-net
  namespace: openshift-storage
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "vlan220", # host public network interface
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.240.0/21", # pod public network range
        "routes": [
          {"dst": "192.168.252.0/23"} # node public network range
        ]
      }
    }

The recommended Pod cluster network configuration for the example is as follows. Omit this if the Multus cluster network is not used (recommended for most users). This configuration should not have a routes section.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: cluster-net
  namespace: openshift-storage
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "vlan221", # host cluster network interface
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.248.0/22" # pod cluster network range
      }
    }

OpenShift worker and storage nodes must be configured to route host traffic to the Pods on the public network through the host public network interface.

The recommended way to configure nodes is using OpenShift NodeNetworkConfigurationPolicy objects. This method can be supported for all OpenShift users regardless of deployment strategy. This method requires the NMState Operator to be installed and enabled. For more information, see link: Kubernetes NMState Operator.

Each node must obtain an IP address on the ODF public network in the node public network address range. Static IP address management is the only IPAM method that can be supported for any OpenShift cluster. Thus, static management is OpenShift Data Foundation supports only the static management method. This requires one NodeNetworkConfigurationPolicy object per host. The template that can be used to configure a host is shown below.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ceph-public-net-shim-{{NODE_NAME}}
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
    kubernetes.io/hostname: {{NODE_NAME}}
  desiredState:
    interfaces:
      - name: odf-pub-shim
        description: Shim interface to connect to ODF public network
        type: mac-vlan
        state: up
        mac-vlan:
          base-iface: vlan220 # host public network interface
          mode: bridge
          promiscuous: true
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 192.168.252.1 # static IP in node public network range
              prefix-length: 23 # node public network range mask
    routes:
      config:
        - destination: 192.168.240.0/21 # pod public network range
          next-hop-interface: odf-pub-shim

First, follow comments in the template to update the base template for the environment being deployed. Then, for each node, copy the template, and fill in {{NODE_NAME}} and a unique static IP for each node.

In addition to the OpenShift Data Foundation health checking, ensure that Multus is configured as expected by performing the following: