红帽全球峰会3.3. Rook-Ceph operator
Rook-Ceph operator is the Rook operator for Ceph in the OpenShift Data Foundation. Rook enables Ceph storage systems to run on the OpenShift Container Platform.
The Rook-Ceph operator is a simple container that automatically bootstraps the storage clusters and monitors the storage daemons to ensure the storage clusters are healthy.
3.3.1. Components
The Rook-Ceph operator manages a number of components as part of the OpenShift Data Foundation deployment.
- Ceph-CSI Driver
-
The operator creates and updates the CSI driver, including a provisioner for each of the two drivers, RADOS block device (RBD) and Ceph filesystem (CephFS) and a volume plugin
daemonsetfor each of the two drivers. - Ceph daemons
- Mons
- The monitors (mons) provide the core metadata store for Ceph.
- OSDs
- The object storage daemons (OSDs) store the data on underlying devices.
- Mgr
- The manager (mgr) collects metrics and provides other internal functions for Ceph.
- RGW
- The RADOS Gateway (RGW) provides the S3 endpoint to the object store.
- MDS
- The metadata server (MDS) provides CephFS shared volumes.
3.3.2. Design diagram
The following image illustrates how Ceph Rook integrates with OpenShift Container Platform.
图 3.3. Rook-Ceph Operator
With Ceph running in the OpenShift Container Platform cluster, OpenShift Container Platform applications can mount block devices and filesystems managed by Rook-Ceph, or can use the S3/Swift API for object storage.
3.3.3. Responsibilities
The Rook-Ceph operator is a container that bootstraps and monitors the storage cluster. It performs the following functions:
- Automates the configuration of storage components
- Starts, monitors, and manages the Ceph monitor pods and Ceph OSD daemons to provide the RADOS storage cluster
Initializes the pods and other artifacts to run the services to manage:
- CRDs for pools
- Object stores (S3/Swift)
- Filesystems
- Monitors the Ceph mons and OSDs to ensure that the storage remains available and healthy
- Deploys and manages Ceph mons placement while adjusting the mon configuration based on cluster size
- Watches the desired state changes requested by the API service and applies the changes
- Initializes the Ceph-CSI drivers that are needed for consuming the storage
- Automatically configures the Ceph-CSI driver to mount the storage to pods
Rook-Ceph Operator architecture
The Rook-Ceph operator image includes all required tools to manage the cluster. There is no change to the data path. However, the operator does not expose all Ceph configurations. Many of the Ceph features like placement groups and crush maps are hidden from the users and are provided with a better user experience in terms of physical resources, pools, volumes, filesystems, and buckets.
3.3.4. Resources
Rook-Ceph operator adds owner references to all the resources it creates in the openshift-storage namespace. When the cluster is uninstalled, the owner references ensure that the resources are all cleaned up. This includes OpenShift Container Platform resources such as configmaps, secrets, services, deployments, daemonsets, and so on.
The Rook-Ceph operator watches CRs to configure the settings determined by OpenShift Data Foundation, which includes CephCluster, CephObjectStore, CephFilesystem, and CephBlockPool.
3.3.5. Lifecycle
Rook-Ceph operator manages the lifecycle of the following pods in the Ceph cluster:
- Rook operator
- A single pod that owns the reconcile of the cluster.
- RBD CSI Driver
- Two provisioner pods, managed by a single deployment.
-
One plugin pod per node, managed by a
daemonset.
- CephFS CSI Driver
- Two provisioner pods, managed by a single deployment.
-
One plugin pod per node, managed by a
daemonset.
- Monitors (mons)
Three mon pods, each with its own deployment.
- Stretch clusters
- Contain five mon pods, one in the arbiter zone and two in each of the other two data zones.
- Manager (mgr)
There is a single mgr pod for the cluster.
- Stretch clusters
- There are two mgr pods (starting with OpenShift Data Foundation 4.8), one in each of the two non-arbiter zones.
- Object storage daemons (OSDs)
- At least three OSDs are created initially in the cluster. More OSDs are added when the cluster is expanded.
- Metadata server (MDS)
- The CephFS metadata server has a single pod.
- RADOS gateway (RGW)
- The Ceph RGW daemon has a single pod.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}