红帽全球峰会4.3. Storage cluster creation
The OpenShift Data Foundation operators themselves provide no storage functionality, and the desired storage configuration must be defined.
After you install the operators, create a new StorageCluster, using either the OpenShift Container Platform console wizard or the CLI and the ocs-operator reconciles this StorageCluster. OpenShift Data Foundation supports a single StorageCluster per installation. Any StorageCluster CRs created after the first one is ignored by ocs-operator reconciliation.
OpenShift Data Foundation allows the following StorageCluster configurations:
- Internal
-
In the Internal mode, all the components run containerized within the OpenShift Container Platform cluster and uses dynamically provisioned persistent volumes (PVs) created against the
StorageClassspecified by the administrator in the installation wizard. - Internal-attached
-
This mode is similar to the Internal mode but the administrator is required to define the local storage devices directly attached to the cluster nodes that the Ceph uses for its backing storage. Also, the administrator need to create the CRs that the local storage operator reconciles to provide the
StorageClass. Theocs-operatoruses thisStorageClassas the backing storage for Ceph. - External
- In this mode, Ceph components do not run inside the OpenShift Container Platform cluster instead connectivity is provided to an external OpenShift Container Storage installation for which the applications can create PVs. The other components run within the cluster as required.
- MCG Standalone
- This mode facilitates the installation of a Multicloud Object Gateway system without an accompanying CephCluster.
After a StorageCluster CR is found, ocs-operator validates it and begins to create subsequent resources to define the storage components.
4.3.1. Internal mode storage cluster
Both internal and internal-attached storage clusters have the same setup process as follows:
|
| Create the storage classes that cluster applications use to create Ceph volumes. |
|
| Create the volume snapshot classes that the cluster applications use to create snapshots of Ceph volumes. |
| Ceph RGW configuration | Create various Ceph object CRs to enable and provide access to the Ceph RGW object storage endpoint. |
| Ceph RBD Configuration |
Create the |
| CephFS Configuration |
Create the |
| Rook-Ceph Configuration |
Create the |
|
|
Create the |
|
|
Create the |
| Job templates |
Create |
| Quickstarts |
Create the |
The RGW (RADOS Gateway) component is deployed only in on-premises environments. It is not created for cloud-based deployments.
4.3.1.1. Cluster Creation
After the ocs-operator creates the CephCluster CR, the rook-operator creates the Ceph cluster according to the desired configuration. The rook-operator configures the following components:
|
Ceph |
Three Ceph |
|
Ceph | This daemon is started and it gathers metrics for the cluster and report them to Prometheus. |
| Ceph OSDs |
These OSDs are created according to the configuration of the |
| CSI provisioners |
These provisioners are started for RBD and |
|
CSI volume plugins and |
The CSI volume plugins for RBD and |
After the CephCluster CR is configured, Rook reconciles the remaining Ceph CRs to complete the setup:
|
|
The |
|
|
The |
|
|
The |
|
|
The |
The operator monitors the Ceph health to ensure that storage platform remains healthy. If a mon daemon goes down for too long a period (10 minutes), Rook starts a new mon in its place so that the full quorum can be fully restored.
When the ocs-operator updates the CephCluster CR, Rook immediately responds to the requested changes to update the cluster configuration.
4.3.1.2. NooBaa System creation
When a NooBaa system is created, the mcg-operator reconciles the following:
Default BackingStore
Depending on the platform that OpenShift Container Platform and OpenShift Data Foundation are deployed on, a default backing store resource is created so that buckets can use it for their placement policy. The different options are as follows:
| Amazon Web Services (AWS) deployment |
The |
| Microsoft Azure deployment |
The |
| Google Cloud Platform (GCP) deployment |
The |
| On-prem deployment |
If RGW exists, the |
| None of the previously mentioned deployments are applicable |
The |
Default BucketClass
A BucketClass with a placement policy to the default BackingStore is created.
NooBaa pods
The following NooBaa pods are created and started:
| Database (DB) | This is a Postgres DB holding metadata, statistics, events, and so on. However, it does not hold the actual data being stored. |
| Core | This is the pod that handles configuration, background processes, metadata management, statistics, and so on. |
| Endpoints |
These pods perform the actual I/O-related work such as deduplication and compression, communicating with different services to write and read data, and so on. The endpoints are integrated with the |
Route
A Route for the NooBaa S3 interface is created for applications that uses S3.
Service
A Service for the NooBaa S3 interface is created for applications that uses S3.
4.3.2. External mode storage cluster
For external storage clusters, ocs-operator follows a slightly different setup process. The ocs-operator looks for the existence of the rook-ceph-external-cluster-details ConfigMap, which must be created by someone else, either the administrator or the Console. For information about how to create the ConfigMap, see Creating an OpenShift Data Foundation Cluster for external mode. The ocs-operator then creates some or all of the following resources, as specified in the ConfigMap:
| External Ceph Configuration |
A ConfigMap that specifies the endpoints of the external |
| External Ceph Credentials Secret | A Secret that contains the credentials to connect to the external Ceph instance. |
| External Ceph StorageClasses | One or more StorageClasses to enable the creation of volumes for RBD, CephFS, and/or RGW. |
| Enable CephFS CSI Driver |
If a |
| Ceph RGW Configuration | If an RGW StorageClass is specified, create various Ceph Object CRs to enable and provide access to the Ceph RGW object storage endpoint. |
After creating the resources specified in the ConfigMap, the StorageCluster creation process proceeds as follows:
|
|
Create the |
|
|
Create the |
|
|
Create the |
|
|
Create the |
4.3.2.1. Cluster Creation
The Rook operator performs the following operations when the CephCluster CR is created in external mode:
-
The operator validates that a connection is available to the remote Ceph cluster. The connection requires
monendpoints and secrets to be imported into the local cluster. -
The CSI driver is configured with the remote connection to Ceph. The RBD and
CephFSprovisioners and volume plugins are started similarly to the CSI driver when configured in internal mode, the connection to Ceph happens to be external to the OpenShift cluster. - Periodically watch for monitor address changes and update the Ceph-CSI configuration accordingly.
4.3.2.2. NooBaa System creation
When a NooBaa system is created, the mcg-operator reconciles the following:
Default BackingStore
Depending on the platform that OpenShift Container Platform and OpenShift Data Foundation are deployed on, a default backing store resource is created so that buckets can use it for their placement policy. The different options are as follows:
| Amazon Web Services (AWS) deployment |
The |
| Microsoft Azure deployment |
The |
| Google Cloud Platform (GCP) deployment |
The |
| On-prem deployment |
If RGW exists, the |
| None of the previously mentioned deployments are applicable |
The |
Default BucketClass
A BucketClass with a placement policy to the default BackingStore is created.
NooBaa pods
The following NooBaa pods are created and started:
| Database (DB) | This is a Postgres DB holding metadata, statistics, events, and so on. However, it does not hold the actual data being stored. |
| Core | This is the pod that handles configuration, background processes, metadata management, statistics, and so on. |
| Endpoints |
These pods perform the actual I/O-related work such as deduplication and compression, communicating with different services to write and read data, and so on. The endpoints are integrated with the |
Route
A Route for the NooBaa S3 interface is created for applications that uses S3.
Service
A Service for the NooBaa S3 interface is created for applications that uses S3.
4.3.3. MCG Standalone StorageCluster
In this mode, no CephCluster is created. Instead a NooBaa system CR is created using default values to take advantage of pre-existing StorageClasses in the OpenShift Container Platform. dashboards.
4.3.3.1. NooBaa System creation
When a NooBaa system is created, the mcg-operator reconciles the following:
Default BackingStore
Depending on the platform that OpenShift Container Platform and OpenShift Data Foundation are deployed on, a default backing store resource is created so that buckets can use it for their placement policy. The different options are as follows:
| Amazon Web Services (AWS) deployment |
The |
| Microsoft Azure deployment |
The |
| Google Cloud Platform (GCP) deployment |
The |
| On-prem deployment |
If RGW exists, the |
| None of the previously mentioned deployments are applicable |
The |
Default BucketClass
A BucketClass with a placement policy to the default BackingStore is created.
NooBaa pods
The following NooBaa pods are created and started:
| Database (DB) | This is a Postgres DB holding metadata, statistics, events, and so on. However, it does not hold the actual data being stored. |
| Core | This is the pod that handles configuration, background processes, metadata management, statistics, and so on. |
| Endpoints |
These pods perform the actual I/O-related work such as deduplication and compression, communicating with different services to write and read data, and so on. The endpoints are integrated with the |
Route
A Route for the NooBaa S3 interface is created for applications that uses S3.
Service
A Service for the NooBaa S3 interface is created for applications that uses S3.
4.3.3.2. StorageSystem Creation
As a part of the StorageCluster creation, odf-operator automatically creates a corresponding StorageSystem CR, which exposes the StorageCluster to the OpenShift Data Foundation.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}