Chapter 2. Deploy OpenShift Data Foundation using Red Hat Ceph storage

Procedure

1. Log in to the OpenShift Web Console.
2. Click Ecosystem Software Catalog.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.22.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. In the OpenShift Web Console, click Storage External Systems.
2. Click Connect external systems.
3. Select Red Hat/IBM Ceph Cluster and click Next.

4. In the Connect Red Hat Ceph storage page, provide the necessary information:

   1. Click on the Download Script link to download the python script for extracting Ceph cluster details.

   2. For extracting the Red Hat Ceph Storage (RHCS) cluster details, contact the RHCS administrator to run the downloaded python script on a Red Hat Ceph Storage node with the admin key.

      1. Run the following command on the RHCS node to view the list of available arguments:

         # python3 ceph-external-cluster-details-exporter.py --help

         You can also run the script from inside a MON container (containerized deployment) or from a MON node (RPM deployment).

         Note

         Use the yum install cephadm command and then the cephadm command to deploy your RHCS cluster using containers. You must pull the RHCS container images using the cephadm command, rather than using yum for installing the Ceph packages onto nodes. For more information, see RHCS product documentation.

      2. To retrieve the external cluster details from the RHCS cluster, choose one of the following two options, either a configuration file or command-line flags.

         1. Configuration file

            Use config-file flag. This stores the parameters used during deployment.

            In new deployments, you can save the parameters used during deployment in a configuration file. This file can then be used during upgrade to preserve the parameters as well as adding any additional parameters. Use config-file to set the path to the configuration file.

            Example configuration file saved in /config.ini:

            [Configurations]
            format = bash
            cephfs-filesystem-name = <filesystem-name>
            rbd-data-pool-name = <pool_name>
            ...

            Set the path to the config.ini file using config-file:

            # python3 ceph-external-cluster-details-exporter.py --config-file /config.ini

         2. Command-line flags

            Retrieve the external cluster details from the RHCS cluster, and pass the parameters for your deployment.

            # python3 ceph-external-cluster-details-exporter.py \
            --rbd-data-pool-name <rbd block pool name>  [optional arguments]

            For example:

            # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

            RBD parameters

            rbd-data-pool-name

            A mandatory parameter that is used for providing block storage in OpenShift Data Foundation.

            rados-namespace

            Divides an RBD data pool into separate logical namespaces, used for creating RBD PVC in a radosNamespace. Flags required with rados-namespace are restricted-auth-permission and k8s-cluster-name.

            rbd-metadata-ec-pool-name

            (Optional) The name of the erasure coded RBD metadata pool.

            RGW parameters

            rgw-endpoint

            (Optional) This parameter is required only if the object storage is to be provisioned through Ceph Rados Gateway for OpenShift Data Foundation. Provide the endpoint in the following format: <ip_address>:<port>

            Note

            A fully-qualified domain name (FQDN) is also supported in the format <FQDN>:<PORT>.

            rgw-pool-prefix

            (Optional) The prefix of the RGW pools. If not specified, the default prefix is default.

            rgw-tls-cert-path

            (Optional) The file path of the RADOS Gateway endpoint TLS certificate.

            • To provide the TLS certificate and RGW endpoint details to the helper script, ceph-external-cluster-details-exporter.py, run the following command:

              # python3 ceph-external-clustergw-endpoint r-details-exporter.py --rbd-data-pool-name <rbd block pool name> --rgw-endpoint <ip_address>:<port> --rgw-tls-cert-path <file path containing cert>

              This creates a resource to create a Ceph Object Store CR such as Kubernetes Secret containing the TLS certificate. All the intermediate certificates including private keys need to be stored in the certificate file.

            rgw-skip-tls

            (Optional) This parameter ignores the TLS certification validation when a self-signed certificate is provided (not recommended).

            Monitoring parameters

            monitoring-endpoint

            (Optional) This parameter accepts comma-separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.

            monitoring-endpoint-port

            (Optional) It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.

            Ceph parameters

            ceph-conf

            (Optional) The name of the Ceph configuration file.

            run-as-user

            (Optional) This parameter is used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as:

            • caps: [mgr] allow command config
            • caps: [mon] allow r, allow command quorum_status, allow command version
            • caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index

            CephFS parameters

            cephfs-metadata-pool-name

            (Optional) The name of the CephFS metadata pool.

            cephfs-data-pool-name

            (Optional) The name of the CephFS data pool.

            cephfs-filesystem-name

            (Optional) The name of the CephFS filesystem.

            Output parameters

            dry-run

            (Optional) This parameter helps to print the executed commands without running them.

            output

            (Optional) The file where the output is required to be stored.

            Multicluster parameters

            k8s-cluster-name

            (Optional) Kubernetes cluster name.

            cluster-name

            (Optional) The Ceph cluster name.

            restricted-auth-permission

            (Optional) This parameter restricts cephCSIKeyrings auth permissions to specific pools and clusters. Mandatory flags that need to be set with this are rbd-data-pool-name and cluster-name. You can also pass the cephfs-filesystem-name flag if there is CephFS user restriction so that permission is restricted to a particular CephFS filesystem.

            Note

            This parameter must be applied only for the new deployments. To restrict csi-users per pool and per cluster, you need to create new csi-users and new secrets for those csi-users.

            Example with restricted auth permission:

            # python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

            Example of JSON output generated using the python script:

            [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

      3. Save the JSON output to a file with .json extension

         Note

         For OpenShift Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) to be uploaded using the JSON file remain unchanged on the RHCS external cluster after the storage cluster creation.

      4. Run the command when there is a multi-tenant deployment in which the RHCS cluster is already connected to OpenShift Data Foundation deployment with a lower version.

         # python3 ceph-external-cluster-details-exporter.py --upgrade

   3. Click Browse to select and upload the JSON file.

      The content of the JSON file is populated and displayed in the text box.

   4. (Optional) Select In-transit encryption to enable encryption for entire Ceph traffic including data using Ceh msgrv2.

      Note

      Verify that the external Red Hat Ceph Storage (RHCS) cluster is configured for in‑transit encryption if required. Follow the documentation for detailed configuration steps before enabling in‑transit encryption.

   5. (Optional) Select the advanced configuration options that apply to your environment:

      1. Use Ceph RBD as the default StorageClass

         Sets the RBD StorageClass as the cluster default.

      2. Use external PostgreSQL (Technology preview)

         Uses an external PostgreSQL instance instead of the in‑cluster database.

         Additional advanced options may be available in the UI. Select only those that are required for the deployment policy and environment.

      3. Enable automatic backup

         Allows automatic backup for the Multicloud Object Gateway metadata database.

         1. Select the backup frequency: Daily, Weekly, or Monthly.

         2. Specify the number of backups to retain.

            Note

            This option is disabled if the Use external PostgreSQL option is selected.

5. Review the selections and click Connect.

1. In the OpenShift Web Console, navigate to Storage Data Foundation Storage System ocs-external-storagecluster.
2. Verify that Status of StorageCluster is Ready and has a green tick.
3. To verify that OpenShift Data Foundation, pods and StorageClass are successfully installed, see Verifying your external mode OpenShift Data Foundation installation for external Ceph storage system.

Procedure

1. Identify the Ceph failure domain and the corresponding OpenShift nodes.

   1. Check the Ceph failure domain:

      $ ceph osd tree

      Example output:

      ID  CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
      ...
      -9         0.50000      host osd-0
       3    hdd  0.25000          osd.3       up   1.00000  1.00000
      ...

      In this example, the failure domain is host, and the device class is hdd.

   2. Identify the OpenShift worker nodes and map them to the Ceph failure domains:

      $ oc get nodes

      Example output:

      NAME              STATUS   ROLES    AGE     VERSION
      compute-0         Ready    worker   2d23h   v1.34.1
      compute-1         Ready    worker   2d23h   v1.34.1
      compute-2         Ready    worker   2d23h   v1.34.1

   3. Verify the node labels that you want to use for topology awareness:

      $ oc get nodes --show-labels

      In this example, the label topology.kubernetes.io/hostname and its values are chosen, for example, compute-0, compute-1, and compute-2, when configuring topology-aware external cluster resources.

2. Create separate block pools for each failure domain.

   In this example, a replica-3 pool is created for each failure domain.

   $ ceph osd pool create pool-1-data 32 32 replicated
   $ ceph osd pool set pool-1-data size 3
   $ ceph osd pool create pool-2-data 32 32 replicated
   $ ceph osd pool set pool-2-data size 3
   $ ceph osd pool create pool-3-data 32 32 replicated
   $ ceph osd pool set pool-3-data size 3

   Verify the pools:

   $ ceph osd pool ls detail

3. Generate the external cluster resources with topology-aware settings.

   Run the following command on the Ceph client:

   $ python3 /etc/rook-external/create-external-cluster-resources.py \
     --rbd-data-pool-name rbd \
     --topology-pools pool-1-data,pool-2-data,pool-3-data \
     --topology-failure-domain-label host \
     --topology-failure-domain-values compute-0,compute-1,compute-2

   where

   --rbd-data-pool-name

   Specifies an existing RBD pool.

   --topology-pools

   Specifies the topology-constrained pools created for each failure domain.

   --topology-failure-domain-label

   Specifies the Ceph failure domain label, for example, host.

   --topology-failure-domain-values

   Specifies the corresponding OpenShift failure domain values, for example, compute-0,compute-1,compute-2.

4. Upload the generated JSON file when creating the external cluster in the OpenShift web console.

5. Verify that topology awareness is configured.

   1. Verify that the operator configuration contains the topology domain label:

      $ oc get operatorconfigs.csi.ceph.io -o yaml | grep topology -A3

      Expected output:

      topology:
        domainLabels:
        - kubernetes.io/hostname

   2. Verify that the topology-aware storage class contains the topology parameters:

      $ oc get sc ocs-external-storagecluster-ceph-non-resilient-rbd -o yaml

      Example output:

      parameters:
        topologyConstrainedPools: |-
          [
            {
              "poolName": "pool-1-data",
              "domainSegments": [
                {
                  "domainLabel": "hostname",
                  "value": "compute-0"
                }
              ]
            },
            {
              "poolName": "pool-2-data",
              "domainSegments": [
                {
                  "domainLabel": "hostname",
                  "value": "compute-1"
                }
              ]
            },
            {
              "poolName": "pool-3-data",
              "domainSegments": [
                {
                  "domainLabel": "hostname",
                  "value": "compute-2"
                }
              ]
            }
          ]
        topologyFailureDomainLabel: host
        topologyFailureDomainValues: compute-0,compute-1,compute-2
        topologyPools: pool-1-data,pool-2-data,pool-3-data

6. Test topology-aware provisioning by creating a persistent volume claim (PVC) and a pod.

   Apply the following YAML:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: topo-verify-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
     storageClassName: ocs-external-storagecluster-ceph-non-resilient-rbd
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: topo-verify-pod
   spec:
     containers:
     - name: test-container
       image: busybox
       command: [ "sh", "-c", "sleep 3600" ]
       volumeMounts:
       - mountPath: /data
         name: my-volume
     volumes:
     - name: my-volume
       persistentVolumeClaim:
         claimName: topo-verify-pvc

   The WaitForFirstConsumer volume binding mode ensures that the PV is provisioned only after the pod is scheduled, so topology constraints can be applied.

7. Verify data placement in the expected failure domain.

   1. Get the image name for the provisioned PV:

      $ kubectl get pv $(kubectl get pvc topo-verify-pvc -o jsonpath='{.spec.volumeName}') -o jsonpath='{.spec.csi.volumeAttributes.imageName}'

   2. Check usage from the Ceph side:

      $ rbd du pool-1-data/<image_name>

   3. Identify the OSDs that store the image data:

      $ ceph osd map pool-1-data rbd_header.<image_name>

      Example output:

      ... -> pg 14.1ed52b0 (14.10) -> up ([4,5,6], p4) acting ([4,5,6], p4)

   4. Verify that the OSDs belong to the expected failure domains:

      $ ceph osd tree | grep -E "osd.4|osd.5|osd.6"

      Example output:

      5    hdd  0.25000          osd.5       up   1.00000  1.00000
      6    hdd  0.25000          osd.6       up   1.00000  1.00000
      4    hdd  0.25000          osd.4       up   1.00000  1.00000