Deploying OpenShift Data Foundation in external mode

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.18.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. Click Operators → Installed Operators to view all the installed operators.

   Ensure that the Project selected is openshift-storage.

2. Click OpenShift Data Foundation and then click Create StorageSystem.

3. In the Backing storage page, select the following options:

   1. Select Full deployment for the Deployment type option.
   2. Select Connect an external storage platform from the available options.
   3. Select Red Hat Ceph Storage for Storage platform.
   4. Click Next.

4. In the Connection details page, provide the necessary information:

   1. Click on the Download Script link to download the python script for extracting Ceph cluster details.

   2. For extracting the Red Hat Ceph Storage (RHCS) cluster details, contact the RHCS administrator to run the downloaded python script on a Red Hat Ceph Storage node with the admin key.

      1. Run the following command on the RHCS node to view the list of available arguments:

         # python3 ceph-external-cluster-details-exporter.py --help

         Important

         Use python instead of python3 if the Red Hat Ceph Storage 4.x cluster is deployed on Red Hat Enterprise Linux 7.x (RHEL 7.x) cluster.

         You can also run the script from inside a MON container (containerized deployment) or from a MON node (RPM deployment).

         Note

         Use the yum install cephadm command and then the cephadm command to deploy your RHCS cluster using containers. You must pull the RHCS container images using the cephadm command, rather than using yum for installing the Ceph packages onto nodes. For more information, see RHCS product documentation.

      2. To retrieve the external cluster details from the RHCS cluster, choose one of the following two options, either a configuration file or command-line flags.

         1. Configuration file

            Use config-file flag. This stores the parameters used during deployment.

            In new deployments, you can save the parameters used during deployment in a configuration file. This file can then be used during upgrade to preserve the parameters as well as adding any additional parameters. Use config-file to set the path to the configuration file.

            Example configuration file saved in /config.ini:

            [Configurations]
            format = bash
            cephfs-filesystem-name = <filesystem-name>
            rbd-data-pool-name = <pool_name>
            ...

            Set the path to the config.ini file using config-file:

            # python3 ceph-external-cluster-details-exporter.py --config-file /config.ini

         2. Command-line flags

            Retrieve the external cluster details from the RHCS cluster, and pass the parameters for your deployment.

            # python3 ceph-external-cluster-details-exporter.py \
            --rbd-data-pool-name <rbd block pool name>  [optional arguments]

            For example:

            # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

            RBD parameters

            rbd-data-pool-name

            A mandatory parameter that is used for providing block storage in OpenShift Data Foundation.

            rados-namespace

            Divides an RBD data pool into separate logical namespaces, used for creating RBD PVC in a radosNamespace. Flags required with rados-namespace are restricted-auth-permission and k8s-cluster-name.

            rbd-metadata-ec-pool-name

            (Optional) The name of the erasure coded RBD metadata pool.

            RGW parameters

            rgw-endpoint

            (Optional) This parameter is required only if the object storage is to be provisioned through Ceph Rados Gateway for OpenShift Data Foundation. Provide the endpoint in the following format: <ip_address>:<port>

            Note

            A fully-qualified domain name (FQDN) is also supported in the format <FQDN>:<PORT>.

            rgw-pool-prefix

            (Optional) The prefix of the RGW pools. If not specified, the default prefix is default.

            rgw-tls-cert-path

            (Optional) The file path of the RADOS Gateway endpoint TLS certificate.

            • To provide the TLS certificate and RGW endpoint details to the helper script, ceph-external-cluster-details-exporter.py, run the following command:

              # python3 ceph-external-clustergw-endpoint r-details-exporter.py --rbd-data-pool-name <rbd block pool name> --rgw-endpoint <ip_address>:<port> --rgw-tls-cert-path <file path containing cert>

              This creates a resource to create a Ceph Object Store CR such as Kubernetes secret containing the TLS certificate. All the intermediate certificates including private keys need to be stored in the certificate file.

            rgw-skip-tls

            (Optional) This parameter ignores the TLS certification validation when a self-signed certificate is provided (not recommended).

            Monitoring parameters

            monitoring-endpoint

            (Optional) This parameter accepts comma-separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.

            monitoring-endpoint-port

            (Optional) It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.

            Ceph parameters

            ceph-conf

            (Optional) The name of the Ceph configuration file.

            run-as-user

            (Optional) This parameter is used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as:

            • caps: [mgr] allow command config
            • caps: [mon] allow r, allow command quorum_status, allow command version
            • caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index

            CephFS parameters

            cephfs-metadata-pool-name

            (Optional) The name of the CephFS metadata pool.

            cephfs-data-pool-name

            (Optional) The name of the CephFS data pool.

            cephfs-filesystem-name

            (Optional) The name of the CephFS filesystem.

            Output parameters

            dry-run

            (Optional) This parameter helps to print the executed commands without running them.

            output

            (Optional) The file where the output is required to be stored.

            Multicluster parameters

            k8s-cluster-name

            (Optional) Kubernetes cluster name.

            cluster-name

            (Optional) The Ceph cluster name.

            restricted-auth-permission

            (Optional) This parameter restricts cephCSIKeyrings auth permissions to specific pools and clusters. Mandatory flags that need to be set with this are rbd-data-pool-name and cluster-name. You can also pass the cephfs-filesystem-name flag if there is CephFS user restriction so that permission is restricted to a particular CephFS filesystem.

            Note

            This parameter must be applied only for the new deployments. To restrict csi-users per pool and per cluster, you need to create new csi-users and new secrets for those csi-users.

            Example with restricted auth permission:

            # python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

            Example of JSON output generated using the python script:

            [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

      3. Save the JSON output to a file with .json extension

         Note

         For OpenShift Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) to be uploaded using the JSON file remain unchanged on the RHCS external cluster after the storage cluster creation.

      4. Run the command when there is a multi-tenant deployment in which the RHCS cluster is already connected to OpenShift Data Foundation deployment with a lower version.

         # python3 ceph-external-cluster-details-exporter.py --upgrade

   3. Click Browse to select and upload the JSON file.

      The content of the JSON file is populated and displayed in the text box.

   4. Click Next

      The Next button is enabled only after you upload the .json file.

5. In the Review and create page, review if all the details are correct:

   • To modify any configuration settings, click Back to go back to the previous configuration page.
6. Click Create StorageSystem.

1. In the OpenShift Web Console, navigate to Installed Operators → OpenShift Data Foundation → Storage System → ocs-external-storagecluster-storagesystem → Resources.
2. Verify that Status of StorageCluster is Ready and has a green tick.
3. To verify that OpenShift Data Foundation, pods and StorageClass are successfully installed, see Verifying your external mode OpenShift Data Foundation installation for external Ceph storage system.