Deploying OpenShift Data Foundation in external mode

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.19.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. In the OpenShift Web Console, click Storage → Data Foundation → Storage Systems → Create StorageSystem.

2. In the Backing storage page, select the following options:

   1. Select Full deployment for the Deployment type option.
   2. Select Connect an external storage platform from the available options.
   3. Select Red Hat Ceph Storage for Storage platform.
   4. Click Next.

3. In the Connection details page, provide the necessary information:

   1. Click on the Download Script link to download the python script for extracting Ceph cluster details.

   2. For extracting the Red Hat Ceph Storage (RHCS) cluster details, contact the RHCS administrator to run the downloaded python script on a Red Hat Ceph Storage node with the admin key.

      1. Run the following command on the RHCS node to view the list of available arguments:

         # python3 ceph-external-cluster-details-exporter.py --help

         Copy to Clipboard Toggle word wrap
         Important

         Use python instead of python3 if the Red Hat Ceph Storage 4.x cluster is deployed on Red Hat Enterprise Linux 7.x (RHEL 7.x) cluster.

         You can also run the script from inside a MON container (containerized deployment) or from a MON node (RPM deployment).

         Note

         Use the yum install cephadm command and then the cephadm command to deploy your RHCS cluster using containers. You must pull the RHCS container images using the cephadm command, rather than using yum for installing the Ceph packages onto nodes. For more information, see RHCS product documentation.

      2. To retrieve the external cluster details from the RHCS cluster, choose one of the following two options, either a configuration file or command-line flags.

         1. Configuration file

            Use config-file flag. This stores the parameters used during deployment.

            In new deployments, you can save the parameters used during deployment in a configuration file. This file can then be used during upgrade to preserve the parameters as well as adding any additional parameters. Use config-file to set the path to the configuration file.

            Example configuration file saved in /config.ini:

            [Configurations]
            format = bash
            cephfs-filesystem-name = <filesystem-name>
            rbd-data-pool-name = <pool_name>
            ...

            Copy to Clipboard Toggle word wrap

            Set the path to the config.ini file using config-file:

            # python3 ceph-external-cluster-details-exporter.py --config-file /config.ini

            Copy to Clipboard Toggle word wrap

         2. Command-line flags

            Retrieve the external cluster details from the RHCS cluster, and pass the parameters for your deployment.

            # python3 ceph-external-cluster-details-exporter.py \
            --rbd-data-pool-name <rbd block pool name>  [optional arguments]

            Copy to Clipboard Toggle word wrap

            For example:

            # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

            Copy to Clipboard Toggle word wrap

            RBD parameters

            rbd-data-pool-name

            A mandatory parameter that is used for providing block storage in OpenShift Data Foundation.

            rados-namespace

            Divides an RBD data pool into separate logical namespaces, used for creating RBD PVC in a radosNamespace. Flags required with rados-namespace are restricted-auth-permission and k8s-cluster-name.

            rbd-metadata-ec-pool-name

            (Optional) The name of the erasure coded RBD metadata pool.

            RGW parameters

            rgw-endpoint

            (Optional) This parameter is required only if the object storage is to be provisioned through Ceph Rados Gateway for OpenShift Data Foundation. Provide the endpoint in the following format: <ip_address>:<port>

            Note

            A fully-qualified domain name (FQDN) is also supported in the format <FQDN>:<PORT>.

            rgw-pool-prefix

            (Optional) The prefix of the RGW pools. If not specified, the default prefix is default.

            rgw-tls-cert-path

            (Optional) The file path of the RADOS Gateway endpoint TLS certificate.

            • To provide the TLS certificate and RGW endpoint details to the helper script, ceph-external-cluster-details-exporter.py, run the following command:

              # python3 ceph-external-clustergw-endpoint r-details-exporter.py --rbd-data-pool-name <rbd block pool name> --rgw-endpoint <ip_address>:<port> --rgw-tls-cert-path <file path containing cert>

              Copy to Clipboard Toggle word wrap

              This creates a resource to create a Ceph Object Store CR such as Kubernetes secret containing the TLS certificate. All the intermediate certificates including private keys need to be stored in the certificate file.

            rgw-skip-tls

            (Optional) This parameter ignores the TLS certification validation when a self-signed certificate is provided (not recommended).

            Monitoring parameters

            monitoring-endpoint

            (Optional) This parameter accepts comma-separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.

            monitoring-endpoint-port

            (Optional) It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.

            Ceph parameters

            ceph-conf

            (Optional) The name of the Ceph configuration file.

            run-as-user

            (Optional) This parameter is used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as:

            • caps: [mgr] allow command config
            • caps: [mon] allow r, allow command quorum_status, allow command version
            • caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index

            CephFS parameters

            cephfs-metadata-pool-name

            (Optional) The name of the CephFS metadata pool.

            cephfs-data-pool-name

            (Optional) The name of the CephFS data pool.

            cephfs-filesystem-name

            (Optional) The name of the CephFS filesystem.

            Output parameters

            dry-run

            (Optional) This parameter helps to print the executed commands without running them.

            output

            (Optional) The file where the output is required to be stored.

            Multicluster parameters

            k8s-cluster-name

            (Optional) Kubernetes cluster name.

            cluster-name

            (Optional) The Ceph cluster name.

            restricted-auth-permission

            (Optional) This parameter restricts cephCSIKeyrings auth permissions to specific pools and clusters. Mandatory flags that need to be set with this are rbd-data-pool-name and cluster-name. You can also pass the cephfs-filesystem-name flag if there is CephFS user restriction so that permission is restricted to a particular CephFS filesystem.

            Note

            This parameter must be applied only for the new deployments. To restrict csi-users per pool and per cluster, you need to create new csi-users and new secrets for those csi-users.

            Example with restricted auth permission:

            # python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

            Copy to Clipboard Toggle word wrap

            Example of JSON output generated using the python script:

            [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

            Copy to Clipboard Toggle word wrap

      3. Save the JSON output to a file with .json extension

         Note

         For OpenShift Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) to be uploaded using the JSON file remain unchanged on the RHCS external cluster after the storage cluster creation.

      4. Run the command when there is a multi-tenant deployment in which the RHCS cluster is already connected to OpenShift Data Foundation deployment with a lower version.

         # python3 ceph-external-cluster-details-exporter.py --upgrade

         Copy to Clipboard Toggle word wrap

   3. Click Browse to select and upload the JSON file.

      The content of the JSON file is populated and displayed in the text box.

   4. Click Next

      The Next button is enabled only after you upload the .json file.

4. In the Review and create page, review if all the details are correct:

   • To modify any configuration settings, click Back to go back to the previous configuration page.
5. Click Create StorageSystem.

1. In the OpenShift Web Console, navigate to Storage → Data Foundation → Storage System → ocs-external-storagecluster.
2. Verify that Status of StorageCluster is Ready and has a green tick.
3. To verify that OpenShift Data Foundation, pods and StorageClass are successfully installed, see Verifying your external mode OpenShift Data Foundation installation for external Ceph storage system.

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.19.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. In the OpenShift Web Console, click Storage → Data Foundation → Storage Systems → Create StorageSystem.

2. In the Backing storage page, select the following options:

   1. Select Full deployment for the Deployment type option.
   2. Select Connect an external storage platform from the available options.
   3. Select IBM FlashSystem Storage from the Storage platform list.
   4. Click Next.

3. In the Create storage class page, provide the following information:

   1. Enter a name for the storage class.

      When creating block storage persistent volumes, select the storage class <storage_class_name> for best performance. The storage class allows direct I/O path to the FlashSystem.

   2. Enter the following details of IBM FlashSystem connection:

      • IP address
      • User name
      • Password
      • Pool name
   3. Select thick or thin for the Volume mode.
   4. Click Next.

1. In the Capacity and nodes page, provide the necessary details:

   1. Select a value for Requested capacity.

      The available options are 0.5 TiB, 2 TiB, and 4 TiB. The requested capacity is dynamically allocated on the infrastructure storage class.

   2. Select at least three nodes in three different zones.

      It is recommended to start with at least 14 CPUs and 34 GiB of RAM per node. If the nodes selected do not match the OpenShift Data Foundation cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster will be deployed. For minimum starting node requirements, see the Resource requirements section in the Planning guide.

   3. Click Next.

2. Optional: In the Security and network page, provide the necessary details:

   1. To enable encryption, select Enable data encryption for block and file storage.

      1. Choose any one or both Encryption level:

         • Cluster-wide encryption to encrypt the entire cluster (block and file).
         • StorageClass encryption to create encrypted persistent volume (block only) using encryption enabled storage class.

      2. Select the Connect to an external key management service checkbox. This is optional for cluster-wide encryption.

         1. Key Management Service Provider is set to Vault by default.
         2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>'), Port number, and Token.

      3. Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:

         1. Enter the Key Value secret path in the Backend Path that is dedicated and unique to OpenShift Data Foundation.
         2. Optional: Enter TLS Server Name and Vault Enterprise Namespace.
         3. Provide CA Certificate, Client Certificate, and Client Private Key by uploading the respective PEM encoded certificate file.
      4. Click Save.

      5. Select Default (SDN) if you are using a single network or Custom (Multus) if you are using multiple network interfaces.

         1. Select a Public Network Interface from the dropdown.
         2. Select a Cluster Network Interface from the dropdown. NOTE: If you are using only one additional network interface, select the single NetworkAttachementDefinition, that is, ocs-public-cluster for the Public Network Interface, and leave the Cluster Network Interface blank.
      6. Click Next.

   2. To enable in-transit encryption, select In-transit encryption.

      1. Select a Network.
      2. Click Next.

3. In the Review and create page, review if all the details are correct:

   • To modify any configuration settings, click Back to go back to the previous configuration page.
4. Click Create StorageSystem.

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d

   Copy to Clipboard Toggle word wrap

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

   Remove any config sections that reference the OpenShift Data Foundation storage classes as shown in the following example and save it.

   Expand

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . . Copy to Clipboard Toggle word wrap

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . . Copy to Clipboard Toggle word wrap

   Show more

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Data Foundation PVCs.

3. List the pods consuming the PVC.

   In this example, the alertmanagerMain and prometheusK8s pods that were consuming the PVCs are in the Terminating state. You can delete the PVCs once these pods are no longer using OpenShift Data Foundation PVC.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                                               READY   STATUS      RESTARTS AGE
   pod/alertmanager-main-0                            3/3     Terminating   0      10h
   pod/alertmanager-main-1                            3/3     Terminating   0      10h
   pod/alertmanager-main-2                            3/3     Terminating   0      10h
   pod/cluster-monitoring-operator-84cd9df668-zhjfn   1/1     Running       0      18h
   pod/grafana-5db6fd97f8-pmtbf                       2/2     Running       0      10h
   pod/kube-state-metrics-895899678-z2r9q             3/3     Running       0      10h
   pod/node-exporter-4njxv                            2/2     Running       0      18h
   pod/node-exporter-b8ckz                            2/2     Running       0      11h
   pod/node-exporter-c2vp5                            2/2     Running       0      18h
   pod/node-exporter-cq65n                            2/2     Running       0      18h
   pod/node-exporter-f5sm7                            2/2     Running       0      11h
   pod/node-exporter-f852c                            2/2     Running       0      18h
   pod/node-exporter-l9zn7                            2/2     Running       0      11h
   pod/node-exporter-ngbs8                            2/2     Running       0      18h
   pod/node-exporter-rv4v9                            2/2     Running       0      18h
   pod/openshift-state-metrics-77d5f699d8-69q5x       3/3     Running       0      10h
   pod/prometheus-adapter-765465b56-4tbxx             1/1     Running       0      10h
   pod/prometheus-adapter-765465b56-s2qg2             1/1     Running       0      10h
   pod/prometheus-k8s-0                               6/6     Terminating   1      9m47s
   pod/prometheus-k8s-1                               6/6     Terminating   1      9m47s
   pod/prometheus-operator-cbfd89f9-ldnwc             1/1     Running       0      43m
   pod/telemeter-client-7b5ddb4489-2xfpz              3/3     Running       0      10h
   
   NAME                                                      STATUS  VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-0   Bound    pvc-2eb79797-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-1   Bound    pvc-2ebeee54-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-2   Bound    pvc-2ec6a9cf-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-0        Bound    pvc-3162a80c-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-1        Bound    pvc-316e99e2-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h

   Copy to Clipboard Toggle word wrap

4. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   Copy to Clipboard Toggle word wrap
   Expand

   Before editing

   . . . storage: pvc: claim: registry-cephfs-rwx-pvc . . . Copy to Clipboard Toggle word wrap

   After editing

   . . . storage: emptyDir: {} . . . Copy to Clipboard Toggle word wrap

   Show more

   In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

   Copy to Clipboard Toggle word wrap

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   Copy to Clipboard Toggle word wrap

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete the PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

   Copy to Clipboard Toggle word wrap

   <pvc-name>

   Is the name of the PVC