Deploying OpenShift Data Foundation in external mode

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.9.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. Click Operators → Installed Operators to view all the installed operators.

   Ensure that the Project selected is openshift-storage.

2. Click OpenShift Data Foundation and then click Create StorageSystem.

3. In the Backing storage page, select the following options:

   1. Select Connect an external storage platform from the available options.
   2. Select Red Hat Ceph Storage for Storage platform.
   3. Click Next.

4. In the Connection details page, provide the necessary information:

   1. Click on the Download Script link to download the python script for extracting Ceph cluster details.

   2. For extracting the Red Hat Ceph Storage (RHCS) cluster details, contact the RHCS administrator to run the downloaded python script on a Red Hat Ceph Storage node with admin key.

      1. Run the following command on the RHCS node to view the list of available arguments:

         # python3 ceph-external-cluster-details-exporter.py --help

         Important

         Use python instead of python3 if the Red Hat Ceph Storage 4.x cluster is deployed on Red Hat Enterprise Linux 7.x (RHEL 7.x) cluster.

         Note

         You can also run the script from inside a MON container (containerized deployment) or from a MON node (rpm deployment).

      2. To retrieve the external cluster details from the RHCS cluster, run the following command:

         # python3 ceph-external-cluster-details-exporter.py \
         --rbd-data-pool-name <rbd block pool name>  [optional arguments]

         For example:

         # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

         In this example,

         --rbd-data-pool-name

         A mandatory parameter that is used for providing block storage in OpenShift Data Foundation.

         --rgw-endpoint

         An optional parameter, which needs to provided only if the object storage is to be provisioned through Ceph Rados Gateway for OpenShift Data Foundation. Provide the endpoint in the following format: <ip_address>:<port>

         --monitoring-endpoint

         Is optional. It accepts comma separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.

         --monitoring-endpoint-port

         This is optional. It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.

         --run-as-user

         This is an optional parameter used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as:

         • caps: [mgr] allow command config
         • caps: [mon] allow r, allow command quorum_status, allow command version

         • caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index

           Example of JSON output generated using the python script:

           [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

      3. Save the JSON output to a file with .json extension

         Note

         For OpenShift Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) to be uploaded using the JSON file remains unchanged on the RHCS external cluster after the storage cluster creation.

   3. Click Browse to select and upload the JSON file.

      The content of the JSON file is populated and displayed in the text box.

   4. Click Next

      The Next button is enabled only after you upload the .json file.

5. In the Review and create page, review if all the details are correct:

   • To modify any configuration settings, click Back to go back to the previous configuration page.
6. Click Create StorageSystem.

1. In the OpenShift Web Console, navigate to Installed Operators → OpenShift Data Foundation → Storage System → ocs-external-storagecluster-storagesystem → Resources.
2. Verify that Status of StorageCluster is Ready and has a green tick.
3. To verify that OpenShift Data Foundation, pods and StorageClass are successfully installed, see Verifying your external mode OpenShift Data Foundation installation for external Ceph storage system.

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.9.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Ensure that the Enable option is selected for the Console plugin.
   6. Click Install.

Procedure

1. In the OpenShift Web Console, click Operators → Installed Operators to view all the installed operators.

   Ensure that the Project selected is openshift-storage.

2. Click OpenShift Data Foundation and then click Create StorageSystem.

3. In the Backing storage page, select the following options:

   1. Select Connect an external storage platform from the available options.
   2. Select IBM FlashSystem Storage from the Storage platform list.
   3. Click Next.

4. In the Create storage class page, provide the following information:

   1. Enter a name for the storage class.

      When creating block storage persistent volumes, select the storage class <storage_class_name> for best performance. The storage class allows direct I/O path to the FlashSystem.

   2. Enter the following details of IBM FlashSystem connection:

      • IP address
      • User name
      • Password
      • Pool name
   3. Select thick or thin for the Volume mode.
   4. Click Next.

5. In the Capacity and nodes page, provide the necessary details:

   1. Select a value for Requested capacity.

      The available options are 0.5 TiB, 2 TiB, and 4 TiB. The requested capacity is dynamically allocated on the infrastructure storage class.

   2. Select at least three nodes in three different zones.

      It is recommended to start with at least 14 CPUs and 34 GiB of RAM per node. If the nodes selected do not match the OpenShift Data Foundation cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster will be deployed. For minimum starting node requirements, see the Resource requirements section in the Planning guide.

   3. Click Next.

6. Optional: In the Security and network page, provide the necessary details:

   1. To enable encryption, select Enable data encryption for block and file storage.

   2. Choose any one or both Encryption level:

      • Cluster-wide encryption to encrypt the entire cluster (block and file).
      • StorageClass encryption to create encrypted persistent volume (block only) using encryption enabled storage class.

   3. Select the Connect to an external key management service checkbox. This is optional for cluster-wide encryption.

      1. Key Management Service Provider is set to Vault by default.
      2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>'), Port number, and Token.

   4. Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:

      1. Enter the Key Value secret path in the Backend Path that is dedicated and unique to OpenShift Data Foundation.
      2. Optional: Enter TLS Server Name and Vault Enterprise Namespace.
      3. Provide CA Certificate, Client Certificate, and Client Private Key by uploading the respective PEM encoded certificate file.
   5. Click Save.

   6. Select Default (SDN) if you are using a single network or Custom (Multus) if you are using multiple network interfaces.

      1. Select a Public Network Interface from the dropdown.
      2. Select a Cluster Network Interface from the dropdown. NOTE: If you are using only one additional network interface, select the single NetworkAttachementDefinition, that is, ocs-public-cluster for the Public Network Interface, and leave the Cluster Network Interface blank.
   7. Click Next.

7. In the Review and create page, review if all the details are correct:

   • To modify any configuration settings, click Back to go back to the previous configuration page.
8. Click Create StorageSystem.

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Remove any config sections that reference the OpenShift Data Foundation storage classes as shown in the following example and save it.

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . .

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . .

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Data Foundation PVCs.

3. List the pods consuming the PVC.

   In this example, the alertmanagerMain and prometheusK8s pods that were consuming the PVCs are in the Terminating state. You can delete the PVCs once these pods are no longer using OpenShift Data Foundation PVC.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                                               READY   STATUS      RESTARTS AGE
   pod/alertmanager-main-0                            3/3     Terminating   0      10h
   pod/alertmanager-main-1                            3/3     Terminating   0      10h
   pod/alertmanager-main-2                            3/3     Terminating   0      10h
   pod/cluster-monitoring-operator-84cd9df668-zhjfn   1/1     Running       0      18h
   pod/grafana-5db6fd97f8-pmtbf                       2/2     Running       0      10h
   pod/kube-state-metrics-895899678-z2r9q             3/3     Running       0      10h
   pod/node-exporter-4njxv                            2/2     Running       0      18h
   pod/node-exporter-b8ckz                            2/2     Running       0      11h
   pod/node-exporter-c2vp5                            2/2     Running       0      18h
   pod/node-exporter-cq65n                            2/2     Running       0      18h
   pod/node-exporter-f5sm7                            2/2     Running       0      11h
   pod/node-exporter-f852c                            2/2     Running       0      18h
   pod/node-exporter-l9zn7                            2/2     Running       0      11h
   pod/node-exporter-ngbs8                            2/2     Running       0      18h
   pod/node-exporter-rv4v9                            2/2     Running       0      18h
   pod/openshift-state-metrics-77d5f699d8-69q5x       3/3     Running       0      10h
   pod/prometheus-adapter-765465b56-4tbxx             1/1     Running       0      10h
   pod/prometheus-adapter-765465b56-s2qg2             1/1     Running       0      10h
   pod/prometheus-k8s-0                               6/6     Terminating   1      9m47s
   pod/prometheus-k8s-1                               6/6     Terminating   1      9m47s
   pod/prometheus-operator-cbfd89f9-ldnwc             1/1     Running       0      43m
   pod/telemeter-client-7b5ddb4489-2xfpz              3/3     Running       0      10h
   
   NAME                                                      STATUS  VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-0   Bound    pvc-2eb79797-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-1   Bound    pvc-2ebeee54-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-2   Bound    pvc-2ec6a9cf-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-0        Bound    pvc-3162a80c-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-1        Bound    pvc-316e99e2-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h

4. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   Before editing

   . . . storage: pvc: claim: registry-cephfs-rwx-pvc . . .

   After editing

   . . . storage: emptyDir: {} . . .

   In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete the PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

   <pvc-name>

   Is the name of the PVC