Deploying OpenShift Container Storage in external mode

1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

2. Select openshift-storage from the Project drop down list.

   For more information on the expected number of pods for each component and how it varies depending on the number of nodes, see Table 5.1, “Pods corresponding to OpenShift Container Storage components”

3. Verify that the following pods are in running state:

   Table 5.1. Pods corresponding to OpenShift Container Storage components

   Component	Corresponding pods
   OpenShift Container Storage Operator	ocs-operator-* (1 pod on any worker node)
   Rook-ceph Operator	rook-ceph-operator-* (1 pod on any worker node)
   Multicloud Object Gateway	noobaa-operator-* (1 pod on any worker node) noobaa-core-* (1 pod on any worker node) nooba-db-* (1 pod on any worker node) noobaa-endpoint-* (1 pod on any worker node)
   CSI	cephfs csi-cephfsplugin-* (1 pod on each worker node) csi-cephfsplugin-provisioner-* (2 pods distributed across worker nodes) rbd csi-rbdplugin-* (1 pod on each worker node) csi-rbdplugin-provisioner-* (2 pods distributed across worker nodes)

1. Query for PVCs and OBCs that use the OpenShift Container Storage based storage class provisioners.

   For example :

   $ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-external-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{" Labels: "}{@.metadata.labels}{"\n"}{end}' --all-namespaces|awk '! ( /Namespace: openshift-storage/ && /app:noobaa/ )' | grep -v noobaa-default-backing-store-noobaa-pvc

   Note

   If the external RedHat Ceph Storage cluster is not configured for CephFS, you can ignore the following query command for ocs-external-storagecluster-cephfs.

   $ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-external-storagecluster-cephfs")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces

   Note

   If the external RedHat Ceph Storage cluster is not configured for Object Storage, you can ignore the following query command for ocs-external-storagecluster-ceph-rgw.

   $ oc get obc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-external-storagecluster-ceph-rgw")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces

   $ oc get obc -o=jsonpath='{range .items[?(@.spec.storageClassName=="openshift-storage.noobaa.io")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces

2. Follow these instructions to ensure the PVCs and OBCs listed in the previous step are deleted.

   If you have created PVCs as a part of configuring the monitoring stack, cluster logging operator, or image registry, then you must perform the clean up steps provided in the following sections as required:

   • Section 6.2, “Removing monitoring stack from OpenShift Container Storage”
   • Section 6.3, “Removing OpenShift Container Platform registry from OpenShift Container Storage”

   • Section 6.4, “Removing the cluster logging operator from OpenShift Container Storage”

     For each of the remaining PVCs or OBCs, follow the steps mentioned below:

     1. Determine the pod that is consuming the PVC or OBC.

     2. Identify the controlling API object such as a Deployment, StatefulSet, DaemonSet , Job, or a custom controller.

        Each API object has a metadata field known as OwnerReference. This is a list of associated objects. The OwnerReference with the controller field set to true will point to controlling objects such as ReplicaSet, StatefulSet,DaemonSet and so on.

     3. Ensure that the API object is not consuming PVC or OBC provided by OpenShift Container Storage. Either the object should be deleted or the storage should be replaced. Ask the owner of the project to make sure that it is safe to delete or modify the object.

        Note

        You can ignore the noobaa pods.

     4. Delete the OBCs.

        $ oc delete obc <obc name> -n <project name>

     5. Delete any custom Bucket Class you have created.

        $ oc get bucketclass -A | grep -v noobaa-default-bucket-class

        $ oc delete bucketclass <bucketclass name> -n <project-name>

     6. If you have created any custom Multi Cloud Gateway backingstores, delete each of them.

        1. List and note the backingstores.

           for bs in $(oc get backingstore -o name -n openshift-storage | grep -v noobaa-default-backing-store); do echo "Found backingstore $bs"; echo "Its has the following pods running :"; echo "$(oc get pods -o name -n openshift-storage | grep $(echo ${bs} | cut -f2 -d/))"; done

        2. Delete each of the backingstores listed above and confirm that the corresponding pods and PVCs are deleted.

           for bs in $(oc get backingstore -o name -n openshift-storage | grep -v noobaa-default-backing-store); do echo "Deleting Backingstore $bs"; oc delete -n openshift-storage $bs; done

        3. If any of the backingstores listed above were based on the pv-pool, ensure that the corresponding pod and PVC are also deleted.

           $ oc get pods  -n openshift-storage | grep noobaa-pod | grep -v noobaa-default-backing-store-noobaa-pod

           $ oc get pvc -n openshift-storage --no-headers | grep -v noobaa-db | grep -v noobaa-default-backing-store-noobaa-pvc

     7. Delete the remaining PVCs listed in Step 1.

        $ oc delete pvc <pvc name> -n <project-name>

3. Delete the StorageCluster object and wait for the removal of the associated resources.

   $ oc delete -n openshift-storage storagecluster --all --wait=true

4. Delete the namespace and wait till the deletion is complete. You will need to switch to another project if openshift-storage is the active project.

   1. Switch to another namespace if openshift-storage is the active namespace.

      For example :

      $ oc project default

   2. Delete the openshift-storage namespace.

      $ oc delete project openshift-storage --wait=true --timeout=5m

   3. Wait for approximately five minutes and confirm if the project is deleted successfully.

      $ oc get project  openshift-storage

      Output:

      Error from server (NotFound): namespaces "openshift-storage" not found

      Note

      While uninstalling OpenShift Container Storage, if namespace is not deleted completely and remains in Terminating state, perform the steps in the article Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.

5. Delete the openshift-storage.noobaa.io storage class.

   $ oc delete storageclass  openshift-storage.noobaa.io --wait=true --timeout=5m

6. Confirm all PVs are deleted. If there is any PV left in the Released state, delete it.

   # oc get pv|egrep 'ocs-external-storagecluster-ceph-rbd|ocs-external-storagecluster-cephfs'

   # oc  delete pv <pv name>

7. Remove CustomResourceDefinitions.

   $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io  storageclusterinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io --wait=true --timeout=5m

8. To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,

   1. Click Home → Overview to access the dashboard.
   2. Verify that the Persistent Storage and Object Service tabs no longer appear next to the Cluster tab.

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   8d

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Remove any config sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-external-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . .

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . .

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Container Storage PVCs.

4. List the pods consuming the PVC.

   In this example, the alertmanagerMain and prometheusK8s pods that were consuming the PVCs are in the Terminating state. You can delete the PVCs once these pods are no longer using OpenShift Container Storage PVC.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                                               READY   STATUS      RESTARTS AGE
   pod/alertmanager-main-0                            3/3     Terminating   0      10h
   pod/alertmanager-main-1                            3/3     Terminating   0      10h
   pod/alertmanager-main-2                            3/3     Terminating   0      10h
   pod/cluster-monitoring-operator-84cd9df668-zhjfn   1/1     Running       0      18h
   pod/grafana-5db6fd97f8-pmtbf                       2/2     Running       0      10h
   pod/kube-state-metrics-895899678-z2r9q             3/3     Running       0      10h
   pod/node-exporter-4njxv                            2/2     Running       0      18h
   pod/node-exporter-b8ckz                            2/2     Running       0      11h
   pod/node-exporter-c2vp5                            2/2     Running       0      18h
   pod/node-exporter-cq65n                            2/2     Running       0      18h
   pod/node-exporter-f5sm7                            2/2     Running       0      11h
   pod/node-exporter-f852c                            2/2     Running       0      18h
   pod/node-exporter-l9zn7                            2/2     Running       0      11h
   pod/node-exporter-ngbs8                            2/2     Running       0      18h
   pod/node-exporter-rv4v9                            2/2     Running       0      18h
   pod/openshift-state-metrics-77d5f699d8-69q5x       3/3     Running       0      10h
   pod/prometheus-adapter-765465b56-4tbxx             1/1     Running       0      10h
   pod/prometheus-adapter-765465b56-s2qg2             1/1     Running       0      10h
   pod/prometheus-k8s-0                               6/6     Terminating   1      9m47s
   pod/prometheus-k8s-1                               6/6     Terminating   1      9m47s
   pod/prometheus-operator-cbfd89f9-ldnwc             1/1     Running       0      43m
   pod/telemeter-client-7b5ddb4489-2xfpz              3/3     Running       0      10h
   
   NAME                                                      STATUS  VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-0   Bound    pvc-2eb79797-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-1   Bound    pvc-2ebeee54-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-2   Bound    pvc-2ec6a9cf-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-0        Bound    pvc-3162a80c-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-1        Bound    pvc-316e99e2-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-external-storagecluster-ceph-rbd   19h

5. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   Before editing

   . . . storage: pvc: claim: registry-cephfs-rwx-pvc . . .

   After editing

   . . . storage: emptyDir: {} . . .

   In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m