Deploying OpenShift Container Storage

Procedure

1. Click Operators → OperatorHub in the left pane of the OpenShift Web Console.

2. Click on OpenShift Container Storage.

   You can use the Filter by keyword text box or the filter list to search for OpenShift Container Storage from the list of operators.

3. On the OpenShift Container Storage operator page, click Install.

4. On the Install Operator page, ensure the following options are selected:

   1. Update Channel as stable-4.4
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace PR openshift-storage. If Namespace openshift-storage does not exist, it will be created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual. Approval Strategy is set to Automatic by default.

      • Approval Strategy as Automatic.

        Note

        When you select the Approval Strategy as Automatic, approval is not required either during fresh installation or when updating to the latest version of OpenShift Container Storage.

        1. Click Install
        2. Wait for the install to initiate. This may take up to 20 minutes.
        3. Click Operators → Installed Operators
        4. Ensure the Project is openshift-storage. By default, the Project is openshift-storage.
        5. Wait for the Status of OpenShift Container Storage to change to Succeeded.

      • Approval Strategy as Manual.

        Note

        When you select the Approval Strategy as Manual, approval is required during fresh installation or when updating to the latest version of OpenShift Container Storage.

        1. Click Install
        2. On the Installed Operators page, click ocs-operator.
        3. On the Subscription Details page, click the Install Plan link.
        4. On the InstallPlan Details page, click Preview Install Plan
        5. Review the install plan and click Approve.
        6. Wait for the Status of the Components to change from Unknown to either Created or Present.
        7. Click Operators → Installed Operators
        8. Ensure the Project is openshift-storage. By default, the Project is openshift-storage.
        9. Wait for the Status of OpenShift Container Storage to change to Succeeded.

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. On the Installed Operator page, select openshift-storage from the Project drop down list to switch to the openshift-storage project.

3. Click OpenShift Container Storage operator.

   OpenShift Container Storage operator creates a OCSInitialization resource automatically.

4. On the OpenShift Container Storage operator page, scroll right and click the Storage Cluster tab.

   Figure 1.1. OpenShift Container Storage Operator page

   

5. On the OCS Cluster Services page, click Create OCS Cluster Service.

   Figure 1.2. Create New OCS Service page

   

6. On the Create New OCS Service page, perform the following:

   1. Select at least three worker nodes from the available list of nodes for the use of OpenShift Container Storage service. Ensure that the nodes are in different Location.

   2. Storage Class is set by default depending on the platform.

      gp2 is the default storage class for AWS and thin is the default storage class for VMware.

   3. Select OCS Service Capacity from the drop down list.

      Note

      Once you select the initial storage capacity here, you can add more capacity only in this increment.

7. Click Create.

   The Create button is enabled only after you select three nodes. A new storage cluster of three volumes will be created with one volume per worker node. The default configuration uses a replication factor of 3.

1. Log in to the Red Hat Enterprise Linux based node and open a terminal.

2. Verify that the node has access to the rhel-7-server-extras-rpms repository.

   # subscription-manager repos --list-enabled | grep rhel-7-server

   If you do not see both rhel-7-server-rpms and rhel-7-server-extras-rpms in the output, or if there is no output, run the following commands to enable each repository.

   # subscription-manager repos --enable=rhel-7-server-rpms
   # subscription-manager repos --enable=rhel-7-server-extras-rpms

3. Install the required packages.

   # yum install -y policycoreutils container-selinux

4. Persistently enable container use of the Ceph file system in SELinux.

   # setsebool -P container_use_cephfs on

5. Verify that containers can now access OpenShift Container Storage hosted on this node.

Procedure

1. Click Operators → OperatorHub in the left pane of the OpenShift Web Console.

   Figure 2.1. List of operators in the Operator Hub

   
2. Search for OpenShift Container Storage from the list of operators and click on it.
3. On the OpenShift Container Storage operator page, click Install.

4. On the Create Operator Subscription page:

   Figure 2.2. Create Operator Subscription page

   

   1. Select A specific namespace on the cluster for the Installation Mode option.

      • Select openshift-storage namespace from the drop down menu.
   2. Select stable-4.4 as the update channel.

   3. Select an Approval Strategy:

      • Automatic specifies that you want OpenShift Container Platform to upgrade OpenShift Container Storage automatically.
      • Manual specifies that you want to have control to upgrade OpenShift Container Storage manually.

5. Click Subscribe.

   Figure 2.3. Installed operators

   

   The Installed Operators page is displayed with the status of the operator.

Procedure

1. Click Operators → OperatorHub in the left pane of the OpenShift Web Console.
2. Search for Local Storage Operator from the list of operators and click on it.

3. Click Install.

   Figure 2.4. Create Operator Subscription page

   

4. Select A specific namespace on the cluster for the Installation Mode option.

   • Select local-storage namespace from the drop down menu.
5. Select a desired value for the Update Channel option.
6. Select the desired Approval Strategy.
7. Click Subscribe.
8. Verify that the Local Storage Operator show the Status as Succeeded.

Procedure

1. List and verify the name of the worker nodes with the OpenShift Container Storage label.

   $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage=

   Example output:

   NAME                                        STATUS   ROLES    AGE     VERSION
   ip-10-0-135-71.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2
   ip-10-0-145-125.us-east-2.compute.internal   Ready    worker   6h45m   v1.16.2
   ip-10-0-160-91.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2

2. Log in to each worker node that is used for OpenShift Container Storage resources and find the unique by-id device name for each available raw block device.

   $ oc debug node/<Nodename>

   Example output:

   $ oc debug node/ip-10-0-135-71.us-east-2.compute.internal
   Starting pod/ip-10-0-135-71us-east-2computeinternal-debug ...
   To use host binaries, run `chroot /host`
   Pod IP: 10.0.135.71
   If you don't see a command prompt, try pressing enter.
   sh-4.2# chroot /host
   sh-4.4# lsblk
   NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   nvme1n1                      259:0    0   2.3T  0 disk
   nvme2n1                      259:1    0   2.3T  0 disk
   nvme0n1                      259:2    0   120G  0 disk
   |-nvme0n1p1                  259:3    0   384M  0 part /boot
   |-nvme0n1p2                  259:4    0   127M  0 part /boot/efi
   |-nvme0n1p3                  259:5    0     1M  0 part
   `-nvme0n1p4                  259:6    0 119.5G  0 part
     `-coreos-luks-root-nocrypt 253:0    0 119.5G  0 dm   /sysroot

   In this example, the local devices available are nvme1n1 and nvme2n1.

3. Find the unique by-id device name depending on the hardware serial number for each device.

   sh-4.4#  ls -l /dev/disk/by-id/ | grep Storage
   lrwxrwxrwx. 1 root root 13 Jun 26 07:29 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1924C57D4F1FC5236 -> ../../nvme2n1
   lrwxrwxrwx. 1 root root 13 Jun 26 07:29 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS22ABDB45A3BC9028F -> ../../nvme1n1

   In this example, the local devices available are nvme1n1 and nvme2n1 with a size of 2.3 TiB

   For each worker node that has the OpenShift Container Storage label (a minimum of three), you need to find the unique by-id device names. In this example, the by-id device names are:

   • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1924C57D4F1FC5236
   • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS22ABDB45A3BC9028F

1. Creating PVs by using the LocalVolume CR
2. Creating a new StorageClass

Procedure

1. Create local persistent volumes (PVs) on the storage nodes using LocalVolume custom resource (CR).

   Example of LocalVolume CR local-storage-block.yaml using OpenShift Storage Container label as node selector and by-id device identifier:

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     tolerations:
     - key: "node.ocs.openshift.io/storage"
       value: "true"
       effect: NoSchedule
     nodeSelector:
       nodeSelectorTerms:
         - matchExpressions:
             - key: cluster.ocs.openshift.io/openshift-storage
               operator: In
               values:
                 - ''
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7   # <-- modify this line
           - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8   # <-- modify this line

   Each Amazon EC2 i3en.2xlarge instance has two disks and this example uses both the disks.

2. Create the LocalVolume CR.

   $ oc create -f local-storage-block.yaml

   Example output:

   localvolume.local.storage.openshift.io/local-block created

3. Check if the pods are created.

   $ oc -n local-storage get pods

   Example output:

   NAME                                      READY   STATUS    RESTARTS   AGE
   local-block-local-diskmaker-59rmn         1/1     Running   0          15m
   local-block-local-diskmaker-6n7ct         1/1     Running   0          15m
   local-block-local-diskmaker-jwtsn         1/1     Running   0          15m
   local-block-local-provisioner-6ssxc       1/1     Running   0          15m
   local-block-local-provisioner-swwvx       1/1     Running   0          15m
   local-block-local-provisioner-zmv5j       1/1     Running   0          15m
   local-storage-operator-7848bbd595-686dg   1/1     Running   0          15m

4. Check if the PVs are created.

   You must see a new PV for each of the local storage devices on the three worker nodes. Refer the example in the Finding available storage devices section that shows two available storage devices per worker node with a size 2328Gi for each node.

   $ oc get pv

   Example output:

   NAME               CAPACITY ACCESS MODES  RECLAIM POLICY STATUS       CLAIM     STORAGECLASS  REASON   AGE
   local-pv-1a46bc79  2328Gi   RWO           Delete         Available              localblock             14m
   local-pv-429d90ee  2328Gi   RWO           Delete         Available              localblock             14m
   local-pv-4d0a62e3  2328Gi   RWO           Delete         Available              localblock             14m
   local-pv-55c05d76  2328Gi   RWO           Delete         Available              localblock             14m
   local-pv-5c7b0990  2328Gi   RWO           Delete         Available              localblock             14m
   local-pv-a6b283b   2328Gi   RWO           Delete         Available              localblock             14m

5. Check if a new StorageClass is created due to the creation of LocalVolume CR. This StorageClass is used while creating StorageCluster to create PVCs.

   $ oc get sc | grep localblock

   Example output:

   NAME            PROVISIONER                    RECLAIMPOLICY
   VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION     AGE
   localblock      kubernetes.io/no-provisioner   Delete
   WaitForFirstConsumer  false                3h47m

6. Create StorageCluster CR that uses the localblock StorageClass and the PVs that are created.

   Example of StorageCluster CR ocs-cluster-service.yaml using monDataDirHostPath and localblock StorageClass.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     resources:
       mds:
         limits:
           cpu: 3
           memory: 8Gi
         requests:
           cpu: 1
           memory: 8Gi
       noobaa-core:
         limits:
           cpu: 2
           memory: 8Gi
         requests:
           cpu: 1
           memory: 8Gi
       noobaa-db:
         limits:
           cpu: 2
           memory: 8Gi
         requests:
           cpu: 1
           memory: 8Gi
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
       - count: 2
         dataPVCTemplate:
           spec:
             accessModes:
               - ReadWriteOnce
             resources:
               requests:
                 storage: 2328Gi
             storageClassName: localblock
             volumeMode: Block
         name: ocs-deviceset
         placement: {}
         portable: false
         replica: 3
         resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

7. Create StorageCluster CR.

   $ oc create -f ocs-cluster-service.yaml

   Example output

   storagecluster.ocs.openshift.io/ocs-cluster-service created

Procedure

1. Create the LocalVolume CR for block PVs.

   Example of LocalVolume CR local-storage-block.yaml using OpenShift Container Storage label as node selector:

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     nodeSelector:
       nodeSelectorTerms:
       - matchExpressions:
           - key: cluster.ocs.openshift.io/openshift-storage
             operator: In
             values:
             - ""
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/scsi-36000c2991c27c2e5ba7c47d1e4352de2   # <-- modify this line
           - /dev/disk/by-id/scsi-36000c29682ca9e347926406711f3dc4e   # <-- modify this line
           - /dev/disk/by-id/scsi-36000c296aaf03a9b1e4b01d086bc6348   # <-- modify this line

2. Create LocalVolume CR for block PVs.

   $ oc create -f local-storage-block.yaml

   Example output:

   localvolume.local.storage.openshift.io/local-block created

3. Check if the pods are created.

   $ oc -n local-storage get pods

   Example output:

   NAME                                      READY   STATUS    RESTARTS   AGE
   local-block-local-diskmaker-5brzv         1/1     Running   0          31s
   local-block-local-diskmaker-8sxcs         1/1     Running   0          31s
   local-block-local-diskmaker-s7s9p         1/1     Running   0          31s
   local-block-local-provisioner-9cbw8       1/1     Running   0          31s
   local-block-local-provisioner-cpddv       1/1     Running   0          31s
   local-block-local-provisioner-f6h7h       1/1     Running   0          31s
   local-storage-operator-75b9776b75-vwdzh   1/1     Running   0          2m47s

4. Check the new localblock StorageClass.

   $ oc get sc | grep localblock

   Example output:

   NAME            PROVISIONER                    RECLAIMPOLICY
   VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION     AGE
   localblock      kubernetes.io/no-provisioner   Delete
   WaitForFirstConsumer  false                3h47m

5. Check the PVs that are created with the Available status.

   $ oc get pv

   Example output:

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-150fdc87   2TiB      RWO            Delete           Available           localblock              2m11s
   local-pv-183bfc0a   2TiB      RWO            Delete           Available           localblock              2m11s
   local-pv-b2f5cb25   2TiB      RWO            Delete           Available           localblock              2m21s

   In this example, three PVs are used for OSD storage.

6. Create StorageCluster CR ocs-cluster-service-VMware.yaml that uses the monDataDirHostPath and localblock StorageClass.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
     - count: 1
       dataPVCTemplate:
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 2Ti
           storageClassName: localblock
           volumeMode: Block
       name: ocs-deviceset
       placement: {}
       portable: false
       replica: 3
       resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

7. Create StorageCluster CR.

   $ oc create -f ocs-cluster-service-VMware.yaml

   Example output:

   storagecluster.ocs.openshift.io/ocs-storagecluster created

Procedure

1. Create LocalVolume CR for block PVs.

   Example of LocalVolume CR local-storage-block.yaml using OCS label as node selector.

   apiVersion: local.storage.openshift.io/v1
   kind: LocalVolume
   metadata:
     name: local-block
     namespace: local-storage
     labels:
       app: ocs-storagecluster
   spec:
     nodeSelector:
       nodeSelectorTerms:
       - matchExpressions:
           - key: cluster.ocs.openshift.io/openshift-storage
             operator: In
             values:
             - ""
     storageClassDevices:
       - storageClassName: localblock
         volumeMode: Block
         devicePaths:
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY81260978128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY80440W5U128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB85AABDE128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0A60CB81128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0093D45E128A   # <-- modify this line
           - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYE46F6060128A   # <-- modify this line

2. Create the LocalVolume CR for block PVs.

   $ oc create -f local-storage-block.yaml

3. Check if the pods are created.

   $ oc -n local-storage get pods

4. Check if the PVs are created.

   $ oc get pv

   Example output:

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-150fdc87   2Ti      RWO            Delete           Available           localblock              2m11s
   local-pv-183bfc0a   2Ti      RWO            Delete           Available           localblock              2m11s
   local-pv-b2f5cb25   2Ti      RWO            Delete           Available           localblock              2m21s

5. Check for the new localblock StorageClass.

   $ oc get sc|egrep -e "localblock|NAME"

   Example output:

   NAME       PROVISIONER                  RECLAIMPOLICY
   localblock kubernetes.io/no-provisioner Delete
   
   VOLUMEBINDINGMODE    ALLOWVOLUMEEXPANSION AGE
   WaitForFirstConsumer false                4d23h

6. Create StorageCluster CR that uses the localblock StorageClass and the three PVs that are created.

7. Example of StorageCluster CR cluster-service-metal.yaml using monDataDirHostPath and localblock StorageClass.

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: ocs-storagecluster
     namespace: openshift-storage
   spec:
     manageNodes: false
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
     - count: 2
       dataPVCTemplate:
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 2Ti
           storageClassName: localblock
           volumeMode: Block
       name: ocs-deviceset
       placement: {}
       portable: false
       replica: 3
       resources: {}

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

8. Create the StorageCluster CR.

   $ oc create -f cluster-service-metal.yaml

   Example output:

   storagecluster.ocs.openshift.io/ocs-storagecluster created

Procedure

1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

2. Select openshift-storage from the Project drop down list.

   For more information on the amount of pods to expect for each component and how the amount of pods varies depending on the number of nodes and OSDs, see Table 3.1, “Pods corresponding to storage components for a three worker node cluster”

3. Verify that the following pods are in running and completed state by clicking on the Running and the Completed tabs:

   Table 3.1. Pods corresponding to storage components for a three worker node cluster

   Component	No. of pods	Name of the pod
   Number of pods that you must see for the following components:
   OpenShift Container Storage Operator	1	ocs-operator-*
   Rook-ceph Operator	1	rook-ceph-operator-*
   Multicloud Object Gateway	4	noobaa-operator-* noobaa-core-* nooba-db-* noobaa-endpoint-*
   Mon	3	rook-ceph-mon-* rook-ceph-mon-* rook-ceph-mon-* (on different nodes)
   rook-ceph-mgr	1	rook-ceph-mgr-* (on storage node)
   MDS	2	rook-ceph-mds-ocs-storagecluster-cephfilesystem-* (2 pods on different storage nodes)
   lib-bucket-provisioner	1	lib-bucket-provisioner--* (on any node)
   Number of pods for CSI vary depending on the number of nodes selected as storage nodes (a minimum of 3 nodes)
   CSI	10	cephfs (at least 5 pods) csi-cephfsplugin-* (1 on each node where storage is consumed, that is, 3 pods on different nodes) csi-cephfsplugin-provisioner-* (2 pods on different storage nodes if available) rbd (at least 5 pods in total) csi-rbdplugin-* (one on each node where storage is consumed, that is, 3 pods on different nodes) csi-rbdplugin-provisioner-* (2 pods on different storage nodes if available)
   rook-ceph-drain-canary	3	rook-ceph-drain-canary-* (3 pods, that is, one on each storage node)
   rook-ceph-crashcollector	3	rook-ceph-crashcollector-* (3 pods)
   Number of OSDs vary depending on Count and Replica defined for each StorageDeviceSet in StorageCluster.
   OSD	6	rook-ceph-osd-* (3 pods across different nodes) rook-ceph-osd-prepare-ocs-deviceset-* (3 pods across different nodes)

Procedure

1. List the pods and PVCs that are currently running in the openshift-monitoring namespace.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                           READY   STATUS    RESTARTS   AGE
   pod/alertmanager-main-0         3/3     Running   0          8d
   pod/alertmanager-main-1         3/3     Running   0          8d
   pod/alertmanager-main-2         3/3     Running   0          8d
   pod/cluster-monitoring-
   operator-84457656d-pkrxm        1/1     Running   0          8d
   pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
   pod/kube-state-metrics-
   7d86fb966-rvd9w                 3/3     Running   0          8d
   pod/node-exporter-25894         2/2     Running   0          8d
   pod/node-exporter-4dsd7         2/2     Running   0          8d
   pod/node-exporter-6p4zc         2/2     Running   0          8d
   pod/node-exporter-jbjvg         2/2     Running   0          8d
   pod/node-exporter-jj4t5         2/2     Running   0          6d18h
   pod/node-exporter-k856s         2/2     Running   0          6d18h
   pod/node-exporter-rf8gn         2/2     Running   0          8d
   pod/node-exporter-rmb5m         2/2     Running   0          6d18h
   pod/node-exporter-zj7kx         2/2     Running   0          8d
   pod/openshift-state-metrics-
   59dbd4f654-4clng                3/3     Running   0          8d
   pod/prometheus-adapter-
   5df5865596-k8dzn                1/1     Running   0          7d23h
   pod/prometheus-adapter-
   5df5865596-n2gj9                1/1     Running   0          7d23h
   pod/prometheus-k8s-0            6/6     Running   1          8d
   pod/prometheus-k8s-1            6/6     Running   1          8d
   pod/prometheus-operator-
   55cfb858c9-c4zd9                1/1     Running   0          6d21h
   pod/telemeter-client-
   78fc8fc97d-2rgfp                3/3     Running   0          8d
   
   NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
   persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d

2. Edit the monitoring configmap.

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Remove any config sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.

   Before editing

   . . . apiVersion: v1 data: config.yaml: | alertmanagerMain: volumeClaimTemplate: metadata: name: my-alertmanager-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-storagecluster-ceph-rbd prometheusK8s: volumeClaimTemplate: metadata: name: my-prometheus-claim spec: resources: requests: storage: 40Gi storageClassName: ocs-storagecluster-ceph-rbd kind: ConfigMap metadata: creationTimestamp: "2019-12-02T07:47:29Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "22110" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: fd6d988b-14d7-11ea-84ff-066035b9efa8 . . .

   After editing

   . . . apiVersion: v1 data: config.yaml: | kind: ConfigMap metadata: creationTimestamp: "2019-11-21T13:07:05Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "404352" selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config uid: d12c796a-0c5f-11ea-9832-063cd735b81c . . .

   In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Container Storage PVCs.

4. List the pods consuming the PVC.

   In this example, the alertmanagerMain and prometheusK8s pods that were consuming the PVCs are in the Terminating state. You can delete the PVCs once these pods are no longer using OpenShift Container Storage PVC.

   $ oc get pod,pvc -n openshift-monitoring
   NAME                                               READY   STATUS      RESTARTS AGE
   pod/alertmanager-main-0                            3/3     Terminating   0      10h
   pod/alertmanager-main-1                            3/3     Terminating   0      10h
   pod/alertmanager-main-2                            3/3     Terminating   0      10h
   pod/cluster-monitoring-operator-84cd9df668-zhjfn   1/1     Running       0      18h
   pod/grafana-5db6fd97f8-pmtbf                       2/2     Running       0      10h
   pod/kube-state-metrics-895899678-z2r9q             3/3     Running       0      10h
   pod/node-exporter-4njxv                            2/2     Running       0      18h
   pod/node-exporter-b8ckz                            2/2     Running       0      11h
   pod/node-exporter-c2vp5                            2/2     Running       0      18h
   pod/node-exporter-cq65n                            2/2     Running       0      18h
   pod/node-exporter-f5sm7                            2/2     Running       0      11h
   pod/node-exporter-f852c                            2/2     Running       0      18h
   pod/node-exporter-l9zn7                            2/2     Running       0      11h
   pod/node-exporter-ngbs8                            2/2     Running       0      18h
   pod/node-exporter-rv4v9                            2/2     Running       0      18h
   pod/openshift-state-metrics-77d5f699d8-69q5x       3/3     Running       0      10h
   pod/prometheus-adapter-765465b56-4tbxx             1/1     Running       0      10h
   pod/prometheus-adapter-765465b56-s2qg2             1/1     Running       0      10h
   pod/prometheus-k8s-0                               6/6     Terminating   1      9m47s
   pod/prometheus-k8s-1                               6/6     Terminating   1      9m47s
   pod/prometheus-operator-cbfd89f9-ldnwc             1/1     Running       0      43m
   pod/telemeter-client-7b5ddb4489-2xfpz              3/3     Running       0      10h
   
   NAME                                                      STATUS  VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-0   Bound    pvc-2eb79797-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-1   Bound    pvc-2ebeee54-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-alertmanager-claim-alertmanager-main-2   Bound    pvc-2ec6a9cf-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-0        Bound    pvc-3162a80c-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-storagecluster-ceph-rbd   19h
   persistentvolumeclaim/ocs-prometheus-claim-prometheus-k8s-1        Bound    pvc-316e99e2-1fed-11ea-93e1-0a88476a6a64   40Gi       RWO            ocs-storagecluster-ceph-rbd   19h

5. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.

   $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m

Procedure

1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

   $ oc edit configs.imageregistry.operator.openshift.io

   • For AWS:

     Before editing

     . . storage: pvc: claim: registry-cephfs-rwx-pvc . .

     After editing

     . . storage: . .

     In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

   • For VMware and baremetal:

     Before editing

     . . storage: pvc: claim: registry-cephfs-rwx-pvc . .

     After editing

     . . storage: emptyDir: {} . .

     In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

2. Delete the PVC.

   $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Procedure

1. Remove the ClusterLogging instance in the namespace.

   $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m

   The PVCs in the openshift-logging namespace are now safe to delete.

2. Delete PVCs.

   $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m