Chapter 7. Uninstalling OpenShift Data Foundation

7.1. Uninstalling OpenShift Data Foundation in Internal-attached devices mode
Copy link

Use the steps in this section to uninstall OpenShift Data Foundation.

Uninstall Annotations

Annotations on the Storage Cluster are used to change the behavior of the uninstall process. To define the uninstall behavior, the following two annotations have been introduced in the storage cluster:

uninstall.ocs.openshift.io/cleanup-policy: delete
uninstall.ocs.openshift.io/mode: graceful

The following table provides information on the different values that can used with these annotations:

Expand

Table 7.1. uninstall.ocs.openshift.io uninstall annotations descriptions
Annotation	Value	Default	Behavior
cleanup-policy	delete	Yes	Rook cleans up the physical drives and the `DataDirHostPath`
cleanup-policy	retain	No	Rook does not clean up the physical drives and the `DataDirHostPath`
mode	graceful	Yes	Rook and NooBaa pauses the uninstall process until the administrator/user removes the Persistent Volume Claims (PVCs) and Object Bucket Claims (OBCs)
mode	forced	No	Rook and NooBaa proceeds with uninstall even if the PVCs/OBCs provisioned using Rook and NooBaa exist respectively

Edit the value of the annotation to change the cleanup policy or the uninstall mode.

oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite

$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite

Copy to Clipboard

Toggle word wrap

oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite

$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite

Copy to Clipboard

Toggle word wrap

Expected output for both commands:

storagecluster.ocs.openshift.io/ocs-storagecluster annotated

storagecluster.ocs.openshift.io/ocs-storagecluster annotated

Copy to Clipboard

Toggle word wrap

Prerequisites

Ensure that the OpenShift Data Foundation cluster is in a healthy state. The uninstall process can fail when some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in an unhealthy state, contact Red Hat Customer Support before uninstalling OpenShift Data Foundation.
Ensure that applications are not consuming persistent volume claims (PVCs) or object bucket claims (OBCs) using the storage classes provided by OpenShift Data Foundation.
If any custom resources (such as custom storage classes, cephblockpools) were created by the admin, they must be deleted by the admin after removing the resources which consumed them.

Procedure

Delete the volume snapshots that are using OpenShift Data Foundation.
1. List the volume snapshots from all the namespaces.
  $ oc get volumesnapshot --all-namespaces
  Copy to Clipboard Toggle word wrap
2. From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Data Foundation.
  $ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
  Copy to Clipboard Toggle word wrap
  <VOLUME-SNAPSHOT-NAME>
  Is the name of the volume snapshot
  <NAMESPACE>
  Is the project namespace

Delete PVCs and OBCs that are using OpenShift Data Foundation.

In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Data Foundation are deleted.

If you want to delete the Storage Cluster without deleting the PVCs, you can set the uninstall mode annotation to forced and skip this step. Doing so results in orphan PVCs and OBCs in the system.

Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Data Foundation.
See Removing monitoring stack from OpenShift Data Foundation
Delete OpenShift Container Platform Registry PVCs using OpenShift Data Foundation.
Removing OpenShift Container Platform registry from OpenShift Data Foundation
Delete OpenShift Container Platform logging PVCs using OpenShift Data Foundation.
Removing the cluster logging operator from OpenShift Data Foundation

Delete the other PVCs and OBCs provisioned using OpenShift Data Foundation.

Given below is a sample script to identify the PVCs and OBCs provisioned using OpenShift Data Foundation. The script ignores the PVCs that are used internally by OpenShift Data Foundation.

#!/bin/bash

RBD_PROVISIONER="openshift-storage.rbd.csi.ceph.com"
CEPHFS_PROVISIONER="openshift-storage.cephfs.csi.ceph.com"
NOOBAA_PROVISIONER="openshift-storage.noobaa.io/obc"
RGW_PROVISIONER="openshift-storage.ceph.rook.io/bucket"

NOOBAA_DB_PVC="noobaa-db"
NOOBAA_BACKINGSTORE_PVC="noobaa-default-backing-store-noobaa-pvc"

# Find all the OCS StorageClasses
OCS_STORAGECLASSES=$(oc get storageclasses | grep -e "$RBD_PROVISIONER" -e "$CEPHFS_PROVISIONER" -e "$NOOBAA_PROVISIONER" -e "$RGW_PROVISIONER" | awk '{print $1}')

# List PVCs in each of the StorageClasses
for SC in $OCS_STORAGECLASSES
do
        echo "======================================================================"
        echo "$SC StorageClass PVCs and OBCs"
        echo "======================================================================"
        oc get pvc  --all-namespaces --no-headers 2>/dev/null | grep $SC | grep -v -e "$NOOBAA_DB_PVC" -e "$NOOBAA_BACKINGSTORE_PVC"
        oc get obc  --all-namespaces --no-headers 2>/dev/null | grep $SC
        echo
done

#!/bin/bash

RBD_PROVISIONER="openshift-storage.rbd.csi.ceph.com"
CEPHFS_PROVISIONER="openshift-storage.cephfs.csi.ceph.com"
NOOBAA_PROVISIONER="openshift-storage.noobaa.io/obc"
RGW_PROVISIONER="openshift-storage.ceph.rook.io/bucket"

NOOBAA_DB_PVC="noobaa-db"
NOOBAA_BACKINGSTORE_PVC="noobaa-default-backing-store-noobaa-pvc"

# Find all the OCS StorageClasses
OCS_STORAGECLASSES=$(oc get storageclasses | grep -e "$RBD_PROVISIONER" -e "$CEPHFS_PROVISIONER" -e "$NOOBAA_PROVISIONER" -e "$RGW_PROVISIONER" | awk '{print $1}')

# List PVCs in each of the StorageClasses
for SC in $OCS_STORAGECLASSES
do
        echo "======================================================================"
        echo "$SC StorageClass PVCs and OBCs"
        echo "======================================================================"
        oc get pvc  --all-namespaces --no-headers 2>/dev/null | grep $SC | grep -v -e "$NOOBAA_DB_PVC" -e "$NOOBAA_BACKINGSTORE_PVC"
        oc get obc  --all-namespaces --no-headers 2>/dev/null | grep $SC
        echo
done

Copy to Clipboard

Toggle word wrap

Note

Omit RGW_PROVISIONER for cloud platforms.

Delete the OBCs.
```
oc delete obc <obc-name> -n <project-name>
```
```
$ oc delete obc <obc-name> -n <project-name>
```
Copy to Clipboard Toggle word wrap
<obc-name>
Is the name of the OBC
<project-name>
Is the name of the project
Delete the PVCs.
```
oc delete pvc <pvc-name> -n <project-name>
```
```
$ oc delete pvc <pvc-name> -n <project-name>
```
Copy to Clipboard Toggle word wrap
<pvc-name>
Is the name of the PVC
<project-name>
Is the name of the project
Note
Ensure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.

Delete the Storage System object and wait for the removal of the associated resources.
```
oc delete -n openshift-storage storagesystem --all --wait=true
```
```
$ oc delete -n openshift-storage storagesystem --all --wait=true
```
Copy to Clipboard Toggle word wrap

Check the cleanup pods if the uninstall.ocs.openshift.io/cleanup-policy was set to delete(default) and ensure that their status is Completed.

oc get pods -n openshift-storage | grep -i cleanup

$ oc get pods -n openshift-storage | grep -i cleanup

Copy to Clipboard

Toggle word wrap

Example output:

NAME                                READY   STATUS      RESTARTS   AGE
cluster-cleanup-job-<xx>        	0/1     Completed   0          8m35s
cluster-cleanup-job-<yy>     		0/1     Completed   0          8m35s
cluster-cleanup-job-<zz>     		0/1     Completed   0          8m35s

NAME                                READY   STATUS      RESTARTS   AGE
cluster-cleanup-job-<xx>        	0/1     Completed   0          8m35s
cluster-cleanup-job-<yy>     		0/1     Completed   0          8m35s
cluster-cleanup-job-<zz>     		0/1     Completed   0          8m35s

Copy to Clipboard

Toggle word wrap

Confirm that the directory /var/lib/rook is now empty. This directory is empty only if the uninstall.ocs.openshift.io/cleanup-policy annotation was set to delete(default).

for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done

$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done

Copy to Clipboard

Toggle word wrap

If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSDs on all the OpenShift Data Foundation nodes.
1. Create a debug pod and chroot to the host on the storage node.
  $ oc debug node/<node-name>
  Copy to Clipboard Toggle word wrap
  $ chroot /host
  Copy to Clipboard Toggle word wrap
  <node-name>
  Is the name of the node
2. Get Device names and make note of the OpenShift Data Foundation devices.
  $ dmsetup ls
  Copy to Clipboard Toggle word wrap
  Example output:
  ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)
  Copy to Clipboard Toggle word wrap
3. Remove the mapped device.
  $ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
  Copy to Clipboard Toggle word wrap
  Important
  If the above command gets stuck due to insufficient privileges, run the following commands:
  Press CTRL+Z to exit the above command.
  Find PID of the process which was stuck.
  
  $ ps -ef | grep crypt
  
  Copy to Clipboard Toggle word wrap
  
  Terminate the process using kill command.
  
  $ kill -9 <PID>
  
  Copy to Clipboard Toggle word wrap
  
  <PID>
  Is the process ID
  Verify that the device name is removed.
  
  $ dmsetup ls
  
  Copy to Clipboard Toggle word wrap
Delete the namespace and wait till the deletion is complete. You need to switch to another project if openshift-storage is the active project.
For example:
```
oc project default
```
```
$ oc project default
```
Copy to Clipboard Toggle word wrap
```
oc delete project openshift-storage --wait=true --timeout=5m
```
```
$ oc delete project openshift-storage --wait=true --timeout=5m
```
Copy to Clipboard Toggle word wrap
The project is deleted if the following command returns a NotFound error.
```
oc get project openshift-storage
```
```
$ oc get project openshift-storage
```
Copy to Clipboard Toggle word wrap
Note
While uninstalling OpenShift Data Foundation, if namespace is not deleted completely and remains in Terminating state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.
Delete local storage operator configurations if you have deployed OpenShift Data Foundation using local storage devices. See Removing local storage operator configurations.

Unlabel the storage nodes.

oc label nodes  --all cluster.ocs.openshift.io/openshift-storage-

$ oc label nodes  --all cluster.ocs.openshift.io/openshift-storage-

Copy to Clipboard

Toggle word wrap

oc label nodes  --all topology.rook.io/rack-

$ oc label nodes  --all topology.rook.io/rack-

Copy to Clipboard

Toggle word wrap

Remove the OpenShift Data Foundation taint if the nodes were tainted.
```
oc adm taint nodes --all node.ocs.openshift.io/storage-
```
```
$ oc adm taint nodes --all node.ocs.openshift.io/storage-
```
Copy to Clipboard Toggle word wrap
Confirm all the Persistent volumes (PVs) provisioned using OpenShift Data Foundation are deleted. If there is any PV left in the Released state, delete it.
```
oc get pv
```
```
$ oc get pv
```
Copy to Clipboard Toggle word wrap
```
oc delete pv <pv-name>
```
```
$ oc delete pv <pv-name>
```
Copy to Clipboard Toggle word wrap
<pv-name>
Is the name of the PV

Remove the CustomResourceDefinitions.

oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io storagesystems.odf.openshift.io --wait=true --timeout=5m

$ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io storagesystems.odf.openshift.io --wait=true --timeout=5m

Copy to Clipboard

Toggle word wrap

To ensure that OpenShift Data Foundation is uninstalled completely, on the OpenShift Container Platform Web Console,
1. Click Storage.
2. Verify that OpenShift Data Foundation no longer appears under Storage.

7.1.1. Removing local storage operator configurations
Copy link

Use the instructions in this section only if you have deployed OpenShift Data Foundation using local storage devices.

Note

For OpenShift Data Foundation deployments only using localvolume resources, go directly to step 8.

Procedure

Identify the LocalVolumeSet and the corresponding StorageClassName being used by OpenShift Data Foundation.
```
oc get localvolumesets.local.storage.openshift.io -n openshift-local-storage
```
```
$ oc get localvolumesets.local.storage.openshift.io -n openshift-local-storage
```
Copy to Clipboard Toggle word wrap
Set the variable SC to the StorageClass providing the LocalVolumeSet.
```
export SC="<StorageClassName>"
```
```
$ export SC="<StorageClassName>"
```
Copy to Clipboard Toggle word wrap

List and note the devices to be cleaned up later. Inorder to list the device ids of the disks, follow the procedure mentioned here, See Find the available storage devices.

Example output:

/dev/disk/by-id/scsi-360050763808104bc28000000000000eb
/dev/disk/by-id/scsi-360050763808104bc28000000000000ef
/dev/disk/by-id/scsi-360050763808104bc28000000000000f3

/dev/disk/by-id/scsi-360050763808104bc28000000000000eb
/dev/disk/by-id/scsi-360050763808104bc28000000000000ef
/dev/disk/by-id/scsi-360050763808104bc28000000000000f3

Copy to Clipboard

Toggle word wrap

Delete the LocalVolumeSet.

oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage

$ oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage

Copy to Clipboard

Toggle word wrap

Delete the local storage PVs for the given StorageClassName.

oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv

$ oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv

Copy to Clipboard

Toggle word wrap

Delete the StorageClassName.
```
oc delete sc $SC
```
```
$ oc delete sc $SC
```
Copy to Clipboard Toggle word wrap

Delete the symlinks created by the LocalVolumeSet.

[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done

[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done

Copy to Clipboard

Toggle word wrap

Delete LocalVolumeDiscovery.

oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage

$ oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage

Copy to Clipboard

Toggle word wrap

Remove the LocalVolume resources (if any).

Use the following steps to remove the LocalVolume resources that were used to provision PVs in the current or previous OpenShift Data Foundation version. Also, ensure that these resources are not being used by other tenants on the cluster.

For each of the local volumes, do the following:

Identify the LocalVolume and the corresponding StorageClassName being used by OpenShift Data Foundation.
```
oc get localvolume.local.storage.openshift.io -n openshift-local-storage
```
```
$ oc get localvolume.local.storage.openshift.io -n openshift-local-storage
```
Copy to Clipboard Toggle word wrap
Set the variable LV to the name of the LocalVolume and variable SC to the name of the StorageClass
For example:
```
LV=local-block
SC=localblock
```
```
$ LV=local-block
$ SC=localblock
```
Copy to Clipboard Toggle word wrap

List and note the devices to be cleaned up later.

oc get localvolume -n openshift-local-storage $LV -o jsonpath='{ .spec.storageClassDevices[].devicePaths[] }{"\n"}'

$ oc get localvolume -n openshift-local-storage $LV -o jsonpath='{ .spec.storageClassDevices[].devicePaths[] }{"\n"}'

Copy to Clipboard

Toggle word wrap

Example output:

/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde

/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde

Copy to Clipboard

Toggle word wrap

Delete the local volume resource.

oc delete localvolume -n openshift-local-storage --wait=true $LV

$ oc delete localvolume -n openshift-local-storage --wait=true $LV

Copy to Clipboard

Toggle word wrap

Delete the remaining PVs and StorageClasses if they exist.

oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m
oc delete storageclass $SC --wait --timeout=5m

$ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m
$ oc delete storageclass $SC --wait --timeout=5m

Copy to Clipboard

Toggle word wrap

Clean up the artifacts from the storage nodes for that resource.

[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done

$ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done

Copy to Clipboard

Toggle word wrap

Example output:

Starting pod/node-xxx-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-yyy-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-zzz-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...

Starting pod/node-xxx-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-yyy-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-zzz-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...

Copy to Clipboard

Toggle word wrap

Wipe the disks for each of the local volumesets or local volumes listed in step 1 and 8 respectively so that they can be reused.

List the storage nodes.

oc get nodes -l cluster.ocs.openshift.io/openshift-storage=

oc get nodes -l cluster.ocs.openshift.io/openshift-storage=

Copy to Clipboard

Toggle word wrap

Example output:

NAME      STATUS   ROLES    AGE     VERSION
node-xxx  Ready    worker   4h45m  v1.18.3+6c42de8
node-yyy  Ready    worker   4h46m  v1.18.3+6c42de8
node-zzz  Ready    worker   4h45m  v1.18.3+6c42de8

NAME      STATUS   ROLES    AGE     VERSION
node-xxx  Ready    worker   4h45m  v1.18.3+6c42de8
node-yyy  Ready    worker   4h46m  v1.18.3+6c42de8
node-zzz  Ready    worker   4h45m  v1.18.3+6c42de8

Copy to Clipboard

Toggle word wrap

Obtain the node console and execute chroot /host command when the prompt appears.

oc debug node/node-xxx
Starting pod/node-xxx-debug …
To use host binaries, run `chroot /host`
Pod IP: w.x.y.z
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host

$ oc debug node/node-xxx
Starting pod/node-xxx-debug …
To use host binaries, run `chroot /host`
Pod IP: w.x.y.z
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host

Copy to Clipboard

Toggle word wrap

Store the disk paths in the DISKS variable within quotes. For the list of disk paths, see step 3 and step 8.c for local volumeset and local volume respectively.

Example output:

DISKS="/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3 "
or
DISKS="/dev/sdb /dev/sdc /dev/sdd /dev/sde ".

sh-4.4# DISKS="/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3 "
or
sh-4.2# DISKS="/dev/sdb /dev/sdc /dev/sdd /dev/sde ".

Copy to Clipboard

Toggle word wrap

Run sgdisk --zap-all on all the disks.

for disk in $DISKS; do sgdisk --zap-all $disk;done

sh-4.4# for disk in $DISKS; do sgdisk --zap-all $disk;done

Copy to Clipboard

Toggle word wrap

Example output:

Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.

Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.

Copy to Clipboard

Toggle word wrap

Exit the shell and repeat for the other nodes.
```
exit
exit
exit
exit
Removing debug pod ...
```
```
sh-4.4# exit
exit
sh-4.2# exit
exit
Removing debug pod ...
```
Copy to Clipboard Toggle word wrap

Delete the openshift-local-storage namespace and wait till the deletion is complete. You will need to switch to another project if the openshift-local-storage namespace is the active project.
For example:
```
oc project default
oc delete project openshift-local-storage --wait=true --timeout=5m
```
```
$ oc project default
$ oc delete project openshift-local-storage --wait=true --timeout=5m
```
Copy to Clipboard Toggle word wrap
The project is deleted if the following command returns a NotFound error.
```
oc get project openshift-local-storage
```
```
$ oc get project openshift-local-storage
```
Copy to Clipboard Toggle word wrap

7.2. Removing monitoring stack from OpenShift Data Foundation
Copy link

Use this section to clean up the monitoring stack from OpenShift Data Foundation.

The Persistent Volume Claims (PVCs) that are created as a part of configuring the monitoring stack are in the openshift-monitoring namespace.

Prerequisites

PVCs are configured to use the OpenShift Container Platform monitoring stack.
For more information, see configuring monitoring stack.

Procedure

List the pods and PVCs that are currently running in the openshift-monitoring namespace.

oc get pod,pvc -n openshift-monitoring

$ oc get pod,pvc -n openshift-monitoring

Copy to Clipboard

Toggle word wrap

Example output:

NAME                           READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0         3/3     Running   0          8d
pod/alertmanager-main-1         3/3     Running   0          8d
pod/alertmanager-main-2         3/3     Running   0          8d
pod/cluster-monitoring-
operator-84457656d-pkrxm        1/1     Running   0          8d
pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
pod/kube-state-metrics-
7d86fb966-rvd9w                 3/3     Running   0          8d
pod/node-exporter-25894         2/2     Running   0          8d
pod/node-exporter-4dsd7         2/2     Running   0          8d
pod/node-exporter-6p4zc         2/2     Running   0          8d
pod/node-exporter-jbjvg         2/2     Running   0          8d
pod/node-exporter-jj4t5         2/2     Running   0          6d18h
pod/node-exporter-k856s         2/2     Running   0          6d18h
pod/node-exporter-rf8gn         2/2     Running   0          8d
pod/node-exporter-rmb5m         2/2     Running   0          6d18h
pod/node-exporter-zj7kx         2/2     Running   0          8d
pod/openshift-state-metrics-
59dbd4f654-4clng                3/3     Running   0          8d
pod/prometheus-adapter-
5df5865596-k8dzn                1/1     Running   0          7d23h
pod/prometheus-adapter-
5df5865596-n2gj9                1/1     Running   0          7d23h
pod/prometheus-k8s-0            6/6     Running   1          8d
pod/prometheus-k8s-1            6/6     Running   1          8d
pod/prometheus-operator-
55cfb858c9-c4zd9                1/1     Running   0          6d21h
pod/telemeter-client-
78fc8fc97d-2rgfp                3/3     Running   0          8d

NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d

NAME                           READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0         3/3     Running   0          8d
pod/alertmanager-main-1         3/3     Running   0          8d
pod/alertmanager-main-2         3/3     Running   0          8d
pod/cluster-monitoring-
operator-84457656d-pkrxm        1/1     Running   0          8d
pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
pod/kube-state-metrics-
7d86fb966-rvd9w                 3/3     Running   0          8d
pod/node-exporter-25894         2/2     Running   0          8d
pod/node-exporter-4dsd7         2/2     Running   0          8d
pod/node-exporter-6p4zc         2/2     Running   0          8d
pod/node-exporter-jbjvg         2/2     Running   0          8d
pod/node-exporter-jj4t5         2/2     Running   0          6d18h
pod/node-exporter-k856s         2/2     Running   0          6d18h
pod/node-exporter-rf8gn         2/2     Running   0          8d
pod/node-exporter-rmb5m         2/2     Running   0          6d18h
pod/node-exporter-zj7kx         2/2     Running   0          8d
pod/openshift-state-metrics-
59dbd4f654-4clng                3/3     Running   0          8d
pod/prometheus-adapter-
5df5865596-k8dzn                1/1     Running   0          7d23h
pod/prometheus-adapter-
5df5865596-n2gj9                1/1     Running   0          7d23h
pod/prometheus-k8s-0            6/6     Running   1          8d
pod/prometheus-k8s-1            6/6     Running   1          8d
pod/prometheus-operator-
55cfb858c9-c4zd9                1/1     Running   0          6d21h
pod/telemeter-client-
78fc8fc97d-2rgfp                3/3     Running   0          8d

NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d

Copy to Clipboard

Toggle word wrap

Edit the monitoring configmap.

oc -n openshift-monitoring edit configmap cluster-monitoring-config

$ oc -n openshift-monitoring edit configmap cluster-monitoring-config

Copy to Clipboard

Toggle word wrap

Remove any config sections that reference the OpenShift Data Foundation storage classes as shown in the following example and save it.

Before editing

Expand

.
.
.
apiVersion: v1
data:
  config.yaml: |
    alertmanagerMain:
      volumeClaimTemplate:
        metadata:
          name: my-alertmanager-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
    prometheusK8s:
      volumeClaimTemplate:
        metadata:
          name: my-prometheus-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
kind: ConfigMap
metadata:
  creationTimestamp: "2019-12-02T07:47:29Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "22110"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: fd6d988b-14d7-11ea-84ff-066035b9efa8
.
.
.

.
.
.
apiVersion: v1
data:
  config.yaml: |
    alertmanagerMain:
      volumeClaimTemplate:
        metadata:
          name: my-alertmanager-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
    prometheusK8s:
      volumeClaimTemplate:
        metadata:
          name: my-prometheus-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
kind: ConfigMap
metadata:
  creationTimestamp: "2019-12-02T07:47:29Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "22110"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: fd6d988b-14d7-11ea-84ff-066035b9efa8
.
.
.

Copy to Clipboard

Toggle word wrap

After editing

Expand

.
.
.
apiVersion: v1
data:
  config.yaml: |
kind: ConfigMap
metadata:
  creationTimestamp: "2019-11-21T13:07:05Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "404352"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: d12c796a-0c5f-11ea-9832-063cd735b81c
.
.
.

.
.
.
apiVersion: v1
data:
  config.yaml: |
kind: ConfigMap
metadata:
  creationTimestamp: "2019-11-21T13:07:05Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "404352"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: d12c796a-0c5f-11ea-9832-063cd735b81c
.
.
.

Copy to Clipboard

Toggle word wrap

In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Data Foundation PVCs.

Delete the relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.
```
oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
```
```
$ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
```
Copy to Clipboard Toggle word wrap
<pvc-name>
Is the name of the PVC

7.3. Removing OpenShift Container Platform registry from OpenShift Data Foundation
Copy link

Use this section to clean up the OpenShift Container Platform registry from OpenShift Data Foundation. If you want to configure an alternative storage, see Image registry.

The Persistent Volume Claims (PVCs) that are created as a part of configuring the OpenShift Container Platform registry are in the openshift-image-registry namespace.

Prerequisites

The image registry must have been configured to use an OpenShift Data Foundation PVC.

Procedure

Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section.

oc edit configs.imageregistry.operator.openshift.io

$ oc edit configs.imageregistry.operator.openshift.io

Copy to Clipboard

Toggle word wrap

Expand

Before editing

.
.
.

storage:
    pvc:
        claim: registry-cephfs-rwx-pvc
.
.
.

.
.
.

storage:
    pvc:
        claim: registry-cephfs-rwx-pvc
.
.
.

Copy to Clipboard

Toggle word wrap

After editing

.
.
.
storage:
  emptyDir: {}
.
.
.

.
.
.
storage:
  emptyDir: {}
.
.
.

Copy to Clipboard

Toggle word wrap

In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

Delete the PVC.

oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

$ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m

Copy to Clipboard

Toggle word wrap

<pvc-name>: Is the name of the PVC

7.4. Removing the cluster logging operator from OpenShift Data Foundation
Copy link

Use this section to clean up the cluster logging operator from OpenShift Data Foundation.

The Persistent Volume Claims (PVCs) that are created as a part of configuring the cluster logging operator are in the openshift-logging namespace.

Prerequisites

The cluster logging instance should have been configured to use the OpenShift Data Foundation PVCs.

Procedure

Remove the ClusterLogging instance in the namespace.
```
oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
```
```
$ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
```
Copy to Clipboard Toggle word wrap
The PVCs in the openshift-logging namespace are now safe to delete.

Delete the PVCs.

oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

$ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m

Copy to Clipboard

Toggle word wrap

<pvc-name>: Is the name of the PVC

Chapter 7. Uninstalling OpenShift Data Foundation

7.1. Uninstalling OpenShift Data Foundation in Internal-attached devices mode
Copy link

7.1.1. Removing local storage operator configurations
Copy link

7.2. Removing monitoring stack from OpenShift Data Foundation
Copy link

7.3. Removing OpenShift Container Platform registry from OpenShift Data Foundation
Copy link

7.4. Removing the cluster logging operator from OpenShift Data Foundation
Copy link

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 7. Uninstalling OpenShift Data Foundation

7.1. Uninstalling OpenShift Data Foundation in Internal-attached devices modeCopy linkLink copied to clipboard!

7.1.1. Removing local storage operator configurationsCopy linkLink copied to clipboard!

7.2. Removing monitoring stack from OpenShift Data FoundationCopy linkLink copied to clipboard!

7.3. Removing OpenShift Container Platform registry from OpenShift Data FoundationCopy linkLink copied to clipboard!

7.4. Removing the cluster logging operator from OpenShift Data FoundationCopy linkLink copied to clipboard!

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

7.1. Uninstalling OpenShift Data Foundation in Internal-attached devices mode
Copy link

7.1.1. Removing local storage operator configurations
Copy link

7.2. Removing monitoring stack from OpenShift Data Foundation
Copy link

7.3. Removing OpenShift Container Platform registry from OpenShift Data Foundation
Copy link

7.4. Removing the cluster logging operator from OpenShift Data Foundation
Copy link