OpenShift Container Storage is now OpenShift Data Foundation starting with version 4.9.
Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 5. Uninstalling OpenShift Container Storage
5.1. Uninstalling OpenShift Container Storage in Internal mode Copier lienLien copié sur presse-papiers!
Use the steps in this section to uninstall OpenShift Container Storage.
Uninstall Annotations
Annotations on the Storage Cluster are used to change the behavior of the uninstall process. To define the uninstall behavior, the following two annotations have been introduced in the storage cluster:
-
uninstall.ocs.openshift.io/cleanup-policy: delete -
uninstall.ocs.openshift.io/mode: graceful
The below table provides information on the different values that can used with these annotations:
| Annotation | Value | Default | Behavior |
|---|---|---|---|
| cleanup-policy | delete | Yes |
Rook cleans up the physical drives and the |
| cleanup-policy | retain | No |
Rook does not clean up the physical drives and the |
| mode | graceful | Yes | Rook and NooBaa pauses the uninstall process until the PVCs and the OBCs are removed by the administrator/user |
| mode | forced | No | Rook and NooBaa proceeds with uninstall even if PVCs/OBCs provisioned using Rook and NooBaa exist respectively. |
You can change the cleanup policy or the uninstall mode by editing the value of the annotation by using the following commands:
oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
$ oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
$ oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
Prerequisites
- Ensure that the OpenShift Container Storage cluster is in a healthy state. The uninstall process can fail when some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in an unhealthy state, contact Red Hat Customer Support before uninstalling OpenShift Container Storage.
- Ensure that applications are not consuming persistent volume claims (PVCs) or object bucket claims (OBCs) using the storage classes provided by OpenShift Container Storage.
- If any custom resources (such as custom storage classes, cephblockpools) were created by the admin, they must be deleted by the admin after removing the resources which consumed them.
Procedure
Delete the volume snapshots that are using OpenShift Container Storage.
List the volume snapshots from all the namespaces.
oc get volumesnapshot --all-namespaces
$ oc get volumesnapshot --all-namespacesCopy to Clipboard Copied! Toggle word wrap Toggle overflow From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.
oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
$ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete PVCs and OBCs that are using OpenShift Container Storage.
In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.
If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.
Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.
See Section 5.2, “Removing monitoring stack from OpenShift Container Storage”
Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.
See Section 5.3, “Removing OpenShift Container Platform registry from OpenShift Container Storage”
Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.
See Section 5.4, “Removing the cluster logging operator from OpenShift Container Storage”
Delete other PVCs and OBCs provisioned using OpenShift Container Storage.
Given below is a sample script to identify the PVCs and OBCs provisioned using OpenShift Container Storage. The script ignores the PVCs that are used internally by Openshift Container Storage.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteOmit
RGW_PROVISIONERfor cloud platforms.Delete the OBCs.
oc delete obc <obc name> -n <project name>
$ oc delete obc <obc name> -n <project name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the PVCs.
oc delete pvc <pvc name> -n <project-name>
$ oc delete pvc <pvc name> -n <project-name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteEnsure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.
Delete the Storage Cluster object and wait for the removal of the associated resources.
oc delete -n openshift-storage storagecluster --all --wait=true
$ oc delete -n openshift-storage storagecluster --all --wait=trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check for cleanup pods if the
uninstall.ocs.openshift.io/cleanup-policywas set todelete(default) and ensure that their status isCompleted.oc get pods -n openshift-storage | grep -i cleanup
$ oc get pods -n openshift-storage | grep -i cleanup NAME READY STATUS RESTARTS AGE cluster-cleanup-job-<xx> 0/1 Completed 0 8m35s cluster-cleanup-job-<yy> 0/1 Completed 0 8m35s cluster-cleanup-job-<zz> 0/1 Completed 0 8m35sCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm that the directory
/var/lib/rookis now empty. This directory will be empty only if theuninstall.ocs.openshift.io/cleanup-policyannotation was set todelete(default).for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; done$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow If encryption was enabled at the time of install, remove
dm-cryptmanageddevice-mappermapping from OSD devices on all the OpenShift Container Storage nodes.Create a
debugpod andchrootto the host on the storage node.oc debug node/<node name> chroot /host
$ oc debug node/<node name> $ chroot /hostCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get Device names and make note of the OpenShift Container Storage devices.
dmsetup ls
$ dmsetup ls ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the mapped device.
cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
$ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcryptCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf the above command gets stuck due to insufficient privileges, run the following commands:
-
Press
CTRL+Zto exit the above command. Find PID of the process which was stuck.
ps -ef | grep crypt
$ ps -ef | grep cryptCopy to Clipboard Copied! Toggle word wrap Toggle overflow Terminate the process using
killcommand.kill -9 <PID>
$ kill -9 <PID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the device name is removed.
dmsetup ls
$ dmsetup lsCopy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Press
Delete the namespace and wait till the deletion is complete. You will need to switch to another project if
openshift-storageis the active project.For example:
oc project default oc delete project openshift-storage --wait=true --timeout=5m
$ oc project default $ oc delete project openshift-storage --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow The project is deleted if the following command returns a NotFound error.
oc get project openshift-storage
$ oc get project openshift-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteWhile uninstalling OpenShift Container Storage, if
namespaceis not deleted completely and remains inTerminatingstate, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.- Delete the local storage operator configurations if you have deployed OpenShift Container Storage using local storage devices. See Removing local storage operator configurations.
Unlabel the storage nodes.
oc label nodes --all cluster.ocs.openshift.io/openshift-storage- oc label nodes --all topology.rook.io/rack-
$ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- $ oc label nodes --all topology.rook.io/rack-Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the OpenShift Container Storage taint if the nodes were tainted.
oc adm taint nodes --all node.ocs.openshift.io/storage-
$ oc adm taint nodes --all node.ocs.openshift.io/storage-Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the
Releasedstate, delete it.oc get pv oc delete pv <pv name>
$ oc get pv $ oc delete pv <pv name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the Multicloud Object Gateway storageclass.
oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m
$ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove
CustomResourceDefinitions.oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m
$ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: To ensure that the vault keys are deleted permanently you need to manually delete the metadata associated with the vault key.
NoteExecute this step only if Vault Key/Value (KV) secret engine API, version 2 is used for cluster-wide encryption with Key Management System (KMS) since the vault keys are marked as deleted and not permanently deleted during the uninstallation of OpenShift Container Storage. You can always restore it later if required.
List the keys in the vault.
vault kv list <backend_path>
$ vault kv list <backend_path>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <backend_path>Is the path in the vault where the encryption keys are stored.
For example:
vault kv list kv-v2
$ vault kv list kv-v2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
List the metadata associated with the vault key.
vault kv get kv-v2/<key>
$ vault kv get kv-v2/<key>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For the Multicloud Object Gateway (MCG) key:
vault kv get kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
$ vault kv get kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <key>Is the encryption key.
For Example:
vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
$ vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete the metadata.
vault kv metadata delete kv-v2/<key>
$ vault kv metadata delete kv-v2/<key>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For the MCG key:
vault kv metadata delete kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
$ vault kv metadata delete kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <key>Is the encryption key.
For Example:
vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
$ vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- Repeat these steps to delete the metadata associated with all the vault keys.
To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,
-
Click Home
Overview to access the dashboard. - Verify that the Persistent Storage and Object Service tabs no longer appear next to the Cluster tab.
-
Click Home
5.1.1. Removing local storage operator configurations Copier lienLien copié sur presse-papiers!
Use the instructions in this section only if you have deployed OpenShift Container Storage using local storage devices.
For OpenShift Container Storage deployments only using localvolume resources, go directly to step 8.
Procedure
-
Identify the
LocalVolumeSetand the correspondingStorageClassNamebeing used by OpenShift Container Storage. Set the variable SC to the
StorageClassproviding theLocalVolumeSet.export SC="<StorageClassName>"
$ export SC="<StorageClassName>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
LocalVolumeSet.oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage
$ oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local storage PVs for the given
StorageClassName.oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv$ oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pvCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
StorageClassName.oc delete sc $SC
$ oc delete sc $SCCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the symlinks created by the
LocalVolumeSet.[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete
LocalVolumeDiscovery.oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage
$ oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Removing
LocalVolumeresources (if any).Use the following steps to remove the
LocalVolumeresources that were used to provision PVs in the current or previous OpenShift Container Storage version. Also, ensure that these resources are not being used by other tenants on the cluster.For each of the local volumes, do the following:
-
Identify the
LocalVolumeand the correspondingStorageClassNamebeing used by OpenShift Container Storage. Set the variable LV to the name of the LocalVolume and variable SC to the name of the StorageClass
For example:
LV=local-block SC=localblock
$ LV=local-block $ SC=localblockCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local volume resource.
oc delete localvolume -n openshift-local-storage --wait=true $LV
$ oc delete localvolume -n openshift-local-storage --wait=true $LVCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the remaining PVs and StorageClasses if they exist.
oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m oc delete storageclass $SC --wait --timeout=5m$ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m $ oc delete storageclass $SC --wait --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Clean up the artifacts from the storage nodes for that resource.
[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done$ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Identify the
5.2. Removing monitoring stack from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
Use this section to clean up the monitoring stack from OpenShift Container Storage.
The PVCs that are created as a part of configuring the monitoring stack are in the openshift-monitoring namespace.
Prerequisites
PVCs are configured to use OpenShift Container Platform monitoring stack.
For information, see configuring monitoring stack.
Procedure
List the pods and PVCs that are currently running in the
openshift-monitoringnamespace.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the monitoring
configmap.oc -n openshift-monitoring edit configmap cluster-monitoring-config
$ oc -n openshift-monitoring edit configmap cluster-monitoring-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove any
configsections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.Before editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
alertmanagerMainandprometheusK8smonitoring components are using the OpenShift Container Storage PVCs.Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.
oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
$ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.3. Removing OpenShift Container Platform registry from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
Use this section to clean up OpenShift Container Platform registry from OpenShift Container Storage. If you want to configure an alternative storage, see image registry
The PVCs that are created as a part of configuring OpenShift Container Platform registry are in the openshift-image-registry namespace.
Prerequisites
- The image registry should have been configured to use an OpenShift Container Storage PVC.
Procedure
Edit the
configs.imageregistry.operator.openshift.ioobject and remove the content in the storage section.oc edit configs.imageregistry.operator.openshift.io
$ oc edit configs.imageregistry.operator.openshift.ioCopy to Clipboard Copied! Toggle word wrap Toggle overflow Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow + In this example, the PVC is called
registry-cephfs-rwx-pvc, which is now safe to delete.Delete the PVC.
oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.4. Removing the cluster logging operator from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
Use this section to clean up the cluster logging operator from OpenShift Container Storage.
The PVCs that are created as a part of configuring cluster logging operator are in the openshift-logging namespace.
Prerequisites
- The cluster logging instance should have been configured to use OpenShift Container Storage PVCs.
Procedure
Remove the
ClusterLogginginstance in the namespace.oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
$ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow The PVCs in the
openshift-loggingnamespace are now safe to delete.Delete PVCs.
oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow