OpenShift Container Storage is now OpenShift Data Foundation starting with version 4.9.
Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 5. Uninstalling OpenShift Container Storage
5.1. Uninstalling OpenShift Container Storage in Internal mode Copier lienLien copié sur presse-papiers!
Use the steps in this section to uninstall OpenShift Container Storage.
Uninstall Annotations
Annotations on the Storage Cluster are used to change the behavior of the uninstall process. To define the uninstall behavior, the following two annotations have been introduced in the storage cluster:
-
uninstall.ocs.openshift.io/cleanup-policy: delete
-
uninstall.ocs.openshift.io/mode: graceful
The below table provides information on the different values that can used with these annotations:
Annotation | Value | Default | Behavior |
---|---|---|---|
cleanup-policy | delete | Yes |
Rook cleans up the physical drives and the |
cleanup-policy | retain | No |
Rook does not clean up the physical drives and the |
mode | graceful | Yes | Rook and NooBaa pauses the uninstall process until the PVCs and the OBCs are removed by the administrator/user |
mode | forced | No | Rook and NooBaa proceeds with uninstall even if PVCs/OBCs provisioned using Rook and NooBaa exist respectively. |
You can change the cleanup policy or the uninstall mode by editing the value of the annotation by using the following commands:
oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
$ oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
$ oc annotate storagecluster -n openshift-storage ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
Prerequisites
- Ensure that the OpenShift Container Storage cluster is in a healthy state. The uninstall process can fail when some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in an unhealthy state, contact Red Hat Customer Support before uninstalling OpenShift Container Storage.
- Ensure that applications are not consuming persistent volume claims (PVCs) or object bucket claims (OBCs) using the storage classes provided by OpenShift Container Storage.
- If any custom resources (such as custom storage classes, cephblockpools) were created by the admin, they must be deleted by the admin after removing the resources which consumed them.
Procedure
Delete the volume snapshots that are using OpenShift Container Storage.
List the volume snapshots from all the namespaces.
oc get volumesnapshot --all-namespaces
$ oc get volumesnapshot --all-namespaces
Copy to Clipboard Copied! Toggle word wrap Toggle overflow From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.
oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
$ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete PVCs and OBCs that are using OpenShift Container Storage.
In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.
If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to
forced
and skip this step. Doing this results in orphan PVCs and OBCs in the system.Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.
For more information, see Section 5.2, “Removing monitoring stack from OpenShift Container Storage”.
Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.
For more information, see Section 5.3, “Removing OpenShift Container Platform registry from OpenShift Container Storage”.
Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.
For more information, see Section 5.4, “Removing the cluster logging operator from OpenShift Container Storage”.
Delete other PVCs and OBCs provisioned using OpenShift Container Storage.
Following script is sample script to identify the PVCs and OBCs provisioned using OpenShift Container Storage. The script ignores the PVCs that are used internally by Openshift Container Storage.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteOmit
RGW_PROVISIONER
for cloud platforms.Delete the OBCs.
oc delete obc <obc name> -n <project name>
$ oc delete obc <obc name> -n <project name>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the PVCs.
oc delete pvc <pvc name> -n <project-name>
$ oc delete pvc <pvc name> -n <project-name>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteEnsure that you have removed any custom backing stores, bucket classes, etc., created in the cluster.
Delete the Storage Cluster object and wait for the removal of the associated resources.
oc delete -n openshift-storage storagecluster --all --wait=true
$ oc delete -n openshift-storage storagecluster --all --wait=true
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check for cleanup pods if the
uninstall.ocs.openshift.io/cleanup-policy
was set todelete
(default) and ensure that their status isCompleted
.oc get pods -n openshift-storage | grep -i cleanup
$ oc get pods -n openshift-storage | grep -i cleanup NAME READY STATUS RESTARTS AGE cluster-cleanup-job-<xx> 0/1 Completed 0 8m35s cluster-cleanup-job-<yy> 0/1 Completed 0 8m35s cluster-cleanup-job-<zz> 0/1 Completed 0 8m35s
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm that the directory
/var/lib/rook
is now empty. This directory will be empty only if theuninstall.ocs.openshift.io/cleanup-policy
annotation was set todelete
(default).for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; done
$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If encryption was enabled at the time of install, remove
dm-crypt
manageddevice-mapper
mapping from OSD devices on all the OpenShift Container Storage nodes.Create a
debug
pod andchroot
to the host on the storage node.oc debug node/<node name> chroot /host
$ oc debug node/<node name> $ chroot /host
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Get Device names and make note of the OpenShift Container Storage devices.
dmsetup ls
$ dmsetup ls ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the mapped device.
cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
$ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf the above command gets stuck due to insufficient privileges, run the following commands:
-
Press
CTRL+Z
to exit the above command. Find PID of the process which was stuck.
ps -ef | grep crypt
$ ps -ef | grep crypt
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Terminate the process using
kill
command.kill -9 <PID>
$ kill -9 <PID>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the device name is removed.
dmsetup ls
$ dmsetup ls
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Press
Delete the namespace and wait till the deletion is complete. You need to switch to another project if
openshift-storage
is the active project.For example:
oc project default oc delete project openshift-storage --wait=true --timeout=5m
$ oc project default $ oc delete project openshift-storage --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The project is deleted if the following command returns a NotFound error.
oc get project openshift-storage
$ oc get project openshift-storage
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteWhile uninstalling OpenShift Container Storage, if
namespace
is not deleted completely and remains inTerminating
state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.- Delete the local storage operator configurations if you have deployed OpenShift Container Storage using local storage devices. See Removing local storage operator configurations.
Unlabel the storage nodes.
oc label nodes --all cluster.ocs.openshift.io/openshift-storage- oc label nodes --all topology.rook.io/rack-
$ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- $ oc label nodes --all topology.rook.io/rack-
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the OpenShift Container Storage taint if the nodes were tainted.
oc adm taint nodes --all node.ocs.openshift.io/storage-
$ oc adm taint nodes --all node.ocs.openshift.io/storage-
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the
Released
state, delete it.oc get pv oc delete pv <pv name>
$ oc get pv $ oc delete pv <pv name>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the Multicloud Object Gateway storageclass.
oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m
$ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove
CustomResourceDefinitions
.oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m
$ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: To ensure that the vault keys are deleted permanently you need to manually delete the metadata associated with the vault key.
NoteExecute this step only if Vault Key/Value (KV) secret engine API, version 2 is used for cluster-wide encryption with Key Management System (KMS) since the vault keys are marked as deleted and not permanently deleted during the uninstallation of OpenShift Container Storage. You can always restore it later if required.
List the keys in the vault.
vault kv list <backend_path>
$ vault kv list <backend_path>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow <backend_path>
Is the path in the vault where the encryption keys are stored.
For example:
vault kv list kv-v2
$ vault kv list kv-v2
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
List the metadata associated with the vault key.
vault kv get kv-v2/<key>
$ vault kv get kv-v2/<key>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For the Multicloud Object Gateway (MCG) key:
vault kv get kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
$ vault kv get kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow <key>
Is the encryption key.
For Example:
vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
$ vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete the metadata.
vault kv metadata delete kv-v2/<key>
$ vault kv metadata delete kv-v2/<key>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For the MCG key:
vault kv metadata delete kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
$ vault kv metadata delete kv-v2/NOOBAA_ROOT_SECRET_PATH/<key>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow <key>
Is the encryption key.
For Example:
vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
$ vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- Repeat these steps to delete the metadata associated with all the vault keys.
To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,
- Click Storage.
- Verify that Overview no longer appears under Storage.
5.1.1. Removing local storage operator configurations Copier lienLien copié sur presse-papiers!
Use the instructions in this section only if you have deployed OpenShift Container Storage using local storage devices.
For OpenShift Container Storage deployments using only localvolume
resources, refer step 8.
Procedure
-
Identify the
LocalVolumeSet
and the correspondingStorageClassName
being used by OpenShift Container Storage. Set the variable SC to the
StorageClass
providing theLocalVolumeSet
.export SC="<StorageClassName>"
$ export SC="<StorageClassName>"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
LocalVolumeSet
.oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage
$ oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local storage PVs for the given
StorageClassName
.oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv
$ oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
StorageClassName
.oc delete sc $SC
$ oc delete sc $SC
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the symlinks created by the
LocalVolumeSet
.[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete
LocalVolumeDiscovery
.oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage
$ oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Removing
LocalVolume
resources (if any).Use the following steps to remove the
LocalVolume
resources used to provision PVs in the current or previous OpenShift Container Storage version. Also, ensure that these resources are not being used by other tenants in the cluster.For each of the local volumes, perform the following:
-
Identify the
LocalVolume
and the correspondingStorageClassName
being used by OpenShift Container Storage. Set the variable LV to the name of the LocalVolume and variable SC to the name of the StorageClass
For example:
LV=local-block SC=localblock
$ LV=local-block $ SC=localblock
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local volume resource.
oc delete localvolume -n local-storage --wait=true $LV
$ oc delete localvolume -n local-storage --wait=true $LV
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the remaining PVs and StorageClasses if they exist.
oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m oc delete storageclass $SC --wait --timeout=5m
$ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m $ oc delete storageclass $SC --wait --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Clean up the artifacts from the storage nodes for that resource.
[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
$ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Identify the
5.2. Removing monitoring stack from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
Use this section to clean up the monitoring stack from OpenShift Container Storage.
The PVCs that are created as a part of configuring the monitoring stack are in the openshift-monitoring
namespace.
Prerequisites
PVCs are configured to use OpenShift Container Platform monitoring stack.
For information, see configuring monitoring stack.
Procedure
List the pods and PVCs that are currently running in the
openshift-monitoring
namespace.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the monitoring
configmap
.oc -n openshift-monitoring edit configmap cluster-monitoring-config
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove any
config
sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.Before editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
alertmanagerMain
andprometheusK8s
monitoring components are using the OpenShift Container Storage PVCs.Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.
oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
$ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.3. Removing OpenShift Container Platform registry from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
Use this section to clean up OpenShift Container Platform registry from OpenShift Container Storage. If you want to configure an alternative storage, see image registry
The PVCs that are created as a part of configuring OpenShift Container Platform registry are in the openshift-image-registry
namespace.
Prerequisites
- The image registry should have been configured to use an OpenShift Container Storage PVC.
Procedure
Edit the
configs.imageregistry.operator.openshift.io
object and remove the content in the storage section.oc edit configs.imageregistry.operator.openshift.io
$ oc edit configs.imageregistry.operator.openshift.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the PVC is called
registry-cephfs-rwx-pvc
, which is now safe to delete.Delete the PVC.
oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.4. Removing the cluster logging operator from OpenShift Container Storage Copier lienLien copié sur presse-papiers!
To clean the cluster logging operator from the OpenShift Container Storage, follow the steps in the procedure.
The PVCs created as a part of configuring cluster logging operator are in the openshift-logging
namespace.
Prerequisites
- The cluster logging instance must be configured to use OpenShift Container Storage PVCs.
Procedure
Remove the
ClusterLogging
instance in the namespace.oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
$ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The PVCs in the
openshift-logging
namespace are now safe to delete.Delete PVCs.
oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow