OpenShift Container Storage is now OpenShift Data Foundation starting with version 4.9.
Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 4. Uninstalling OpenShift Container Storage
Use the steps in this section to uninstall OpenShift Container Storage instead of the Uninstall option from the user interface.
Prerequisites
- Make sure that the OpenShift Container Storage cluster is in healthy state. The deletion might fail if some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in unhealthy state, you should contact Red Hat Customer Support before uninstalling OpenShift Container Storage.
- Delete any applications that are consuming persistent volume claims (PVCs) or object bucket claims (OBCs) based on the OpenShift Container Storage storage classes and then delete PVCs and OBCs that are using OpenShift Container Storage storage classes.
Procedure
List the storage classes and take a note of the storage classes with the following storage class provisioners:
-
openshift-storage.rbd.csi.ceph.com
-
openshift-storage.cephfs.csi.ceph.com
openshift-storage.noobaa.io/obc
For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Query for PVCs and OBCs that are using the storage class provisioners listed in the previous step.
oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{" Labels: "}{@.metadata.labels}{"\n"}{end}' --all-namespaces|awk '! ( /Namespace: openshift-storage/ && /app:noobaa/ )'
$ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{" Labels: "}{@.metadata.labels}{"\n"}{end}' --all-namespaces|awk '! ( /Namespace: openshift-storage/ && /app:noobaa/ )'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-cephfs")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces
$ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-cephfs")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces
Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get obc -o=jsonpath='{range .items[?(@.spec.storageClassName=="openshift-storage.noobaa.io")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces
$ oc get obc -o=jsonpath='{range .items[?(@.spec.storageClassName=="openshift-storage.noobaa.io")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIgnore any NooBaa PVCs in the
openshift-storage
namespace.Follow these instructions to ensure that the PVCs listed in the previous step are deleted:
- Determine the pod that is consuming the PVC.
Identify the controlling object such as a
Deployment
,StatefulSet
,DeamonSet
,Job
, or a custom controller.Each object has a metadata field known as
OwnerReference
. This is a list of associated objects. TheOwnerReference
with thecontroller
field set totrue
will point to controlling objects such asReplicaSet
,StatefulSet
,DaemonSet
and so on.- Ensure that the object is safe to delete by asking the owner of the project and then delete it.
Delete the PVCs and OBCs.
oc delete pvc <pvc name> -n <project-name> oc delete obc <obc name> -n <project name>
$ oc delete pvc <pvc name> -n <project-name> $ oc delete obc <obc name> -n <project name>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you have created any PVCs as a part of configuring the monitoring stack, cluster logging operator, or prometheus registry, then you must perform the clean up steps provided in the following sections as required:
List and note the backing local volume objects. If no results found, then skip step 8 & 9.
for sc in $(oc get storageclass|grep 'kubernetes.io/no-provisioner' |grep -E $(oc get storagecluster -n openshift-storage -o jsonpath='{ .items[*].spec.storageDeviceSets[*].dataPVCTemplate.spec.storageClassName}' | sed 's/ /|/g')| awk '{ print $1 }'); do echo -n "StorageClass: $sc "; oc get storageclass $sc -o jsonpath=" { 'LocalVolume: ' }{ .metadata.labels['local\.storage\.openshift\.io/owner-name'] } { '\n' }"; done
$ for sc in $(oc get storageclass|grep 'kubernetes.io/no-provisioner' |grep -E $(oc get storagecluster -n openshift-storage -o jsonpath='{ .items[*].spec.storageDeviceSets[*].dataPVCTemplate.spec.storageClassName}' | sed 's/ /|/g')| awk '{ print $1 }'); do echo -n "StorageClass: $sc "; oc get storageclass $sc -o jsonpath=" { 'LocalVolume: ' }{ .metadata.labels['local\.storage\.openshift\.io/owner-name'] } { '\n' }"; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
StorageClass: localblock LocalVolume: local-block
StorageClass: localblock LocalVolume: local-block
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
StorageCluster
object.oc delete -n openshift-storage storagecluster --all --wait=true
$ oc delete -n openshift-storage storagecluster --all --wait=true
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the namespace and wait till the deletion is complete.
oc delete project openshift-storage --wait=true --timeout=5m
$ oc delete project openshift-storage --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteYou will need to switch to another project if openshift-storage was the active project.
For example
oc project default
$ oc project default
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Clean up the storage operator artifacts on each node.
for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /var/lib/rook; done
$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /var/lib/rook; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure you can see removed directory /var/lib/rook in the output.
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local volume created during the deployment and for each of the local volumes listed in step 4.
For each of the local volumes, do the following:
Set the variable
LV
to the name of the LocalVolume and variableSC
to name of the StorageClass.For example
LV=local-block SC=localblock
$ LV=local-block $ SC=localblock
Copy to Clipboard Copied! Toggle word wrap Toggle overflow List and note the devices to be cleaned up later.
oc get localvolume -n local-storage $LV -o jsonpath='{ .spec.storageClassDevices[*].devicePaths[*] }'
$ oc get localvolume -n local-storage $LV -o jsonpath='{ .spec.storageClassDevices[*].devicePaths[*] }'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol078f5cdde09efc165 /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0defc1d5e2dd07f9e /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0c8e82a3beeb7b7e5
/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol078f5cdde09efc165 /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0defc1d5e2dd07f9e /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0c8e82a3beeb7b7e5
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local volume resource.
oc delete localvolume -n local-storage --wait=true $LV
$ oc delete localvolume -n local-storage --wait=true $LV
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the remaining PVs and StorageClasses if they exist.
oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m oc delete storageclass $SC --wait --timeout=5m
$ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m $ oc delete storageclass $SC --wait --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Clean up the artifacts from the storage nodes for that resource.
[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
$ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Wipe the disks for each of the local volumes listed in step 4 so that they can be reused.
List the storage nodes.
oc get nodes -l cluster.ocs.openshift.io/openshift-storage=
$ oc get nodes -l cluster.ocs.openshift.io/openshift-storage=
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME STATUS ROLES AGE VERSION ip-10-0-134-65.us-east-2.compute.internal Ready worker 4h45m v1.17.1 ip-10-0-155-149.us-east-2.compute.internal Ready worker 4h46m v1.17.1 ip-10-0-162-89.us-east-2.compute.internal Ready worker 4h45m v1.17.1
NAME STATUS ROLES AGE VERSION ip-10-0-134-65.us-east-2.compute.internal Ready worker 4h45m v1.17.1 ip-10-0-155-149.us-east-2.compute.internal Ready worker 4h46m v1.17.1 ip-10-0-162-89.us-east-2.compute.internal Ready worker 4h45m v1.17.1
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Obtain the node console and execute
chroot /host
command when the prompt appears.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Store the disk paths gathered in step 8(ii) in the
DISKS
variable within quotes.DISKS="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol078f5cdde09efc165 /dev/disk/by-id/nvme-Amazon_Elasti_Block_Store_vol0defc1d5e2dd07f9e /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0c8e82a3beeb7b7e5"
sh-4.2# DISKS="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol078f5cdde09efc165 /dev/disk/by-id/nvme-Amazon_Elasti_Block_Store_vol0defc1d5e2dd07f9e /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0c8e82a3beeb7b7e5"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run
sgdisk --zap-all
on all the disks:for disk in $DISKS; do sgdisk --zap-all $disk;done
sh-4.4# for disk in $DISKS; do sgdisk --zap-all $disk;done
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIgnore file-not-found warnings as they refer to disks that are on other machines.
Exit the shell and repeat for the other nodes.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete the storage classes with an
openshift-storage
provisioner listed in step 1.oc delete storageclass <storageclass-name> --wait=true --timeout=5m
$ oc delete storageclass <storageclass-name> --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example:
oc delete storageclass ocs-storagecluster-ceph-rbd ocs-storagecluster-cephfs openshift-storage.noobaa.io --wait=true --timeout=5m
$ oc delete storageclass ocs-storagecluster-ceph-rbd ocs-storagecluster-cephfs openshift-storage.noobaa.io --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Unlabel the storage nodes.
oc label nodes --all cluster.ocs.openshift.io/openshift-storage- oc label nodes --all topology.rook.io/rack-
$ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- $ oc label nodes --all topology.rook.io/rack-
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteYou can ignore the warnings displayed for the unlabeled nodes such as
label <label> not found
.Remove
CustomResourceDefinitions
.oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusterinitializations.ocs.openshift.io storageclusters.ocs.openshift.io --wait=true --timeout=5m
$ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusterinitializations.ocs.openshift.io storageclusters.ocs.openshift.io --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteUninstalling OpenShift Container Storage clusters on AWS deletes all the OpenShift Container Storage data stored on the target buckets, however, neither the target buckets created by the user nor the ones that were automatically created during the OpenShift Container Storage installation get deleted and the data that does not belong to OpenShift Container Storage remains on these target buckets.
- To make sure that OpenShift Container Storage is uninstalled, verify that the openshift-storage namespace no longer exists and the storage dashboard no longer appears in the UI.
While uninstalling OpenShift Container Storage, if namespace is not deleted completely and remains in Terminating
state, perform the steps in the article https://access.redhat.com/solutions/3881901 to identify objects that are blocking the namespace from being terminated. OpenShift objects such as Cephcluster
, StorageCluster
, NooBaa
, and PVC
that have the finalizers might be the cause for the namespace to be in Terminating
state. If PVC has a finalizer, force delete the associated pod to remove the finalizer.
4.1. Removing monitoring stack from OpenShift Container Storage Link kopierenLink in die Zwischenablage kopiert!
Use this section to clean up monitoring stack from OpenShift Container Storage.
The PVCs that are created as a part of configuring the monitoring stack are in the openshift-monitoring
namespace.
Prerequisites
PVCs are configured to use OpenShift Container Platform monitoring stack.
For information, see configuring monitoring stack.
Procedure
List the pods and PVCs that are currently running in the
openshift-monitoring
namespace.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the monitoring
configmap
.oc -n openshift-monitoring edit configmap cluster-monitoring-config
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove any
config
sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
alertmanagerMain
andprometheusK8s
monitoring components are using the OpenShift Container Storage PVCs.List the pods consuming the PVC.
In this example, the
alertmanagerMain
andprometheusK8s
pods that were consuming the PVCs are in theTerminating
state. You can delete the PVCs once these pods are no longer using OpenShift Container Storage PVC.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.
oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
$ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.2. Removing OpenShift Container Platform registry from OpenShift Container Storage Link kopierenLink in die Zwischenablage kopiert!
Use this section to clean up OpenShift Container Platform registry from OpenShift Container Storage. If you want to configure an alternative storage, see: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.4/html-single/registry/architecture-component-imageregistry
The PVCs that are created as a part of configuring OpenShift Container Platform registry are in the openshift-image-registry
namespace.
Prerequisites
- The image registry should have been configured to use an OpenShift Container Storage PVC.
Procedure
Edit the
configs.imageregistry.operator.openshift.io
object and remove the content in the storage section.oc edit configs.imageregistry.operator.openshift.io
$ oc edit configs.imageregistry.operator.openshift.io
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For AWS:
Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
. . storage: . .
. . storage: . .
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the PVC is called
registry-cephfs-rwx-pvc
, which is now safe to delete.For VMware and baremetal:
Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the PVC is called
registry-cephfs-rwx-pvc
, which is now safe to delete.
Delete the PVC.
oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.3. Removing the cluster logging operator from OpenShift Container Storage Link kopierenLink in die Zwischenablage kopiert!
Use this section to clean up the cluster logging operator from OpenShift Container Storage.
The PVCs that are created as a part of configuring cluster logging operator are in openshift-logging
namespace.
Prerequisites
- The cluster logging instance should have been configured to use OpenShift Container Storage PVCs.
Procedure
Remove the
ClusterLogging
instance in the namespace.oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
$ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The PVCs in the
openshift-logging
namespace are now safe to delete.Delete PVCs.
oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
Copy to Clipboard Copied! Toggle word wrap Toggle overflow