Red Hat Summit Red Hat SummitSuporteConsoleDesenvolvedoresComeçar um testeContato

Este conteúdo não está disponível no idioma selecionado.

Chapter 4. Uninstalling OpenShift Container Storage

4.1. Uninstalling OpenShift Container Storage in Internal-attached devices mode
Copiar o link

Use the steps in this section to uninstall OpenShift Container Storage.

Uninstall Annotations

Annotations on the Storage Cluster are used to change the behavior of the uninstall process. To define the uninstall behavior, the following two annotations have been introduced in the storage cluster:

  • uninstall.ocs.openshift.io/cleanup-policy: delete
  • uninstall.ocs.openshift.io/mode: graceful

The below table provides information on the different values that can used with these annotations:

Expand
Table 4.1. uninstall.ocs.openshift.io uninstall annotations descriptions
AnnotationValueDefaultBehavior

cleanup-policy

delete

Yes

Rook cleans up the physical drives and the DataDirHostPath

cleanup-policy

retain

No

Rook does not clean up the physical drives and the DataDirHostPath

mode

graceful

Yes

Rook and NooBaa pauses the uninstall process until the PVCs and the OBCs are removed by the administrator/user

mode

forced

No

Rook and NooBaa proceeds with uninstall even if PVCs/OBCs provisioned using Rook and NooBaa exist respectively.

You can change the cleanup policy or the uninstall mode by editing the value of the annotation by using the following commands: 

$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
Copy to Clipboard Toggle word wrap 
$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
Copy to Clipboard Toggle word wrap

Prerequisites

  • Ensure that the OpenShift Container Storage cluster is in a healthy state. The uninstall process can fail when some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in an unhealthy state, contact Red Hat Customer Support before uninstalling OpenShift Container Storage.
  • Ensure that applications are not consuming persistent volume claims (PVCs) or object bucket claims (OBCs) using the storage classes provided by OpenShift Container Storage.
  • If any custom resources (such as custom storage classes, cephblockpools) were created by the admin, they must be deleted by the admin after removing the resources which consumed them.

Procedure

  1. Delete the volume snapshots that are using OpenShift Container Storage.

    1. List the volume snapshots from all the namespaces. 

      $ oc get volumesnapshot --all-namespaces
      Copy to Clipboard Toggle word wrap

    2. From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage. 

      $ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
      Copy to Clipboard Toggle word wrap

  2. Delete PVCs and OBCs that are using OpenShift Container Storage.

    In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.

    If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.

    1. Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.

      See Removing monitoring stack from OpenShift Container Storage

    2. Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.

      Removing OpenShift Container Platform registry from OpenShift Container Storage

    3. Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.

      Removing the cluster logging operator from OpenShift Container Storage

  3. Delete the Storage Cluster object and wait for the removal of the associated resources. 

    $ oc delete -n openshift-storage storagecluster --all --wait=true
    Copy to Clipboard Toggle word wrap

  4. Check for cleanup pods if the uninstall.ocs.openshift.io/cleanup-policy was set to delete(default) and ensure that their status is Completed. 

    $ oc get pods -n openshift-storage | grep -i cleanup
NAME                                READY   STATUS      RESTARTS   AGE
cluster-cleanup-job-<xx>        	0/1     Completed   0          8m35s
cluster-cleanup-job-<yy>     		0/1     Completed   0          8m35s
cluster-cleanup-job-<zz>     		0/1     Completed   0          8m35s
    Copy to Clipboard Toggle word wrap

  5. Confirm that the directory /var/lib/rook is now empty. This directory will be empty only if the uninstall.ocs.openshift.io/cleanup-policy annotation was set to delete(default). 

    $ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done
    Copy to Clipboard Toggle word wrap

  6. If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from OSD devices on all the OpenShift Container Storage nodes.

    1. Create a debug pod and chroot to the host on the storage node. 

      $ oc debug node/<node name>
$ chroot /host
      Copy to Clipboard Toggle word wrap

    2. Get Device names and make note of the OpenShift Container Storage devices. 

      $ dmsetup ls
ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)
      Copy to Clipboard Toggle word wrap

    3. Remove the mapped device. 

      $ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
      Copy to Clipboard Toggle word wrap
      Note

      If the above command gets stuck due to insufficient privileges, run the following commands:

      • Press CTRL+Z to exit the above command.

      • Find PID of the process which was stuck. 

        $ ps -ef | grep crypt
        Copy to Clipboard Toggle word wrap

      • Terminate the process using kill command. 

        $ kill -9 <PID>
        Copy to Clipboard Toggle word wrap

      • Verify that the device name is removed. 

        $ dmsetup ls
        Copy to Clipboard Toggle word wrap

  7. Delete the namespace and wait till the deletion is complete. You will need to switch to another project if openshift-storage is the active project.

    For example: 

    $ oc project default
$ oc delete project openshift-storage
    Copy to Clipboard Toggle word wrap

    The project is deleted if the following command returns a NotFound error. 

    $ oc get project openshift-storage
    Copy to Clipboard Toggle word wrap
    Note

    While uninstalling OpenShift Container Storage, if namespace is not deleted completely and remains in Terminating state, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.

  8. Delete local storage operator configurations if you have deployed OpenShift Container Storage using local storage devices. See Removing local storage operator configurations.

  9. Unlabel the storage nodes. 

    $ oc label nodes  --all cluster.ocs.openshift.io/openshift-storage-
$ oc label nodes  --all topology.rook.io/rack-
    Copy to Clipboard Toggle word wrap

  10. Remove the OpenShift Container Storage taint if the nodes were tainted. 

    $ oc adm taint nodes --all node.ocs.openshift.io/storage-
    Copy to Clipboard Toggle word wrap

  11. Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the Released state, delete it. 

    $ oc get pv
$ oc delete pv <pv name>
    Copy to Clipboard Toggle word wrap

  12. Delete the Multicloud Object Gateway storageclass. 

    $ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

  13. Remove CustomResourceDefinitions. 

    $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

  14. To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,

    1. Click Storage.
    2. Verify that Overview no longer appears under Storage.

4.1.1. Removing local storage operator configurations
Copiar o link

Use the instructions in this section only if you have deployed OpenShift Container Storage using local storage devices.

Note

For OpenShift Container Storage deployments only using localvolume resources, go directly to step 8.

Procedure

  1. Identify the LocalVolumeSet and the corresponding StorageClassName being used by OpenShift Container Storage. 

    $ oc get localvolumesets.local.storage.openshift.io -n openshift-local-storage
    Copy to Clipboard Toggle word wrap

  2. Set the variable SC to the StorageClass providing the LocalVolumeSet. 

    $ export SC="<StorageClassName>"
    Copy to Clipboard Toggle word wrap

  3. List and note the devices to be cleaned up later. Inorder to list the device ids of the disks, please follow the procedure mentioned here, See Find the available storage devices.

    Example output: 

    /dev/disk/by-id/scsi-360050763808104bc28000000000000eb
/dev/disk/by-id/scsi-360050763808104bc28000000000000ef
/dev/disk/by-id/scsi-360050763808104bc28000000000000f3
    Copy to Clipboard Toggle word wrap

  4. Delete the LocalVolumeSet. 

    $ oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage
    Copy to Clipboard Toggle word wrap

  5. Delete the local storage PVs for the given StorageClassName. 

    $ oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv
    Copy to Clipboard Toggle word wrap

  6. Delete the StorageClassName. 

    $ oc delete sc $SC
    Copy to Clipboard Toggle word wrap

  7. Delete the symlinks created by the LocalVolumeSet. 

    [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
    Copy to Clipboard Toggle word wrap

  8. Delete LocalVolumeDiscovery. 

    $ oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage
    Copy to Clipboard Toggle word wrap

  9. Removing LocalVolume resources (if any).

    Use the following steps to remove the LocalVolume resources that were used to provision PVs in the current or previous OpenShift Container Storage version. Also, ensure that these resources are not being used by other tenants on the cluster.

    For each of the local volumes, do the following:

    1. Identify the LocalVolume and the corresponding StorageClassName being used by OpenShift Container Storage. 

      $ oc get localvolume.local.storage.openshift.io -n openshift-local-storage
      Copy to Clipboard Toggle word wrap

    2. Set the variable LV to the name of the LocalVolume and variable SC to the name of the StorageClass

      For example: 

      $ LV=local-block
$ SC=localblock
      Copy to Clipboard Toggle word wrap

    3. List and note the devices to be cleaned up later. 

      $ oc get localvolume -n openshift-local-storage $LV -o jsonpath='{ .spec.storageClassDevices[].devicePaths[] }{"\n"}'
      Copy to Clipboard Toggle word wrap

      Example output: 

      /dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
      Copy to Clipboard Toggle word wrap

    4. Delete the local volume resource. 

      $ oc delete localvolume -n openshift-local-storage --wait=true $LV
      Copy to Clipboard Toggle word wrap

    5. Delete the remaining PVs and StorageClasses if they exist. 

      $ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m
$ oc delete storageclass $SC --wait --timeout=5m
      Copy to Clipboard Toggle word wrap

    6. Clean up the artifacts from the storage nodes for that resource. 

      $ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done
      Copy to Clipboard Toggle word wrap

      Example output: 

      Starting pod/node-xxx-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-yyy-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
Starting pod/node-zzz-debug ...
To use host binaries, run `chroot /host`
removed '/mnt/local-storage/localblock/nvme2n1'
removed directory '/mnt/local-storage/localblock'

Removing debug pod ...
      Copy to Clipboard Toggle word wrap

  10. Wipe the disks for each of the local volumesets or local volumes listed in step 1 and 8 respectively so that they can be reused.

    1. List the storage nodes. 

      oc get nodes -l cluster.ocs.openshift.io/openshift-storage=
      Copy to Clipboard Toggle word wrap

      Example output: 

      NAME      STATUS   ROLES    AGE     VERSION
node-xxx  Ready    worker   4h45m  v1.18.3+6c42de8
node-yyy  Ready    worker   4h46m  v1.18.3+6c42de8
node-zzz  Ready    worker   4h45m  v1.18.3+6c42de8
      Copy to Clipboard Toggle word wrap

    2. Obtain the node console and execute chroot /host command when the prompt appears. 

      $ oc debug node/node-xxx
Starting pod/node-xxx-debug …
To use host binaries, run `chroot /host`
Pod IP: w.x.y.z
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
      Copy to Clipboard Toggle word wrap

    3. Store the disk paths in the DISKS variable within quotes. For the list of disk paths, see step 3 and step 8.c for local volumeset and local volume respectively.

      Example output: 

      sh-4.4# DISKS="/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3 "
or
sh-4.2# DISKS="/dev/sdb /dev/sdc /dev/sdd /dev/sde ".
      Copy to Clipboard Toggle word wrap

    4. Run sgdisk --zap-all on all the disks. 

      sh-4.4# for disk in $DISKS; do sgdisk --zap-all $disk;done
      Copy to Clipboard Toggle word wrap

      Example output: 

      Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
      Copy to Clipboard Toggle word wrap

    5. Exit the shell and repeat for the other nodes. 

      sh-4.4# exit
exit
sh-4.2# exit
exit
Removing debug pod ...
      Copy to Clipboard Toggle word wrap

  11. Delete the openshift-local-storage namespace and wait till the deletion is complete. You will need to switch to another project if the openshift-local-storage namespace is the active project.

    For example: 

    $ oc project default
$ oc delete project openshift-local-storage --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

    The project is deleted if the following command returns a NotFound error. 

    $ oc get project openshift-local-storage
    Copy to Clipboard Toggle word wrap

4.2. Removing monitoring stack from OpenShift Container Storage
Copiar o link

Use this section to clean up the monitoring stack from OpenShift Container Storage.

The PVCs that are created as a part of configuring the monitoring stack are in the openshift-monitoring namespace.

Prerequisites

Procedure

  1. List the pods and PVCs that are currently running in the openshift-monitoring namespace. 

    $ oc get pod,pvc -n openshift-monitoring
NAME                           READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0         3/3     Running   0          8d
pod/alertmanager-main-1         3/3     Running   0          8d
pod/alertmanager-main-2         3/3     Running   0          8d
pod/cluster-monitoring-
operator-84457656d-pkrxm        1/1     Running   0          8d
pod/grafana-79ccf6689f-2ll28    2/2     Running   0          8d
pod/kube-state-metrics-
7d86fb966-rvd9w                 3/3     Running   0          8d
pod/node-exporter-25894         2/2     Running   0          8d
pod/node-exporter-4dsd7         2/2     Running   0          8d
pod/node-exporter-6p4zc         2/2     Running   0          8d
pod/node-exporter-jbjvg         2/2     Running   0          8d
pod/node-exporter-jj4t5         2/2     Running   0          6d18h
pod/node-exporter-k856s         2/2     Running   0          6d18h
pod/node-exporter-rf8gn         2/2     Running   0          8d
pod/node-exporter-rmb5m         2/2     Running   0          6d18h
pod/node-exporter-zj7kx         2/2     Running   0          8d
pod/openshift-state-metrics-
59dbd4f654-4clng                3/3     Running   0          8d
pod/prometheus-adapter-
5df5865596-k8dzn                1/1     Running   0          7d23h
pod/prometheus-adapter-
5df5865596-n2gj9                1/1     Running   0          7d23h
pod/prometheus-k8s-0            6/6     Running   1          8d
pod/prometheus-k8s-1            6/6     Running   1          8d
pod/prometheus-operator-
55cfb858c9-c4zd9                1/1     Running   0          6d21h
pod/telemeter-client-
78fc8fc97d-2rgfp                3/3     Running   0          8d

NAME                                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-0   Bound    pvc-0d519c4f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-1   Bound    pvc-0d5a9825-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-alertmanager-claim-alertmanager-main-2   Bound    pvc-0d6413dc-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-0        Bound    pvc-0b7c19b0-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
persistentvolumeclaim/my-prometheus-claim-prometheus-k8s-1        Bound    pvc-0b8aed3f-15a5-11ea-baa0-026d231574aa   40Gi       RWO            ocs-storagecluster-ceph-rbd   8d
    Copy to Clipboard Toggle word wrap

  2. Edit the monitoring configmap. 

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
    Copy to Clipboard Toggle word wrap

  3. Remove any config sections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.

    Before editing

    Expand
    .
.
.
apiVersion: v1
data:
  config.yaml: |
    alertmanagerMain:
      volumeClaimTemplate:
        metadata:
          name: my-alertmanager-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
    prometheusK8s:
      volumeClaimTemplate:
        metadata:
          name: my-prometheus-claim
        spec:
          resources:
            requests:
              storage: 40Gi
          storageClassName: ocs-storagecluster-ceph-rbd
kind: ConfigMap
metadata:
  creationTimestamp: "2019-12-02T07:47:29Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "22110"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: fd6d988b-14d7-11ea-84ff-066035b9efa8
.
.
.
    Copy to Clipboard Toggle word wrap

    After editing

    Expand
    .
.
.
apiVersion: v1
data:
  config.yaml: |
kind: ConfigMap
metadata:
  creationTimestamp: "2019-11-21T13:07:05Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "404352"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: d12c796a-0c5f-11ea-9832-063cd735b81c
.
.
.
    Copy to Clipboard Toggle word wrap

    In this example, alertmanagerMain and prometheusK8s monitoring components are using the OpenShift Container Storage PVCs.

  4. Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes. 

    $ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

4.3. Removing OpenShift Container Platform registry from OpenShift Container Storage
Copiar o link

Use this section to clean up OpenShift Container Platform registry from OpenShift Container Storage. If you want to configure an alternative storage, see image registry

The PVCs that are created as a part of configuring OpenShift Container Platform registry are in the openshift-image-registry namespace.

Prerequisites

  • The image registry should have been configured to use an OpenShift Container Storage PVC.

Procedure

  1. Edit the configs.imageregistry.operator.openshift.io object and remove the content in the storage section. 

    $ oc edit configs.imageregistry.operator.openshift.io
    Copy to Clipboard Toggle word wrap
    Expand

    Before editing 

    .
.
.
managementState: Managed
storage:
  pvc:
    claim: registry-cephfs-rwx-pvc
.
.
.
    Copy to Clipboard Toggle word wrap

    After editing 

    .
.
.
managementState: Removed
storage:
  emptyDir: {}
.
.
.
    Copy to Clipboard Toggle word wrap

    In this example, the PVC is called registry-cephfs-rwx-pvc, which is now safe to delete.

  2. Delete the PVC. 

    $ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

4.4. Removing the cluster logging operator from OpenShift Container Storage
Copiar o link

To clean the cluster logging operator from the OpenShift Container Storage, follow the steps in the procedure.

The PVCs created as a part of configuring cluster logging operator are in the openshift-logging namespace.

Prerequisites

  • The cluster logging instance must be configured to use OpenShift Container Storage PVCs.

Procedure

  1. Remove the ClusterLogging instance in the namespace. 

    $ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap

    The PVCs in the openshift-logging namespace are now safe to delete.

  2. Delete PVCs. 

    $ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
    Copy to Clipboard Toggle word wrap
Voltar ao topo
Red Hat logoGithubredditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar. Explore nossas atualizações recentes.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja o Blog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

Theme

© 2025 Red Hat