OpenShift Container Storage is now OpenShift Data Foundation starting with version 4.9.
Deploying OpenShift Container Storage using IBM Z infrastructure
How to install and set up your IBM Z environment
Abstract
Making open source more inclusive Copy linkLink copied to clipboard!
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation Copy linkLink copied to clipboard!
We appreciate your input on our documentation. Do let us know how we can make it better. To give feedback:
For simple comments on specific passages:
- Make sure you are viewing the documentation in the Multi-page HTML format. In addition, ensure you see the Feedback button in the upper right corner of the document.
- Use your mouse cursor to highlight the part of text that you want to comment on.
- Click the Add Feedback pop-up that appears below the highlighted text.
- Follow the displayed instructions.
For submitting more complex feedback, create a Bugzilla ticket:
- Go to the Bugzilla website.
- As the Component, use Documentation.
- Fill in the Description field with your suggestion for improvement. Include a link to the relevant part(s) of documentation.
- Click Submit Bug.
Preface Copy linkLink copied to clipboard!
Red Hat OpenShift Container Storage 4.8 supports deployment on existing Red Hat OpenShift Container Platform (RHOCP) IBM Z clusters in connected or disconnected environments along with out-of-the-box support for proxy environments.
Only internal Openshift Container Storage clusters are supported on IBM Z. See Planning your deployment and Preparing to deploy OpenShift Container Storage for more information about deployment requirements.
To deploy OpenShift Container Storage, follow the appropriate deployment process for your environment:
Internal Attached Devices mode
Chapter 1. Preparing to deploy OpenShift Container Storage Copy linkLink copied to clipboard!
When you deploy OpenShift Container Storage on OpenShift Container Platform using local storage devices, you can create internal cluster resources. This approach internally provisions base services. Then all applications can access additional storage classes.
Before you begin the deployment of Red Hat OpenShift Container Storage using local storage, ensure that your resource requirements are met. See requirements for installing OpenShift Container Storage using local storage devices.
After you have addressed the above, follow these steps in the order given:
1.1. Requirements for installing OpenShift Container Storage using local storage devices Copy linkLink copied to clipboard!
Node requirements
The cluster must consist of at least three OpenShift Container Platform worker nodes with locally attached-storage devices on each of them.
- Each of the three selected nodes must have at least one raw block device available to be used by OpenShift Container Storage.
- The devices you use must be empty; the disks must not include physical volumes (PVs), volume groups (VGs), or logical volumes (LVs) remaining on the disk.
For more information, see the Resource requirements section in the Planning guide.
- For storage nodes, FCP storage devices are required.
- Multicloud Object Gateway is not supported.
Minimum starting node requirements [Technology Preview]
An OpenShift Container Storage cluster is deployed with minimum configuration when the standard deployment resource requirement is not met.
For more information, see Resource requirements section in the Planning guide.
Chapter 2. Deploy OpenShift Container Storage using local storage devices Copy linkLink copied to clipboard!
Deploying OpenShift Container Storage on OpenShift Container Platform using local storage devices provides you with the option to create internal cluster resources. Follow this deployment method to use local storage to back persistent volumes for your OpenShift Container Platform applications.
Use this section to deploy OpenShift Container Storage on IBM Z infrastructure where OpenShift Container Platform is already installed.
2.1. Installing Red Hat OpenShift Container Storage Operator Copy linkLink copied to clipboard!
You can install Red Hat OpenShift Container Storage Operator using the Red Hat OpenShift Container Platform Operator Hub.
Prerequisites
- Access to an OpenShift Container Platform cluster using an account with cluster-admin and operator installation permissions.
- You have at least three worker nodes in the Red Hat OpenShift Container Platform cluster.
- You have satisfied any additional requirements required. For more information, see Planning your deployment.
When you need to override the cluster-wide default node selector for OpenShift Container Storage, you can use the following command to specify a blank node selector for the
openshift-storagenamespace (create openshift-storage namespace in this case):oc annotate namespace openshift-storage openshift.io/node-selector=
$ oc annotate namespace openshift-storage openshift.io/node-selector=Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Taint a node as
infrato ensure only Red Hat OpenShift Container Storage resources are scheduled on that node. This helps you save on subscription costs. For more information, see How to use dedicated worker nodes for Red Hat OpenShift Container Storage chapter in Managing and Allocating Storage Resources guide.
Procedure
- Log in to OpenShift Web Console.
- Click Operators → OperatorHub.
- Search for OpenShift Container Storage from the list of operators and click on it.
- Click Install.
Set the following options on the Install Operator page:
- Channel as stable-4.8.
- Installation Mode as A specific namespace on the cluster.
-
Installed Namespace as Operator recommended namespace openshift-storage. If Namespace
openshift-storagedoes not exist, it will be created during the operator installation. - Approval Strategy as Automatic or Manual.
Click Install.
If you select Automatic updates, the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your operator without any intervention.
If you select Manual updates, the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to have the operator updated to the new version.
Verification step
- Verify that the OpenShift Container Storage Operator shows a green tick indicating successful installation.
2.2. Installing Local Storage Operator Copy linkLink copied to clipboard!
You can install the Local Storage Operator using the Red Hat OpenShift Container Platform Operator Hub.
Prerequisite
- Access to an OpenShift Container Platform cluster using an account with cluster-admin and Operator installation permissions.
Procedure
- Log in to the OpenShift Web Console.
- Click Operators → OperatorHub.
-
Type
local storagein the Filter by keyword… box to search forLocal Storageoperator from the list of operators and click on it. - Click Install.
Set the following options on the Install Operator page:
- Channel as stable-4.8.
- Installation Mode as A specific namespace on the cluster.
- Installed Namespace as Operator recommended namespace openshift-local-storage.
- Approval Strategy as Automatic.
- Click Install.
Verification step
-
Verify that the Local Storage Operator shows the Status as
Succeeded.
2.3. Finding available storage devices (optional) Copy linkLink copied to clipboard!
This step is additional information and can be skipped as the disks are automatically discovered during storage cluster creation. Use this procedure to identify the device names for each of the three or more worker nodes that you have labeled with the OpenShift Container Storage label cluster.ocs.openshift.io/openshift-storage='' before creating Persistent Volumes (PV) for IBM Z.
Procedure
List and verify the name of the worker nodes with the OpenShift Container Storage label.
oc get nodes -l=cluster.ocs.openshift.io/openshift-storage=
$ oc get nodes -l=cluster.ocs.openshift.io/openshift-storage=Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
NAME STATUS ROLES AGE VERSION bmworker01 Ready worker 6h45m v1.16.2 bmworker02 Ready worker 6h45m v1.16.2 bmworker03 Ready worker 6h45m v1.16.2
NAME STATUS ROLES AGE VERSION bmworker01 Ready worker 6h45m v1.16.2 bmworker02 Ready worker 6h45m v1.16.2 bmworker03 Ready worker 6h45m v1.16.2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Log in to each worker node that is used for OpenShift Container Storage resources and find the unique
by-iddevice name for each available raw block device.oc debug node/<node name>
$ oc debug node/<node name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, for
bmworker01, the available local device issdb.Identify the unique ID for each of the devices selected in Step 2.
sh-4.4#ls -l /dev/disk/by-id/ | grep sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-360050763808104bc2800000000000259 -> ../../sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-SIBM_2145_00e020412f0aXX00 -> ../../sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-0x60050763808104bc2800000000000259 -> ../../sdb
sh-4.4#ls -l /dev/disk/by-id/ | grep sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-360050763808104bc2800000000000259 -> ../../sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-SIBM_2145_00e020412f0aXX00 -> ../../sdb lrwxrwxrwx. 1 root root 9 Feb 3 16:49 scsi-0x60050763808104bc2800000000000259 -> ../../sdbCopy to Clipboard Copied! Toggle word wrap Toggle overflow In the above example, the ID for the local device
sdbscsi-0x60050763808104bc2800000000000259
scsi-0x60050763808104bc2800000000000259Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Repeat the above step to identify the device ID for all the other nodes that have the storage devices to be used by OpenShift Container Storage. See this Knowledge Base article for more details.
2.4. Creating OpenShift Container Storage cluster on IBM Z Copy linkLink copied to clipboard!
Use this procedure to create storage cluster on IBM Z.
Prerequisites
- Ensure that all the requirements in the Requirements for installing OpenShift Container Storage using local storage devices section are met.
- You must have three worker nodes with the same storage type and size attached to each node (for example, 200 GB) to use local storage devices on IBM Z or LinuxONE.
Procedure
- Log into the OpenShift Web Console.
Click Operators → Installed Operators to view all the installed operators.
Ensure that the Project selected is openshift-storage.
- Click OpenShift Container Storage.
- Click Create Instance link of Storage Cluster.
- Select Internal-Attached devices for the Select Mode. By default, Internal is selected.
Create a storage cluster using the wizard that includes disk discovery, storage class creation, and storage cluster creation.
You are prompted to install the Local Storage Operator if it is not already installed. Click Install and install the operator as described in Installing Local Storage Operator.
- Discover disks
You can discover a list of potentially usable disks on the selected nodes. Block disks and partitions that are not in use and available for provisioning persistent volumes (PVs) are discovered.
Choose one of the following:
- All nodes to discover disks in all the nodes.
Select nodes to discover disks from a subset of the listed nodes.
To find specific worker nodes in the cluster, you can filter nodes on the basis of Name or Label. Name allows you to search by name of the node and Label allows you to search by selecting the predefined label.
If the nodes selected do not match the OpenShift Container Storage cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster will be deployed. For minimum starting node requirements, see Resource requirements section in Planning guide.
NoteIf the nodes to be selected are tainted and not discovered in the wizard, follow the steps provided in the Red Hat Knowledgebase Solution as a workaround to add tolerations for Local Storage Operator resources.
- Click Next.
- Create Storage Class
You can create a dedicated storage class to consume storage by filtering a set of storage volumes.
- Enter the Volume Set Name.
- Enter the Storage Class Name. By default, the volume set name appears for the storage class name.
The nodes selected for disk discovery in the earlier step are displayed in the Filter Disks section. Choose one of the following:
- All nodes to select all the nodes for which you discovered the devices.
Select nodes to select a subset of the nodes for which you discovered the devices.
To find specific worker nodes in the cluster, you can filter nodes on the basis of Name or Label. Name allows you to search by name of the node and Label allows you to search by selecting the predefined label.
It is recommended that the worker nodes are spread across three different physical nodes, racks or failure domains for high availability.
NoteEnsure OpenShift Container Storage rack labels are aligned with physical racks in the datacenter to prevent a double node failure at the failure domain level.
Select the required Disk Type. The following options are available:
All
Selects all types of disks present on the nodes. By default, this option is selected.
SSD/NVME
Selects only SSD NVME type of disks.
HDD
Selects only HDD type of disks.
In the Advanced section, you can set the following:
Volume Mode
Block is selected by default.
Disk Size
Minimum and maximum available size of the device that needs to be included.
NoteYou must set a minimum size of 100GB for the device.
Max Disk Limit
This indicates the maximum number of PVs that can be created on a node. If this field is left empty, then PVs are created for all the available disks on the matching nodes.
(Optional) You can view the selected capacity of the disks on the selected nodes using the Select Capacity chart.
This chart might take a few minutes to reflect the disks that are discovered in the previous step.
You can click on the Nodes and Disks links on the chart to bring up the list of nodes and disks to view more details.
- Click Next.
Click Yes in the message alert to confirm the creation of the storage class.
After the local volume set and storage class are created, it is not possible to go back to the step.
- Create Storage Cluster
Select the required storage class.
You might need to wait a couple of minutes for the storage nodes corresponding to the selected storage class to get populated. The nodes corresponding to the storage class are displayed based on the storage class that you selected from the drop down list.
- Click Next.
- (Optional) In the Encryption section, set the toggle to Enabled to enable data encryption on the cluster. Security configuration:
- Click Save.
- Click Next to review your storage cluster.
Click Create.
The Create button is enabled only when a minimum of three nodes are selected. A new storage cluster with the number of volumes equal to the total number of devices attached to each node is created. The default configuration uses a replication factor of 1.
To expand the capacity of the initial cluster, see Scaling Storage guide.
Verification steps
See Verifying your OpenShift Container Storage installation.
Chapter 3. Verifying OpenShift Container Storage deployment for Internal-attached devices mode Copy linkLink copied to clipboard!
Use this section to verify that OpenShift Container Storage is deployed correctly.
3.1. Verifying the state of the pods Copy linkLink copied to clipboard!
To verify that the pods of OpenShift Containers Storage are in running state, follow the below procedure:
Procedure
- Log in to OpenShift Web Console.
- Click Workloads → Pods from the left pane of the OpenShift Web Console.
Select openshift-storage from the Project drop down list.
For more information on the expected number of pods for each component and how it varies depending on the number of nodes, see Table 3.1, “Pods corresponding to OpenShift Container storage cluster”.
Click on the Running and Completed tabs to verify that the pods are running and in a completed state:
Expand Table 3.1. Pods corresponding to OpenShift Container storage cluster Component Corresponding pods OpenShift Container Storage Operator
-
ocs-operator-*(1 pod on any worker node) -
ocs-metrics-exporter-*
(1 pod on any worker node)
Rook-ceph Operator
rook-ceph-operator-*(1 pod on any worker node)
Multicloud Object Gateway
-
noobaa-operator-*(1 pod on any worker node) -
noobaa-core-*(1 pod on any storage node) -
nooba-db-*(1 pod on any storage node) -
noobaa-endpoint-*(1 pod on any storage node)
MON
rook-ceph-mon-*(3 pods distributed across storage nodes)
MGR
rook-ceph-mgr-*(1 pod on any storage node)
MDS
rook-ceph-mds-ocs-storagecluster-cephfilesystem-*(2 pods distributed across storage nodes)
RGW
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-*(1 pod on any storage node)CSI
cephfs-
csi-cephfsplugin-*(1 pod on each worker node) -
csi-cephfsplugin-provisioner-*(2 pods distributed across storage nodes)
-
rbd-
csi-rbdplugin-*(1 pod on each worker node) -
csi-rbdplugin-provisioner-*(2 pods distributed across storage nodes)
-
rook-ceph-crashcollector
rook-ceph-crashcollector-*(1 pod on each storage node)
OSD
-
rook-ceph-osd-*(1 pod for each device) -
rook-ceph-osd-prepare-ocs-deviceset-*(1 pod for each device)
-
3.2. Verifying the OpenShift Container Storage cluster is healthy Copy linkLink copied to clipboard!
To verify that the cluster of OpenShift Container Storage is healthy, follow the steps in the procedure.
Procedure
- Click Storage → Overview and click the Block and File tab.
- In the Status card, verify that Storage Cluster and Data Resiliency has a green tick mark.
- In the Details card, verify that the cluster information is displayed.
For more information on the health of the OpenShift Container Storage clusters using the Block and File dashboard, see Monitoring OpenShift Container Storage.
3.3. Verifying that the OpenShift Container Storage specific storage classes exist Copy linkLink copied to clipboard!
To verify the storage classes exists in the cluster, follow the steps in the procedure.
Procedure
- Click Storage → Storage Classes from the OpenShift Web Console.
Verify that the following storage classes are created with the OpenShift Container Storage cluster creation:
-
ocs-storagecluster-ceph-rbd -
ocs-storagecluster-cephfs -
openshift-storage.noobaa.io -
ocs-storagecluster-ceph-rgw
-
Chapter 4. Uninstalling OpenShift Container Storage Copy linkLink copied to clipboard!
4.1. Uninstalling OpenShift Container Storage in Internal-attached devices mode Copy linkLink copied to clipboard!
Use the steps in this section to uninstall OpenShift Container Storage.
Uninstall Annotations
Annotations on the Storage Cluster are used to change the behavior of the uninstall process. To define the uninstall behavior, the following two annotations have been introduced in the storage cluster:
-
uninstall.ocs.openshift.io/cleanup-policy: delete -
uninstall.ocs.openshift.io/mode: graceful
The below table provides information on the different values that can used with these annotations:
| Annotation | Value | Default | Behavior |
|---|---|---|---|
| cleanup-policy | delete | Yes |
Rook cleans up the physical drives and the |
| cleanup-policy | retain | No |
Rook does not clean up the physical drives and the |
| mode | graceful | Yes | Rook and NooBaa pauses the uninstall process until the PVCs and the OBCs are removed by the administrator/user |
| mode | forced | No | Rook and NooBaa proceeds with uninstall even if PVCs/OBCs provisioned using Rook and NooBaa exist respectively. |
You can change the cleanup policy or the uninstall mode by editing the value of the annotation by using the following commands:
oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/cleanup-policy="retain" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
$ oc -n openshift-storage annotate storagecluster ocs-storagecluster uninstall.ocs.openshift.io/mode="forced" --overwrite
storagecluster.ocs.openshift.io/ocs-storagecluster annotated
Prerequisites
- Ensure that the OpenShift Container Storage cluster is in a healthy state. The uninstall process can fail when some of the pods are not terminated successfully due to insufficient resources or nodes. In case the cluster is in an unhealthy state, contact Red Hat Customer Support before uninstalling OpenShift Container Storage.
- Ensure that applications are not consuming persistent volume claims (PVCs) or object bucket claims (OBCs) using the storage classes provided by OpenShift Container Storage.
- If any custom resources (such as custom storage classes, cephblockpools) were created by the admin, they must be deleted by the admin after removing the resources which consumed them.
Procedure
Delete the volume snapshots that are using OpenShift Container Storage.
List the volume snapshots from all the namespaces.
oc get volumesnapshot --all-namespaces
$ oc get volumesnapshot --all-namespacesCopy to Clipboard Copied! Toggle word wrap Toggle overflow From the output of the previous command, identify and delete the volume snapshots that are using OpenShift Container Storage.
oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>
$ oc delete volumesnapshot <VOLUME-SNAPSHOT-NAME> -n <NAMESPACE>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete PVCs and OBCs that are using OpenShift Container Storage.
In the default uninstall mode (graceful), the uninstaller waits till all the PVCs and OBCs that use OpenShift Container Storage are deleted.
If you wish to delete the Storage Cluster without deleting the PVCs beforehand, you may set the uninstall mode annotation to "forced" and skip this step. Doing so will result in orphan PVCs and OBCs in the system.
Delete OpenShift Container Platform monitoring stack PVCs using OpenShift Container Storage.
See Removing monitoring stack from OpenShift Container Storage
Delete OpenShift Container Platform Registry PVCs using OpenShift Container Storage.
Removing OpenShift Container Platform registry from OpenShift Container Storage
Delete OpenShift Container Platform logging PVCs using OpenShift Container Storage.
Removing the cluster logging operator from OpenShift Container Storage
Delete the Storage Cluster object and wait for the removal of the associated resources.
oc delete -n openshift-storage storagecluster --all --wait=true
$ oc delete -n openshift-storage storagecluster --all --wait=trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check for cleanup pods if the
uninstall.ocs.openshift.io/cleanup-policywas set todelete(default) and ensure that their status isCompleted.oc get pods -n openshift-storage | grep -i cleanup
$ oc get pods -n openshift-storage | grep -i cleanup NAME READY STATUS RESTARTS AGE cluster-cleanup-job-<xx> 0/1 Completed 0 8m35s cluster-cleanup-job-<yy> 0/1 Completed 0 8m35s cluster-cleanup-job-<zz> 0/1 Completed 0 8m35sCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm that the directory
/var/lib/rookis now empty. This directory will be empty only if theuninstall.ocs.openshift.io/cleanup-policyannotation was set todelete(default).for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; done$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host ls -l /var/lib/rook; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow If encryption was enabled at the time of install, remove
dm-cryptmanageddevice-mappermapping from OSD devices on all the OpenShift Container Storage nodes.Create a
debugpod andchrootto the host on the storage node.oc debug node/<node name> chroot /host
$ oc debug node/<node name> $ chroot /hostCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get Device names and make note of the OpenShift Container Storage devices.
dmsetup ls
$ dmsetup ls ocs-deviceset-0-data-0-57snx-block-dmcrypt (253:1)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the mapped device.
cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcrypt
$ cryptsetup luksClose --debug --verbose ocs-deviceset-0-data-0-57snx-block-dmcryptCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf the above command gets stuck due to insufficient privileges, run the following commands:
-
Press
CTRL+Zto exit the above command. Find PID of the process which was stuck.
ps -ef | grep crypt
$ ps -ef | grep cryptCopy to Clipboard Copied! Toggle word wrap Toggle overflow Terminate the process using
killcommand.kill -9 <PID>
$ kill -9 <PID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the device name is removed.
dmsetup ls
$ dmsetup lsCopy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Press
Delete the namespace and wait till the deletion is complete. You will need to switch to another project if
openshift-storageis the active project.For example:
oc project default oc delete project openshift-storage
$ oc project default $ oc delete project openshift-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow The project is deleted if the following command returns a NotFound error.
oc get project openshift-storage
$ oc get project openshift-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteWhile uninstalling OpenShift Container Storage, if
namespaceis not deleted completely and remains inTerminatingstate, perform the steps in Troubleshooting and deleting remaining resources during Uninstall to identify objects that are blocking the namespace from being terminated.- Delete local storage operator configurations if you have deployed OpenShift Container Storage using local storage devices. See Removing local storage operator configurations.
Unlabel the storage nodes.
oc label nodes --all cluster.ocs.openshift.io/openshift-storage- oc label nodes --all topology.rook.io/rack-
$ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- $ oc label nodes --all topology.rook.io/rack-Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the OpenShift Container Storage taint if the nodes were tainted.
oc adm taint nodes --all node.ocs.openshift.io/storage-
$ oc adm taint nodes --all node.ocs.openshift.io/storage-Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm all PVs provisioned using OpenShift Container Storage are deleted. If there is any PV left in the
Releasedstate, delete it.oc get pv oc delete pv <pv name>
$ oc get pv $ oc delete pv <pv name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the Multicloud Object Gateway storageclass.
oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5m
$ oc delete storageclass openshift-storage.noobaa.io --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove
CustomResourceDefinitions.oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5m
$ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusters.ocs.openshift.io cephclients.ceph.rook.io cephobjectrealms.ceph.rook.io cephobjectzonegroups.ceph.rook.io cephobjectzones.ceph.rook.io cephrbdmirrors.ceph.rook.io --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow To ensure that OpenShift Container Storage is uninstalled completely, on the OpenShift Container Platform Web Console,
- Click Storage.
- Verify that Overview no longer appears under Storage.
4.1.1. Removing local storage operator configurations Copy linkLink copied to clipboard!
Use the instructions in this section only if you have deployed OpenShift Container Storage using local storage devices.
For OpenShift Container Storage deployments only using localvolume resources, go directly to step 8.
Procedure
Identify the
LocalVolumeSetand the correspondingStorageClassNamebeing used by OpenShift Container Storage.oc get localvolumesets.local.storage.openshift.io -n openshift-local-storage
$ oc get localvolumesets.local.storage.openshift.io -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set the variable SC to the
StorageClassproviding theLocalVolumeSet.export SC="<StorageClassName>"
$ export SC="<StorageClassName>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow List and note the devices to be cleaned up later. Inorder to list the device ids of the disks, please follow the procedure mentioned here, See Find the available storage devices.
Example output:
/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3
/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
LocalVolumeSet.oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storage
$ oc delete localvolumesets.local.storage.openshift.io <name-of-volumeset> -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local storage PVs for the given
StorageClassName.oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pv$ oc get pv | grep $SC | awk '{print $1}'| xargs oc delete pvCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
StorageClassName.oc delete sc $SC
$ oc delete sc $SCCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the symlinks created by the
LocalVolumeSet.[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete
LocalVolumeDiscovery.oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storage
$ oc delete localvolumediscovery.local.storage.openshift.io/auto-discover-devices -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Removing
LocalVolumeresources (if any).Use the following steps to remove the
LocalVolumeresources that were used to provision PVs in the current or previous OpenShift Container Storage version. Also, ensure that these resources are not being used by other tenants on the cluster.For each of the local volumes, do the following:
Identify the
LocalVolumeand the correspondingStorageClassNamebeing used by OpenShift Container Storage.oc get localvolume.local.storage.openshift.io -n openshift-local-storage
$ oc get localvolume.local.storage.openshift.io -n openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set the variable LV to the name of the LocalVolume and variable SC to the name of the StorageClass
For example:
LV=local-block SC=localblock
$ LV=local-block $ SC=localblockCopy to Clipboard Copied! Toggle word wrap Toggle overflow List and note the devices to be cleaned up later.
oc get localvolume -n openshift-local-storage $LV -o jsonpath='{ .spec.storageClassDevices[].devicePaths[] }{"\n"}'$ oc get localvolume -n openshift-local-storage $LV -o jsonpath='{ .spec.storageClassDevices[].devicePaths[] }{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
/dev/sdb /dev/sdc /dev/sdd /dev/sde
/dev/sdb /dev/sdc /dev/sdd /dev/sdeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the local volume resource.
oc delete localvolume -n openshift-local-storage --wait=true $LV
$ oc delete localvolume -n openshift-local-storage --wait=true $LVCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the remaining PVs and StorageClasses if they exist.
oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m oc delete storageclass $SC --wait --timeout=5m$ oc delete pv -l storage.openshift.com/local-volume-owner-name=${LV} --wait --timeout=5m $ oc delete storageclass $SC --wait --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Clean up the artifacts from the storage nodes for that resource.
[[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; done$ [[ ! -z $SC ]] && for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name }'); do oc debug node/${i} -- chroot /host rm -rfv /mnt/local-storage/${SC}/; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Wipe the disks for each of the local volumesets or local volumes listed in step 1 and 8 respectively so that they can be reused.
List the storage nodes.
oc get nodes -l cluster.ocs.openshift.io/openshift-storage=
oc get nodes -l cluster.ocs.openshift.io/openshift-storage=Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
NAME STATUS ROLES AGE VERSION node-xxx Ready worker 4h45m v1.18.3+6c42de8 node-yyy Ready worker 4h46m v1.18.3+6c42de8 node-zzz Ready worker 4h45m v1.18.3+6c42de8
NAME STATUS ROLES AGE VERSION node-xxx Ready worker 4h45m v1.18.3+6c42de8 node-yyy Ready worker 4h46m v1.18.3+6c42de8 node-zzz Ready worker 4h45m v1.18.3+6c42de8Copy to Clipboard Copied! Toggle word wrap Toggle overflow Obtain the node console and execute
chroot /hostcommand when the prompt appears.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Store the disk paths in the DISKS variable within quotes. For the list of disk paths, see step 3 and step 8.c for local volumeset and local volume respectively.
Example output:
DISKS="/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3 " DISKS="/dev/sdb /dev/sdc /dev/sdd /dev/sde ".
sh-4.4# DISKS="/dev/disk/by-id/scsi-360050763808104bc28000000000000eb /dev/disk/by-id/scsi-360050763808104bc28000000000000ef /dev/disk/by-id/scsi-360050763808104bc28000000000000f3 " or sh-4.2# DISKS="/dev/sdb /dev/sdc /dev/sdd /dev/sde ".Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run
sgdisk --zap-allon all the disks.for disk in $DISKS; do sgdisk --zap-all $disk;done
sh-4.4# for disk in $DISKS; do sgdisk --zap-all $disk;doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Exit the shell and repeat for the other nodes.
exit exit
sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ...Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Delete the
openshift-local-storagenamespace and wait till the deletion is complete. You will need to switch to another project if theopenshift-local-storagenamespace is the active project.For example:
oc project default oc delete project openshift-local-storage --wait=true --timeout=5m
$ oc project default $ oc delete project openshift-local-storage --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow The project is deleted if the following command returns a NotFound error.
oc get project openshift-local-storage
$ oc get project openshift-local-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow
4.2. Removing monitoring stack from OpenShift Container Storage Copy linkLink copied to clipboard!
Use this section to clean up the monitoring stack from OpenShift Container Storage.
The PVCs that are created as a part of configuring the monitoring stack are in the openshift-monitoring namespace.
Prerequisites
PVCs are configured to use OpenShift Container Platform monitoring stack.
For information, see configuring monitoring stack.
Procedure
List the pods and PVCs that are currently running in the
openshift-monitoringnamespace.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the monitoring
configmap.oc -n openshift-monitoring edit configmap cluster-monitoring-config
$ oc -n openshift-monitoring edit configmap cluster-monitoring-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove any
configsections that reference the OpenShift Container Storage storage classes as shown in the following example and save it.Before editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Expand Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
alertmanagerMainandprometheusK8smonitoring components are using the OpenShift Container Storage PVCs.Delete relevant PVCs. Make sure you delete all the PVCs that are consuming the storage classes.
oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5m
$ oc delete -n openshift-monitoring pvc <pvc-name> --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow
4.3. Removing OpenShift Container Platform registry from OpenShift Container Storage Copy linkLink copied to clipboard!
Use this section to clean up OpenShift Container Platform registry from OpenShift Container Storage. If you want to configure an alternative storage, see image registry
The PVCs that are created as a part of configuring OpenShift Container Platform registry are in the openshift-image-registry namespace.
Prerequisites
- The image registry should have been configured to use an OpenShift Container Storage PVC.
Procedure
Edit the
configs.imageregistry.operator.openshift.ioobject and remove the content in the storage section.oc edit configs.imageregistry.operator.openshift.io
$ oc edit configs.imageregistry.operator.openshift.ioCopy to Clipboard Copied! Toggle word wrap Toggle overflow Expand Before editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After editing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the PVC is called
registry-cephfs-rwx-pvc, which is now safe to delete.Delete the PVC.
oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-image-registry --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow
4.4. Removing the cluster logging operator from OpenShift Container Storage Copy linkLink copied to clipboard!
To clean the cluster logging operator from the OpenShift Container Storage, follow the steps in the procedure.
The PVCs created as a part of configuring cluster logging operator are in the openshift-logging namespace.
Prerequisites
- The cluster logging instance must be configured to use OpenShift Container Storage PVCs.
Procedure
Remove the
ClusterLogginginstance in the namespace.oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5m
$ oc delete clusterlogging instance -n openshift-logging --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow The PVCs in the
openshift-loggingnamespace are now safe to delete.Delete PVCs.
oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5m
$ oc delete pvc <pvc-name> -n openshift-logging --wait=true --timeout=5mCopy to Clipboard Copied! Toggle word wrap Toggle overflow