OpenShift Container Storage is now OpenShift Data Foundation starting with version 4.9.
Chapter 8. Scaling storage nodes
To scale the storage capacity of OpenShift Container Storage, you can do either of the following:
- Scale up storage nodes - Add storage capacity to the existing OpenShift Container Storage worker nodes
- Scale out storage nodes - Add new worker nodes containing storage capacity
8.1. Requirements for scaling storage nodes Copy linkLink copied to clipboard!
Before you proceed to scale the storage nodes, refer to the following sections to understand the node requirements for your specific Red Hat OpenShift Container Storage instance:
- Platform requirements
Storage device requirements
Always ensure that you have plenty of storage capacity.
If storage ever fills completely, it is not possible to add capacity or delete or migrate content away from the storage to free up space. Completely full storage is very difficult to recover.
Capacity alerts are issued when cluster storage capacity reaches 75% (near-full) and 85% (full) of total capacity. Always address capacity warnings promptly, and review your storage regularly to ensure that you do not run out of storage space.
If you do run out of storage space completely, contact Red Hat Customer Support.
8.2. Scaling up storage by adding capacity to your OpenShift Container Storage nodes using local storage devices Copy linkLink copied to clipboard!
Use this procedure to add storage capacity (additional storage devices) to your configured local storage based OpenShift Container Storage worker nodes on Red Hat Virtualization infrastructures.
Prerequisites
- You must be logged into the OpenShift Container Platform cluster.
- You must have installed the Local Storage Operator. See Install Local Storage Operator
-
If you have upgraded from a previous version of OpenShift Container Storage, create a
LocalVolumeSetobject to enable automatic provisioning of devices as described in Post-update configuration changes. -
If you upgraded to OpenShift Container Storage 4.6 from a previous version, ensure that you have followed post-upgrade procedures to create the
LocalVolumeDiscoveryobject. See Post-update configuration changes for details. - You must have three OpenShift Container Platform worker nodes with the same storage type and size attached to each node (for example, 2TB NVMe drive) as the original OpenShift Container Storage StorageCluster was created with.
Procedure
To add capacity, you can either use a storage class that you provisioned during the deployment or any other storage class that matches the filter.
On the OpenShift web console, click on Operators
Installed Operators. - Click OpenShift Container Storage Operator.
Click Storage Cluster tab.
- The visible list should have only one item. Click (⋮) on the far right to extend the options menu.
Select Add Capacity from the options menu.
- Select the Storage Class for which you added disks or the new storage class depending on your requirement. Available Capacity displayed is based on the local disks available in storage class.
Click Add.
You might need to wait a couple of minutes for the storage cluster to reach Ready state.
Verification steps
Navigate to Overview
Persistent Storage tab, then check the Capacity breakdown card. Note that the capacity increases based on your selections.
Verify that the new OSDs and their corresponding new PVCs are created.
To view the state of the newly created OSDs:
-
Click Workloads
Pods from the OpenShift Web Console. -
Select
openshift-storagefrom the Project drop-down list.
-
Click Workloads
To view the state of the PVCs:
-
Click Storage
Persistent Volume Claims from the OpenShift Web Console. -
Select
openshift-storagefrom the Project drop-down list.
-
Click Storage
(Optional) If data encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
Identify the node(s) where the new OSD pod(s) are running.
oc get -o=custom-columns=NODE:.spec.nodeName pod/<OSD pod name>
$ oc get -o=custom-columns=NODE:.spec.nodeName pod/<OSD pod name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example:
oc get -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm
oc get -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqmCopy to Clipboard Copied! Toggle word wrap Toggle overflow For each of the nodes identified in previous step, do the following:
Create a debug pod and open a chroot environment for the selected host(s).
oc debug node/<node name> chroot /host
$ oc debug node/<node name> $ chroot /hostCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run “lsblk” and check for the “crypt” keyword beside the
ocs-devicesetname(s)lsblk
$ lsblkCopy to Clipboard Copied! Toggle word wrap Toggle overflow
OpenShift Container Storage does not support cluster reduction either by reducing OSDs or reducing nodes.
8.3. Scaling out storage capacity by adding new nodes Copy linkLink copied to clipboard!
To scale out storage capacity, you need to perform the following:
- Add a new node to increase the storage capacity when existing worker nodes are already running at their maximum supported OSDs, which is the increment of 3 OSDs of the capacity selected during initial configuration.
- Verify that the new node is added successfully
- Scale up the storage capacity after the node is added
8.3.1. Adding a node using a local storage device Copy linkLink copied to clipboard!
Use this procedure to add a node on Red Hat Virtualization infrastructures.
Prerequisites
- You must be logged into the OpenShift Container Platform (RHOCP) cluster.
-
If you upgraded to OpenShift Container Storage 4.6 from a previous version, ensure that you have followed post-upgrade procedures to create the
LocalVolumeDiscoveryobject. See Post-update configuration changes for details. - You must have three OpenShift Container Platform worker nodes with the same storage type and size attached to each node (for example, 2TB SSD or 2TB NVMe drive) as the original OpenShift Container Storage StorageCluster was created with.
- If you have upgraded from a previous version of OpenShift Container Storage, create a LocalVolumeSet object to enable automatic provisioning of devices as described in Post-update configuration changes.
Procedure
- Create a new VM on Red Hat Virtualization with the required infrastructure. See Platform requirements.
- Create a new OpenShift Container Platform worker node using the new VM.
Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in
Pendingstate:oc get csr
$ oc get csrCopy to Clipboard Copied! Toggle word wrap Toggle overflow Approve all required OpenShift Container Storage CSRs for the new node:
oc adm certificate approve <Certificate_Name>
$ oc adm certificate approve <Certificate_Name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Click Compute
Nodes, confirm if the new node is in Ready state. Apply the OpenShift Container Storage label to the new node using any one of the following:
- From User interface
-
For the new node, click Action Menu (⋮)
Edit Labels -
Add
cluster.ocs.openshift.io/openshift-storageand click Save.
-
For the new node, click Action Menu (⋮)
- From Command line interface
Execute the following command to apply the OpenShift Container Storage label to the new node:
oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
$ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Click Operators
Installed Operators from the OpenShift Web Console. From the Project drop-down list, make sure to select the project where the Local Storage Operator is installed.
- Click on Local Storage.
- Click the Local Volume Discovery tab
-
Beside the
LocalVolumeDiscovery, click Action menu (⋮)Edit Local Volume Discovery. -
In the YAML, add the hostname of the new node in the values field under the
node selector. - Click Save.
- Click the Local Volume Sets tab.
-
Beside the
LocalVolumeSet, click Action menu (⋮)Edit Local Volume Set. In the YAML, add the hostname of the new node in the
valuesfield under thenode selector.Figure 8.1. YAML showing the addition of new hostnames
- Click Save.
It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.
Verification steps
- To verify that the new node is added, see Verifying the addition of a new node.
8.3.2. Verifying the addition of a new node Copy linkLink copied to clipboard!
Execute the following command and verify that the new node is present in the output:
oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1
$ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1Copy to Clipboard Copied! Toggle word wrap Toggle overflow Click Workloads
Pods, confirm that at least the following pods on the new node are in Running state: -
csi-cephfsplugin-* -
csi-rbdplugin-*
-
8.3.3. Scaling up storage capacity Copy linkLink copied to clipboard!
After you add a new node to OpenShift Container Storage, you must scale up the storage capacity as described in Scaling up storage by adding capacity.