Chapter 5. Scaling storage of VMware OpenShift Data Foundation cluster
5.1. Scaling up storage of VMware OpenShift Data Foundation cluster Copy linkLink copied to clipboard!
To increase the storage capacity in a dynamically created VMware storage cluster on user-provisioned and installer-provisioned infrastructures, you can add storage capacity and performance to your configured Red Hat OpenShift Data Foundation worker nodes.
You can scale up storage capacity of a VMware Red Hat OpenShift Data Foundation cluster in two ways:
5.1.1. Scaling up storage on a dynamically provisioned VMware cluster by adding a new set of OSDs Copy linkLink copied to clipboard!
To increase the storage capacity in a dynamically created storage cluster on a VMware user-provisioned infrastructure, you can add storage capacity and performance to your configured Red Hat OpenShift Data Foundation worker nodes.
Prerequisites
- Administrative privilege to the OpenShift Container Platform Console.
- A running OpenShift Data Foundation Storage Cluster.
- Make sure that the disk is of the same size and type as the disk used during initial deployment.
Procedure
- Log in to the OpenShift Web Console.
-
Click Operators
Installed Operators. - Click OpenShift Data Foundation Operator.
Click the Storage Systems tab.
- Click the Action Menu (⋮) on the far right of the storage system name to extend the options menu.
- Select Add Capacity from the options menu.
- Select the Storage Class. Choose the storage class which you wish to use to provision new storage devices.
- Click Add.
-
To check the status, navigate to Storage
Data Foundation and verify that Storage Systemin the Status card has a green tick.
Verification steps
Verify the Raw Capacity card.
-
In the OpenShift Web Console, click Storage
Data Foundation. - In the Status card of the Overview tab, click Storage System and then click the storage system link from the pop up that appears.
In the Block and File tab, check the Raw Capacity card.
Note that the capacity increases based on your selections.
NoteThe raw capacity does not take replication into account and shows the full capacity.
-
In the OpenShift Web Console, click Storage
Verify that the new OSDs and their corresponding new Persistent Volume Claims (PVCs) are created.
To view the state of the newly created OSDs:
-
Click Workloads
Pods from the OpenShift Web Console. Select
openshift-storagefrom the Project drop-down list.NoteIf the Show default projects option is disabled, use the toggle button to list all the default projects.
-
Click Workloads
To view the state of the PVCs:
-
Click Storage
Persistent Volume Claims from the OpenShift Web Console. Select
openshift-storagefrom the Project drop-down list.NoteIf the Show default projects option is disabled, use the toggle button to list all the default projects.
-
Click Storage
Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
Identify the nodes where the new OSD pods are running.
$ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/<OSD-pod-name><OSD-pod-name>Is the name of the OSD pod.
For example:
$ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqmExample output:
NODE compute-1
For each of the nodes identified in the previous step, do the following:
Create a debug pod and open a chroot environment for the selected hosts.
$ oc debug node/<node-name><node-name>Is the name of the node.
$ chroot /host
Check for the
cryptkeyword beside theocs-devicesetnames.$ lsblk
Cluster reduction is supported only with the Red Hat Support Team’s assistance.
5.1.2. Scaling up storage capacity on VMware a cluster by resizing existing OSDs Copy linkLink copied to clipboard!
To increase the storage capacity on a cluster, you can add storage capacity by resizing existing OSDs.
Before resizing OSDs, verify that the underlying datastore has enough available space for the resize to succeed.
Prerequisites
- You have administrative privilege to the OpenShift Container Platform Console.
- You have a running OpenShift Data Foundation Storage Cluster.
Procedure
Update the
dataPVCTemplatesize for thestorageDeviceSetswith the new desired size using theoc patchcommand.storageDeviceSets: - name: example-deviceset count: 3 resources: {} placement: {} dataPVCTemplate: spec: storageClassName: accessModes: - ReadWriteOnce volumeMode: Block resources: requests: storage: 512GiIn this example YAML, the
storageparameter understorageDeviceSetsreflects the current size of512Gi.Using the
oc patchcommand:Get the current OSD storage for the
storageDeviceSetsyou are increasing storage for:oc get storagecluster ocs-storagecluster -n openshift-storage -o jsonpath=' {.spec.storageDeviceSets[0].dataPVCTemplate.spec.resources.requests.storage} ' 512GiIncrease the storage with the desired value (the following example reflect the size change of 2Ti):
oc patch storagecluster ocs-storagecluster -n openshift-storage --type merge --patch "$(oc get storagecluster ocs-storagecluster -n openshift-storage -o jsonpath=' {.spec.storageDeviceSets[0]} ' | jq '.dataPVCTemplate.spec.resources.requests.storage="2Ti"' | jq -c '{spec: {storageDeviceSets: [.]}}')" storagecluster.ocs.openshift.io/ocs-storagecluster patched
- Wait for the OSDs to restart.
Confirm that the resize took effect:
$ oc get pvc -l ceph.rook.io/DeviceSet -n openshift-storageVerify that for all the resized OSDs, resize is completed and reflected correctly in the
CAPACITYcolumn of the command output.- If the resize did not take effect, restart the OSD pods again. It may take multiple restarts for the resize to complete.
5.2. Scaling up a cluster created using local storage devices Copy linkLink copied to clipboard!
To scale up an OpenShift Data Foundation cluster which was created using local storage devices, you need to add a new disk to the storage node. The new disks size must be of the same size as the disks used during the deployment because OpenShift Data Foundation does not support heterogeneous disks/OSDs.
For deployments having three failure domains, you can scale up the storage by adding disks in the multiples of three, with the same number of disks coming from nodes in each of the failure domains. For example, if we scale by adding six disks, two disks are taken from nodes in each of the three failure domains. If the number of disks is not in multiples of three, it will only consume the disk to the maximum in the multiple of three while the remaining disks remain unused.
For deployments having less than three failure domains, there is a flexibility to add any number of disks. Make sure to verify that flexible scaling is enabled. For information, refer to the Knowledgebase article Verify if flexible scaling is enabled.
Flexible scaling features get enabled at the time of deployment and cannot be enabled or disabled later on.
Prerequisites
- Administrative privilege to the OpenShift Container Platform Console.
- A running OpenShift Data Foundation Storage Cluster.
- Make sure that the disks to be used for scaling are attached to the storage node
-
Make sure that
LocalVolumeDiscoveryandLocalVolumeSetobjects are created.
Procedure
To add capacity, you can either use a storage class that you provisioned during the deployment or any other storage class that matches the filter.
-
In the OpenShift Web Console, click Operators
Installed Operators. - Click OpenShift Data Foundation Operator.
Click the Storage Systems tab.
- Click the Action menu (⋮) next to the visible list to extend the options menu.
- Select Add Capacity from the options menu.
- Select the Storage Class for which you added disks or the new storage class depending on your requirement. Available Capacity displayed is based on the local disks available in storage class.
- Click Add.
-
To check the status, navigate to Storage
Data Foundation and verify that the Storage System in the Status card has a green tick.
Verification steps
Verify the Raw Capacity card.
-
In the OpenShift Web Console, click Storage
Data Foundation. - In the Status card of the Overview tab, click Storage System and then click the storage system link from the pop up that appears.
In the Block and File tab, check the Raw Capacity card.
Note that the capacity increases based on your selections.
NoteThe raw capacity does not take replication into account and shows the full capacity.
-
In the OpenShift Web Console, click Storage
Verify that the new OSDs and their corresponding new Persistent Volume Claims (PVCs) are created.
To view the state of the newly created OSDs:
-
Click Workloads
Pods from the OpenShift Web Console. Select
openshift-storagefrom the Project drop-down list.NoteIf the Show default projects option is disabled, use the toggle button to list all the default projects.
-
Click Workloads
To view the state of the PVCs:
-
Click Storage
Persistent Volume Claims from the OpenShift Web Console. Select
openshift-storagefrom the Project drop-down list.NoteIf the Show default projects option is disabled, use the toggle button to list all the default projects.
-
Click Storage
Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
Identify the nodes where the new OSD pods are running.
$ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/<OSD-pod-name><OSD-pod-name>Is the name of the OSD pod.
For example:
$ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqmExample output:
NODE compute-1
For each of the nodes identified in the previous step, do the following:
Create a debug pod and open a chroot environment for the selected host(s).
$ oc debug node/<node-name><node-name>Is the name of the node.
$ chroot /host
Check for the
cryptkeyword beside theocs-devicesetnames.$ lsblk
Cluster reduction is supported only with the Red Hat Support Team’s assistance.
5.3. Scaling out storage capacity on a VMware cluster Copy linkLink copied to clipboard!
5.3.1. Adding a node to an installer-provisioned infrastructure Copy linkLink copied to clipboard!
Prerequisites
- You have administrative privilege to the OpenShift Container Platform Console.
- You have a running OpenShift Data Foundation Storage Cluster.
Procedure
-
Navigate to Compute
Machine Sets. On the machine set where you want to add nodes, select Edit Machine Count.
- Add the amount of nodes, and click Save.
-
Click Compute
Nodes and confirm if the new node is in Ready state.
Apply the OpenShift Data Foundation label to the new node.
-
For the new node, click Action menu (⋮)
Edit Labels. - Add cluster.ocs.openshift.io/openshift-storage, and click Save.
-
For the new node, click Action menu (⋮)
It is recommended to add 3 nodes, one each in different zones. You must add 3 nodes and perform this procedure for all of them. In case of bare metal installer-provisioned infrastructure deployment, you must expand the cluster first. For instructions, see Expanding the cluster.
Verification steps
Execute the following command in the terminal and verify that the new node is present in the output:
$ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1On the OpenShift web console, click Workloads
Pods, confirm that at least the following pods on the new node are in Running state: -
csi-cephfsplugin-* -
csi-rbdplugin-*
-
To scale up storage capacity:
- For dynamic storage devices, see Scaling up storage capacity on a cluster.
5.3.2. Adding a node to an user-provisioned infrastructure Copy linkLink copied to clipboard!
Prerequisites
- You have administrative privilege to the OpenShift Container Platform Console.
- You have a running OpenShift Data Foundation Storage Cluster.
Procedure
Depending on the type of infrastructure, perform the following steps:
- Get a new machine with the required infrastructure. See Platform requirements.
- Create a new OpenShift Container Platform worker node using the new machine.
Check for certificate signing requests (CSRs) that are in
Pendingstate.$ oc get csrApprove all the required CSRs for the new node.
$ oc adm certificate approve <Certificate_Name><Certificate_Name>- Is the name of the CSR.
-
Click Compute
Nodes, confirm if the new node is in Ready state. Apply the OpenShift Data Foundation label to the new node using any one of the following:
- From User interface
-
For the new node, click Action Menu (⋮)
Edit Labels. -
Add
cluster.ocs.openshift.io/openshift-storage, and click Save.
-
For the new node, click Action Menu (⋮)
- From Command line interface
Apply the OpenShift Data Foundation label to the new node.
$ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""<new_node_name>- Is the name of the new node.
Verification steps
Execute the following command in the terminal and verify that the new node is present in the output:
$ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1On the OpenShift web console, click Workloads
Pods, confirm that at least the following pods on the new node are in Running state: -
csi-cephfsplugin-* -
csi-rbdplugin-*
-
To scale up storage capacity:
- For dynamic storage devices, see Scaling up storage capacity on a cluster.
5.3.3. Adding a node using a local storage device Copy linkLink copied to clipboard!
You can add nodes to increase the storage capacity when existing worker nodes are already running at their maximum supported OSDs or when there are not enough resources to add new OSDs on the existing nodes.
Add nodes in the multiple of 3, each of them in different failure domains. Though it is recommended to add nodes in multiples of 3 nodes, you have the flexibility to add one node at a time in flexible scaling deployment. See Knowledgebase article Verify if flexible scaling is enabled
OpenShift Data Foundation does not support heterogeneous disk size and types. The new nodes to be added should have the disk of the same type and size which was used during initial OpenShift Data Foundation deployment.
Prerequisites
- You have administrative privilege to the OpenShift Container Platform Console.
- You have a running OpenShift Data Foundation Storage Cluster.
Procedure
Depending on the type of infrastructure, perform the following steps:
- Get a new machine with the required infrastructure. See Platform requirements.
- Create a new OpenShift Container Platform worker node using the new machine.
Check for certificate signing requests (CSRs) that are in
Pendingstate.$ oc get csrApprove all the required CSRs for the new node.
$ oc adm certificate approve <Certificate_Name><Certificate_Name>- Is the name of the CSR.
-
Click Compute
Nodes, confirm if the new node is in Ready state. Apply the OpenShift Data Foundation label to the new node using any one of the following:
- From User interface
-
For the new node, click Action Menu (⋮)
Edit Labels. -
Add
cluster.ocs.openshift.io/openshift-storage, and click Save.
-
For the new node, click Action Menu (⋮)
- From Command line interface
Apply the OpenShift Data Foundation label to the new node.
$ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""<new_node_name>- Is the name of the new node.
Click Operators
Installed Operators from the OpenShift Web Console. From the Project drop-down list, make sure to select the project where the Local Storage Operator is installed.
- Click Local Storage.
Click the Local Volume Discovery tab.
-
Beside the
LocalVolumeDiscovery, click Action menu (⋮)Edit Local Volume Discovery. -
In the YAML, add the hostname of the new node in the
valuesfield under the node selector. - Click Save.
-
Beside the
Click the Local Volume Sets tab.
-
Beside the
LocalVolumeSet, click Action menu (⋮)Edit Local Volume Set. In the YAML, add the hostname of the new node in the
valuesfield under thenode selector.
- Click Save.
-
Beside the
It is recommended to add 3 nodes, one each in different zones. You must add 3 nodes and perform this procedure for all of them.
Verification steps
Execute the following command in the terminal and verify that the new node is present in the output:
$ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1On the OpenShift web console, click Workloads
Pods, confirm that at least the following pods on the new node are in Running state: -
csi-cephfsplugin-* -
csi-rbdplugin-*
-
To scale up storage capacity:
- For local storage devices, see Scaling up a cluster created using local storage devices