Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 4. Scaling storage nodes

To scale the storage capacity of OpenShift Container Storage in internal mode, you can do either of the following:

  • Scale up storage nodes - Add storage capacity to the existing Red Hat OpenShift Container Storage worker nodes
  • Scale out storage nodes - Add new worker nodes containing storage capacity

For scaling your storage in external mode, see Red Hat Ceph Storage documentation.

4.1. Requirements for scaling storage nodes

Before you proceed to scale the storage nodes, refer to the following sections to understand the node requirements for your specific Red Hat OpenShift Container Storage instance:

Important

Always ensure that you have plenty of storage capacity.

If storage ever fills completely, it is not possible to add capacity or delete or migrate content away from the storage to free up space. Completely full storage is very difficult to recover.

Capacity alerts are issued when cluster storage capacity reaches 75% (near-full) and 85% (full) of total capacity. Always address capacity warnings promptly, and review your storage regularly to ensure that you do not run out of storage space.

If you do run out of storage space completely, contact Red Hat Customer Support.

4.1.1. Supported Deployments for Red Hat OpenShift Container Storage

  • User-provisioned infrastructure:

    • Amazon Web Services (AWS)
    • VMware
    • Bare metal

  • Installer-provisioned infrastructure:

    • Amazon Web Services (AWS)

4.2. Scaling up storage capacity

Depending on the type of your deployment, you can choose one of the following procedures to scale up storage capacity.

4.2.1. Scaling up storage by adding capacity to your OpenShift Container Storage nodes on AWS or VMware infrastructure

Use this procedure to add storage capacity and performance to your configured Red Hat OpenShift Container Storage worker nodes.

Prerequisites

  • A running OpenShift Container Storage Platform
  • Administrative privileges on the OpenShift Web Console

Procedure

  1. Navigate to the OpenShift Web Console.
  2. Click on Operators on the left navigation bar.
  3. Select Installed Operators.

  4. In the window, click OpenShift Container Storage Operator:

    ocs installed operators

  5. In the top navigation bar, scroll right and click Storage Cluster tab.

    OCS Storage Cluster overview
  6. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

  7. Select Add Capacity from the options menu.

    OCS add capacity dialog menu

    From this dialog box, you can set the requested additional capacity and the storage class. Add capacity will show the capacity selected at the time of installation and will allow to add the capacity only in this increment. On AWS, the storage class should be set to gp2. On VMware, the storage class should be set to thin.

    Note

    The effectively provisioned capacity will be three times as much as what you see in the Raw Capacity field because OpenShift Container Storage uses a replica count of 3.

  8. Once you are done with your setting, click Add. You might need to wait a couple of minutes for the storage cluster to reach Ready state.

Verification steps

  1. Navigate to Overview Persistent Storage tab, then check the Capacity breakdown card.

    ocs add capacity expansion verification capacity card aws
  2. Note that the capacity increases based on your selections.
Important

As of OpenShift Container Storage 4.2, cluster reduction, whether by reducing OSDs or nodes, is not supported.

4.2.2. Scaling up storage by adding capacity to your OpenShift Container Storage nodes using local storage devices

Use this procedure to add storage capacity (additional storage devices) to your configured local storage based OpenShift Container Storage worker nodes on bare metal, and VMware infrastructures.

Important

Scaling up storage on Amazon EC2 I3 is a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

Note

For Amazon EC2 I3 infrastructure, adding nodes is the only option for adding capacity, as deployment is done using both the available NVMe devices.

Prerequisites

Procedure

  1. To add storage capacity to OpenShift Container Platform nodes with OpenShift Container Storage installed, you need to

    1. Find the unique by-id identifier for available devices that you want to add, that is, a minimum of one device per worker node. You can follow the procedure for finding available storage devices in the respective deployment guide.

      Note

      Make sure you perform this process for all the existing nodes (minimum of 3) for which you want to add storage.

    2. Add the unique device ID to the LocalVolume custom resource (CR). 

      $ oc edit -n local-storage localvolume local-block

      Example output: 

      spec:
  logLevel: Normal
  managementState: Managed
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: cluster.ocs.openshift.io/openshift-storage
        operator: In
        values:
        - ""
  storageClassDevices:
  - devicePaths:
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402P51P0GGN
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402LM1P0GGN
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402M21P0GGN
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN    # newly  added device by-id
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN    # newly  added device by-id
    - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN    # newly  added device by-id
    storageClassName: localblock
    volumeMode: Block

      Make sure to save the changes after editing the CR.

      Example output: 

      localvolume.local.storage.openshift.io/local-block edited

      You can see in this CR that new devices using by-id have been added. Each new device maps to one NVMe device on the three worker nodes.

      • nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN

  2. Display the newly created PVs with storageclass name used in localVolume CR. 

    $ oc get pv | grep localblock | grep Available

    Example output: 

    local-pv-5ee61dcc   931Gi   RWO     Delete  Available   localblock      2m35s
local-pv-b1fa607a   931Gi   RWO     Delete  Available   localblock      2m27s
local-pv-e971c51d   931Gi   RWO     Delete  Available   localblock      2m22s
...

    There are three more available PVs of same size which will be used for new OSDs.

  3. Navigate to the OpenShift Web Console.
  4. Click on Operators on the left navigation bar.
  5. Select Installed Operators.

  6. In the window, click OpenShift Container Storage Operator:

    ocs installed operators

  7. In the top navigation bar, scroll right and click Storage Cluster tab.

    OCS Storage Cluster overview
  8. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

  9. Select Add Capacity from the options menu.

    OCS add capacity dialog menu lso

    From this dialog box, set the Storage Class name to the name used in the localVolume CR. Available Capacity displayed is based on the local disks available in storage class.

  10. Once you are done with your setting, click Add. You might need to wait a couple of minutes for the storage cluster to reach Ready state.

  11. Verify that the new OSDs and their corresponding new PVCs are created. 

    $ oc get -n openshift-storage pods -l app=rook-ceph-osd

    Example output: 

    NAME                               READY   STATUS    RESTARTS   AGE
rook-ceph-osd-0-77c4fdb758-qshw4   1/1     Running   0          1h
rook-ceph-osd-1-8645c5fbb6-656ks   1/1     Running   0          1h
rook-ceph-osd-2-86895b854f-r4gt6   1/1     Running   0          1h
rook-ceph-osd-3-dc7f787dd-gdnsz    1/1     Running   0          10m
rook-ceph-osd-4-554b5c46dd-hbf9t   1/1     Running   0          10m
rook-ceph-osd-5-5cf94c4448-k94j6   1/1     Running   0          10m

    In the above example, osd-3, osd-4, and osd-5 are the newly added pods to the OpenShift Container Storage cluster. 

    $ oc get pvc -n openshift-storage |grep localblock

    Example output: 

    ocs-deviceset-0-0-qc29m   Bound    local-pv-fc5562d3    931Gi   RWO  localblock  1h
ocs-deviceset-0-1-qdmrl   Bound    local-pv-b1fa607a    931Gi   RWO  localblock  10m
ocs-deviceset-1-0-mpwmk   Bound    local-pv-58cdd0bc    931Gi   RWO  localblock  1h
ocs-deviceset-1-1-85892   Bound    local-pv-e971c51d    931Gi   RWO  localblock  10m
ocs-deviceset-2-0-rll47   Bound    local-pv-29d8ad8d    931Gi   RWO  localblock  1h
ocs-deviceset-2-1-cgth2   Bound    local-pv-5ee61dcc    931Gi   RWO  localblock  10m

    In the above example, we see three new PVCs are created.

Verification steps

  1. Navigate to Overview Persistent Storage tab, then check the Capacity breakdown card.

    ocs add capacity expansion verification capacity card bm

    Note that the capacity increases based on your selections.

    Important

    OpenShift Container Storage does not support cluster reduction either by reducing OSDs or reducing nodes.

4.3. Scaling out storage capacity

To scale out storage capacity, you need to perform the following steps:

  • Add a new node
  • Verify that the new node is added successfully
  • Scale up the storage capacity

4.3.1. Adding a node

You can add nodes to increase the storage capacity when existing worker nodes are already running at their maximum supported OSDs, which is increment of 3 OSDs of the capacity selected during initial configuration.

Depending on the type of your deployment, you can choose one of the following procedures to add a storage node:

4.3.1.1. Adding a node on an AWS installer-provisioned infrastructure

Prerequisites

  • You must be logged into OpenShift Container Platform (OCP) cluster.

Procedure

  1. Navigate to Compute Machine Sets.
  2. On the machine set where you want to add nodes, select Edit Machine Count.
  3. Add the amount of nodes, and click Save.
  4. Click Compute Nodes and confirm if the new node is in Ready state.

  5. Apply the OpenShift Container Storage label to the new node.

    1. For the new node, Action menu (⋮) Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.
Note

It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Verification steps

To verify that the new node is added, see Section 4.3.2, “Verifying the addition of a new node”.

4.3.1.2. Adding a node on an AWS or a VMware user-provisioned infrastructure

Prerequisites

  • You must be logged into OpenShift Container Platform (OCP) cluster.

Procedure

  1. Depending on whether you are adding a node on an AWS user provisioned infrastructure or a VMware user-provisioned infrastructure, perform the following steps:

    • For AWS

      1. Create a new AWS machine instance with the required infrastructure. See Platform requirements.
      2. Create a new OpenShift Container Platform node using the new AWS machine instance.

    • For VMware:

      1. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
      2. Create a new OpenShift Container Platform worker node using the new VM.

  2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state: 

    $ oc get csr

  3. Approve all required OpenShift Container Storage CSRs for the new node: 

    $ oc adm certificate approve <Certificate_Name>
  4. Click Compute Nodes, confirm if the new node is in Ready state.

  5. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface
    1. For the new node, click Action Menu (⋮) Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.
    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node: 

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
      Note

      It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Verification steps

To verify that the new node is added, see Section 4.3.2, “Verifying the addition of a new node”.

4.3.1.3. Adding a node using a local storage device

Use this procedure to add a node on bare metal, Amazon EC2, and VMware infrastructures.

Important

Scaling storage nodes for Amazon EC2 infrastructure is a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

Prerequisites

  • You must be logged into OpenShift Container Platform (OCP) cluster.
  • You must have three OpenShift Container Platform worker nodes with the same storage type and size attached to each node (for example, 2TB NVMe drive) as the original OCS StorageCluster was created with.

Procedure

  1. Depending on whether you are adding a node on bare metal, Amazon EC2, or VMware infrastructure, perform the following steps:

    • For Amazon EC2

      1. Create a new Amazon EC2 I3 machine instance with the required infrastructure. See Creating a MachineSet in AWS and Platform requirements.
      2. Create a new OpenShift Container Platform node using the new Amazon EC2 I3 machine instance.

    • For VMware:

      1. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
      2. Create a new OpenShift Container Platform worker node using the new VM.

    • For bare metal:

      1. Get a new bare metal machine with the required infrastructure. See Platform requirements.
      2. Create a new OpenShift Container Platform node using the new bare metal machine.

  2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state: 

    $ oc get csr

  3. Approve all required OpenShift Container Storage CSRs for the new node: 

    $ oc adm certificate approve <Certificate_Name>
  4. Click Compute Nodes, confirm if the new node is in Ready state.

  5. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface
    1. For the new node, click Action Menu (⋮) Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.
    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node: 

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
      Note

      It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Verification steps

To verify that the new node is added, see Section 4.3.2, “Verifying the addition of a new node”.

4.3.2. Verifying the addition of a new node

  1. Execute the following command and verify that the new node is present in the output: 

    $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

  2. Click Workloads Pods, confirm that at least the following pods on the new node are in Running state:

    • csi-cephfsplugin-*
    • csi-rbdplugin-*

4.3.3. Scaling up storage capacity

To scale up storage capacity, see Scaling up storage by adding capacity.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.