Managing OpenShift Container Storage

Procedure

1. Navigate to Compute → Machine Sets.
2. On the machine set where you want to add nodes, select Edit Machine Count.
3. Add the amount of nodes, and click Save.
4. Click Compute → Nodes and confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node.

   1. For the new node, Action menu (⋮) → Edit Labels.
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

Procedure

1. Depending on whether you are adding a node on an AWS user provisioned infrastructure or a VMware user-provisioned infrastructure, perform the following steps:

   • For AWS

     1. Create a new AWS machine instance with the required infrastructure. See Platform requirements.
     2. Create a new OpenShift Container Platform node using the new AWS machine instance.

   • For VMware:

     1. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
     2. Create a new OpenShift Container Platform worker node using the new VM.

2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

   $ oc get csr

3. Approve all required OpenShift Container Storage CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

4. Click Compute → Nodes, confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Note

     It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Procedure

1. Depending on whether you are adding a node on bare metal, Amazon EC2, or VMware infrastructure, perform the following steps:

   • For Amazon EC2

     1. Create a new Amazon EC2 I3 machine instance with the required infrastructure. See Creating a MachineSet in AWS and Platform requirements.
     2. Create a new OpenShift Container Platform node using the new Amazon EC2 I3 machine instance.

   • For VMware:

     1. Create a new VM on vSphere with the required infrastructure. See Platform requirements.
     2. Create a new OpenShift Container Platform worker node using the new VM.

   • For bare metal:

     1. Get a new bare metal machine with the required infrastructure. See Platform requirements.
     2. Create a new OpenShift Container Platform node using the new bare metal machine.

2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

   $ oc get csr

3. Approve all required OpenShift Container Storage CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

4. Click Compute → Nodes, confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Note

     It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create <backingstore_name> --access-key=<AWS ACCESS KEY> --secret-key=<AWS SECRET ACCESS KEY> --target-bucket <bucket-name>

1. Replace <backingstore_name> with the name of the backingstore.
2. Replace <AWS ACCESS KEY> and <AWS SECRET ACCESS KEY> with an AWS access key ID and secret access key you created for this purpose.

3. Replace <bucket-name> with an existing AWS bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

   The output will be similar to the following:

   INFO[0001] ✅ Exists: NooBaa "noobaa"
   INFO[0002] ✅ Created: BackingStore "aws-resource"
   INFO[0002] ✅ Created: Secret "backing-store-secret-aws-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64>
     AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>

   1. You must supply and encode your own AWS access key ID and secret access key using Base64, and use the results in place of <AWS ACCESS KEY ID ENCODED IN BASE64> and <AWS SECRET ACCESS KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     awsS3:
       secret:
         name: <backingstore-secret-name>
         namespace: noobaa
       targetBucket: <bucket-name>
     type: aws-s3

   1. Replace <bucket-name> with an existing AWS bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <backingstore-secret-name> with the name of the secret created in the previous step.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create ibm-cos <backingstore_name> --access-key=<IBM ACCESS KEY> --secret-key=<IBM SECRET ACCESS KEY> --endpoint=<IBM COS ENDPOINT> --target-bucket <bucket-name>

   1. Replace <backingstore_name> with the name of the backingstore.

   2. Replace <IBM ACCESS KEY>, <IBM SECRET ACCESS KEY>, <IBM COS ENDPOINT> with an IBM access key ID, secret access key and the appropriate regional endpoint that corresponds to the location of the existing IBM bucket.

      To generate the above keys on IBM cloud, you must include HMAC credentials while creating the service credentials for your target bucket.

   3. Replace <bucket-name> with an existing IBM bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "ibm-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-ibm-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     IBM_COS_ACCESS_KEY_ID: <IBM COS ACCESS KEY ID ENCODED IN BASE64>
     IBM_COS_SECRET_ACCESS_KEY: <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>

   1. You must supply and encode your own IBM COS access key ID and secret access key using Base64, and use the results in place of <IBM COS ACCESS KEY ID ENCODED IN BASE64> and <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     ibmCos:
       endpoint: <endpoint>
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBucket: <bucket-name>
     type: ibm-cos

   1. Replace <bucket-name> with an existing IBM COS bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <endpoint> with a regional endpoint that corresponds to the location of the existing IBM bucket name. This argument tells Multicloud Object Gateway which endpoint to use for its backing store, and subsequently, data storage and administration.
   3. Replace <backingstore-secret-name> with the name of the secret created in the previous step.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create azure-blob <backingstore_name> --account-key=<AZURE ACCOUNT KEY> --account-name=<AZURE ACCOUNT NAME> --target-blob-container <blob container name>

   1. Replace <backingstore_name> with the name of the backingstore.
   2. Replace <AZURE ACCOUNT KEY> and <AZURE ACCOUNT NAME> with an AZURE account key and account name you created for this purpose.

   3. Replace <blob container name> with an existing Azure blob container name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "azure-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-azure-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     AccountName: <AZURE ACCOUNT NAME ENCODED IN BASE64>
     AccountKey: <AZURE ACCOUNT KEY ENCODED IN BASE64>

   1. You must supply and encode your own Azure Account Name and Account Key using Base64, and use the results in place of <AZURE ACCOUNT NAME ENCODED IN BASE64> and <AZURE ACCOUNT KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     azureBlob:
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBlobContainer: <blob-container-name>
     type: azure-blob

   1. Replace <blob-container-name> with an existing Azure blob container name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <backingstore-secret-name> with the name of the secret created in the previous step.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create google-cloud-storage <backingstore_name> --private-key-json-file=<PATH TO GCP PRIVATE KEY JSON FILE> --target-bucket <GCP bucket name>

   1. Replace <backingstore_name> with the name of the backingstore.
   2. Replace <PATH TO GCP PRIVATE KEY JSON FILE> with a path to your GCP private key created for this purpose.

   3. Replace <GCP bucket name> with an existing GCP object storage bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "google-gcp"
      INFO[0002] ✅ Created: Secret "backing-store-google-cloud-storage-gcp"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     GoogleServiceAccountPrivateKeyJson: <GCP PRIVATE KEY ENCODED IN BASE64>

   1. You must supply and encode your own GCP service account private key using Base64, and use the results in place of <GCP PRIVATE KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: openshift-storage
   spec:
     googleCloudStorage:
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       targetBucket: <target bucket>
     type: google-cloud-storage

   1. Replace <target bucket> with an existing Google storage bucket. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <backingstore-secret-name> with the name of the secret created in the previous step.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create  pv-pool <backingstore_name> --num-volumes=<NUMBER OF VOLUMES>  --pv-size-gb=<VOLUME SIZE> --storage-class=<LOCAL STORAGE CLASS>

   1. Replace <backingstore_name> with the name of the backingstore.
   2. Replace <NUMBER OF VOLUMES> with the number of volumes you would like to create.
   3. Replace <VOLUME SIZE> with the required size, in GB, of each volume

   4. Replace <LOCAL STORAGE CLASS> with the local storage class, recommended to use ocs-storagecluster-ceph-rbd

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Exists: BackingStore "local-mcg-storage"

1. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore_name>
     namespace: openshift-storage
   spec:
      pvPool:
       numVolumes: <NUMBER OF VOLUMES>
       resources:
         requests:
           storage: <VOLUME SIZE>
       storageClass: <LOCAL STORAGE CLASS>
     type: pv-pool

   1. Replace <backingstore_name> with the name of the backingstore.
   2. Replace <NUMBER OF VOLUMES> with the number of volumes you would like to create.
   3. Replace <VOLUME SIZE> with the required size, in GB, of each volume. Note that the letter G should remain
   4. Replace <LOCAL STORAGE CLASS> with the local storage class, recommended to use ocs-storagecluster-ceph-rbd

Procedure

1. Identify the node and get labels on the node to be replaced. Make a note of the rack label.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and object storage device (OSD) pods that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

5. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

6. Delete the node.

   $ oc delete node <node_name>

7. Get a new bare metal machine with required infrastructure. See Installing a cluster on bare metal.
8. Create a new OpenShift Container Platform node using the new bare metal machine.

9. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

   $ oc get csr

10. Approve all required OpenShift Container Storage CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

11. Click Compute → Nodes in OpenShift Web Console, confirm if the new node is in Ready state.

12. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

13. Add the local storage devices available in these worker nodes to the OpenShift Container Storage StorageCluster.

    1. Add a new disk entry to LocalVolume CR.

       Edit LocalVolume CR and remove or comment out failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}. In this example, the new device is /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB89THF49128A.

       # oc get -n local-storage localvolume
       NAME          AGE
       local-block   25h

       # oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY81260978128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY80440W5U128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB85AABDE128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB89THF49128A
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3e8964d3                          931Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-2-0-79j94   localblock                             25h
       local-pv-414755e0                          931Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-1-0-959rp   localblock                             25h
       local-pv-b481410                           931Gi      RWO            Delete           Available                                               localblock                             3m24s
       local-pv-d9c5cbd6                          931Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-0-0-nvs68   localblock

14. Delete the PV associated with the failed node.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       # osd_id_to_remove=0
       # oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

       In this example, the PVC name is ocs-deviceset-0-0-nvs68.

    2. Identify the PV associated with the PVC.

       # oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in the previous step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-d9c5cbd6   931Gi      RWO            localblock     24h

       In this example, the associated PV is local-pv-d9c5cbd6.

    3. Delete the PVC.

       # oc delete pvc <pvc-name> -n openshift-storage

    4. Delete the PV.

       # oc delete pv local-pv-d9c5cbd6

       Example output:

       persistentvolume "local-pv-d9c5cbd6" deleted

15. Remove the failed OSD from the cluster.

    # oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

16. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

    # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

    Note

    If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

    # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

17. Delete OSD pod deployment and crashcollector pod deployment.

    $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage
    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

18. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    # oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          1d20h

    1. Delete the rook-ceph-operator.

       # oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       # oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD and mon might take several minutes after the operator restarts.

19. Delete the ocs-osd-removal job.

    # oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Make sure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-c-64556f7659-c2ngc                                  1/1     Running     0          6h14m
   rook-ceph-mon-d-7c8b74dc4d-tt6hd                                  1/1     Running     0          4h24m
   rook-ceph-mon-e-57fb8c657-wg5f2                                   1/1     Running     0          162m

   OSD and Mon might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Identify the node and get labels on the node to be replaced. Make a note of the rack label.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and object storage device (OSD) pods that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the node as unschedulable.

   $ oc adm cordon <node_name>

5. Remove the pods which are in Terminating state

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

6. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

7. Delete the node.

   $ oc delete node <node_name>

8. Get a new bare metal machine with required infrastructure. See Installing a cluster on bare metal.
9. Create a new OpenShift Container Platform node using the new bare metal machine.

10. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

    $ oc get csr

11. Approve all required OpenShift Container Storage CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

12. Click Compute → Nodes in OpenShift Web Console, confirm if the new node is in Ready state.

13. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

14. Add the local storage devices available in these worker nodes to the OpenShift Container Storage StorageCluster.

    1. Add a new disk entry to LocalVolume CR.

       Edit LocalVolume CR and remove or comment out failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}. In this example, the new device is /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB89THF49128A.

       # oc get -n local-storage localvolume
       NAME          AGE
       local-block   25h

       # oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY81260978128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY80440W5U128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB85AABDE128A
             - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB89THF49128A
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3e8964d3                          931Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-2-0-79j94   localblock                             25h
       local-pv-414755e0                          931Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-1-0-959rp   localblock                             25h
       local-pv-b481410                           931Gi      RWO            Delete           Available                                               localblock                             3m24s
       local-pv-d9c5cbd6                          931Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-0-0-nvs68   localblock

15. Delete the PV associated with the failed node.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       # osd_id_to_remove=0
       # oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

       In this example, the PVC name is ocs-deviceset-0-0-nvs68.

    2. Identify the PV associated with the PVC.

       # oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in the previous step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-d9c5cbd6   931Gi      RWO            localblock     24h

       In this example, the associated PV is local-pv-d9c5cbd6.

    3. Delete the PVC.

       # oc delete pvc <pvc-name> -n openshift-storage

    4. Delete the PV.

       # oc delete pv local-pv-d9c5cbd6

       Example output:

       persistentvolume "local-pv-d9c5cbd6" deleted

16. Remove the failed OSD from the cluster.

    # oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

17. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

    # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

    Note

    If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

    # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

18. Delete OSD pod deployment and crashcollector pod deployment.

    $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage
    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

19. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    # oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          1d20h

    1. Delete the rook-ceph-operator.

       # oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       # oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD and mon might take several minutes after the operator restarts.

20. Delete the ocs-osd-removal job.

    # oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Make sure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-c-64556f7659-c2ngc                                  1/1     Running     0          6h14m
   rook-ceph-mon-d-7c8b74dc4d-tt6hd                                  1/1     Running     0          4h24m
   rook-ceph-mon-e-57fb8c657-wg5f2                                   1/1     Running     0          162m

   OSD and Mon might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Identify the node and get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

5. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

6. Delete the node.

   $ oc delete node <node_name>

7. Create a new Amazon EC2 I3 machine instance with the required infrastructure. See Supported Infrastructure and Platforms.
8. Create a new OpenShift Container Platform node using the new Amazon EC2 I3 machine instance.

9. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

10. Approve all required OpenShift Container Platform CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

11. Click Compute → Nodes in the OpenShift web console. Confirm if the new node is in Ready state.

12. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

13. Add the local storage devices available in the new worker node to the OpenShift Container Storage StorageCluster.

    1. Add the new disk entries to LocalVolume CR.

       Edit LocalVolume CR. You can either remove or comment out the failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}.

       $ oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       $ oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

       You can see that in this CR the below two new devices using by-id have been added.

       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3646185e   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-3933e86    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-1-v9jp4   localblock  5h1m
       local-pv-8176b2bf   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-0-nvs68   localblock  5h1m
       local-pv-ab7cabb3   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-ac52e8a    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-0-knrgr   localblock  5h1m
       local-pv-b7e6fd37   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-0-rdm7m   localblock  5h1m
       local-pv-cb454338   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-1-h9hfm   localblock  5h1m
       local-pv-da5e3175   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-1-g97lq   localblock  5h
       ...

14. Delete each PV and OSD associated with the failed node using the following steps.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       $ osd_id_to_remove=0
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

    2. Identify the PV associated with the PVC.

       $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-8176b2bf   2328Gi      RWO            localblock     4h49m

       In this example, the associated PV is local-pv-8176b2bf.

    3. Delete the PVC which was identified in earlier steps. In this example, the PVC name is ocs-deviceset-0-0-nvs68.

       $ oc delete pvc ocs-deviceset-0-0-nvs68 -n openshift-storage

       Example output:

       persistentvolumeclaim "ocs-deviceset-0-0-nvs68" deleted

    4. Delete the PV which was identified in earlier steps. In this example, the PV name is local-pv-8176b2bf.

       $ oc delete pv local-pv-8176b2bf

       Example output:

       persistentvolume "local-pv-8176b2bf" deleted

    5. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

    6. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

       # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

       Note

       If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

       # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

    7. Delete the OSD pod deployment.

       $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage

15. Delete crashcollector pod deployment identified in an earlier step.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

16. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    $ oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          5h3m

    1. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD may take several minutes after the operator starts.

17. Delete the ocs-osd-removal job(s).

    $ oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Also, ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-a-64556f7659-c2ngc    1/1     Running     0   5h1m
   rook-ceph-mon-b-7c8b74dc4d-tt6hd    1/1     Running     0   5h1m
   rook-ceph-mon-d-57fb8c657-wg5f2     1/1     Running     0   27m

   OSDs and mon’s might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the node that needs to be replaced. Take a note of its Machine Name.

3. Get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

4. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

5. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

6. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

7. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

8. Click Compute → Machines. Search for the required machine.
9. Besides the required machine, click the Action menu (⋮) → Delete Machine.
10. Click Delete to confirm the machine deletion. A new machine is automatically created.

11. Wait for the new machine to start and transition into Running state.

    Important

    This activity may take at least 5-10 minutes or more.

12. Click Compute → Nodes in the OpenShift web console. Confirm if the new node is in Ready state.

13. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

14. Add the local storage devices available in the new worker node to the OpenShift Container Storage StorageCluster.

    1. Add the new disk entries to LocalVolume CR.

       Edit LocalVolume CR. You can either remove or comment out the failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}.

       $ oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       $ oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

       You can see that in this CR the below two new devices using by-id have been added.

       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3646185e   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-3933e86    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-1-v9jp4   localblock  5h1m
       local-pv-8176b2bf   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-0-nvs68   localblock  5h1m
       local-pv-ab7cabb3   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-ac52e8a    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-0-knrgr   localblock  5h1m
       local-pv-b7e6fd37   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-0-rdm7m   localblock  5h1m
       local-pv-cb454338   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-1-h9hfm   localblock  5h1m
       local-pv-da5e3175   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-1-g97lq   localblock  5h
       ...

15. Delete each PV and OSD associated with the failed node using the following steps.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       $ osd_id_to_remove=0
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

    2. Identify the PV associated with the PVC.

       $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-8176b2bf   2328Gi      RWO            localblock     4h49m

       In this example, the associated PV is local-pv-8176b2bf.

    3. Delete the PVC which was identified in earlier steps. In this example, the PVC name is ocs-deviceset-0-0-nvs68.

       $ oc delete pvc ocs-deviceset-0-0-nvs68 -n openshift-storage

       Example output:

       persistentvolumeclaim "ocs-deviceset-0-0-nvs68" deleted

    4. Delete the PV which was identified in earlier steps. In this example, the PV name is local-pv-8176b2bf.

       $ oc delete pv local-pv-8176b2bf

       Example output:

       persistentvolume "local-pv-8176b2bf" deleted

    5. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

    6. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

       # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

       Note

       If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

       # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

    7. Delete the OSD pod deployment.

       $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage

16. Delete crashcollector pod deployment identified in an earlier step.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

17. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    $ oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          5h3m

    1. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD may take several minutes after the operator starts.

18. Delete the ocs-osd-removal job(s).

    $ oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Also, ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-a-64556f7659-c2ngc    1/1     Running     0   5h1m
   rook-ceph-mon-b-7c8b74dc4d-tt6hd    1/1     Running     0   5h1m
   rook-ceph-mon-d-57fb8c657-wg5f2     1/1     Running     0   27m

   OSDs and mon’s might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Identify the node and get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

5. Remove the pods which are in Terminating state.

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

6. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

7. Delete the node.

   $ oc delete node <node_name>

8. Create a new Amazon EC2 I3 machine instance with the required infrastructure. See Supported Infrastructure and Platforms.
9. Create a new OpenShift Container Platform node using the new Amazon EC2 I3 machine instance.

10. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

    $ oc get csr

11. Approve all required OpenShift Container Platform CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

12. Click Compute → Nodes in the OpenShift web console. Confirm if the new node is in Ready state.

13. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

14. Add the local storage devices available in the new worker node to the OpenShift Container Storage StorageCluster.

    1. Add the new disk entries to LocalVolume CR.

       Edit LocalVolume CR. You can either remove or comment out the failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}.

       $ oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       $ oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

       You can see that in this CR the below two new devices using by-id have been added.

       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3646185e   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-3933e86    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-1-v9jp4   localblock  5h1m
       local-pv-8176b2bf   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-0-nvs68   localblock  5h1m
       local-pv-ab7cabb3   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-ac52e8a    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-0-knrgr   localblock  5h1m
       local-pv-b7e6fd37   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-0-rdm7m   localblock  5h1m
       local-pv-cb454338   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-1-h9hfm   localblock  5h1m
       local-pv-da5e3175   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-1-g97lq   localblock  5h
       ...

15. Delete each PV and OSD associated with the failed node using the following steps.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       $ osd_id_to_remove=0
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

    2. Identify the PV associated with the PVC.

       $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-8176b2bf   2328Gi      RWO            localblock     4h49m

       In this example, the associated PV is local-pv-8176b2bf.

    3. Delete the PVC which was identified in earlier steps. In this example, the PVC name is ocs-deviceset-0-0-nvs68.

       $ oc delete pvc ocs-deviceset-0-0-nvs68 -n openshift-storage

       Example output:

       persistentvolumeclaim "ocs-deviceset-0-0-nvs68" deleted

    4. Delete the PV which was identified in earlier steps. In this example, the PV name is local-pv-8176b2bf.

       $ oc delete pv local-pv-8176b2bf

       Example output:

       persistentvolume "local-pv-8176b2bf" deleted

    5. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

    6. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

       # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

       Note

       If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

       # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

    7. Delete the OSD pod deployment.

       $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage

16. Delete crashcollector pod deployment identified in an earlier step.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

17. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    $ oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          5h3m

    1. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD may take several minutes after the operator starts.

18. Delete the ocs-osd-removal job(s).

    $ oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Also, ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-a-64556f7659-c2ngc    1/1     Running     0   5h1m
   rook-ceph-mon-b-7c8b74dc4d-tt6hd    1/1     Running     0   5h1m
   rook-ceph-mon-d-57fb8c657-wg5f2     1/1     Running     0   27m

   OSDs and mon’s might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the node that needs to be replaced. Take a note of its Machine Name.

3. Get the labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

4. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

5. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

6. Mark the node as unschedulable.

   $ oc adm cordon <node_name>

7. Remove the pods which are in Terminating state.

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

8. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

9. Click Compute → Machines. Search for the required machine.
10. Besides the required machine, click the Action menu (⋮) → Delete Machine.
11. Click Delete to confirm the machine deletion. A new machine is automatically created.

12. Wait for the new machine to start and transition into Running state.

    Important

    This activity may take at least 5-10 minutes or more.

13. Click Compute → Nodes in the OpenShift web console. Confirm if the new node is in Ready state.

14. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

15. Add the local storage devices available in the new worker node to the OpenShift Container Storage StorageCluster.

    1. Add the new disk entries to LocalVolume CR.

       Edit LocalVolume CR. You can either remove or comment out the failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}.

       $ oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       $ oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7
         #   - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
             - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

       You can see that in this CR the below two new devices using by-id have been added.

       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS6F45C01D7E84FE3E9
       • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS636BC945B4ECB9AE4

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3646185e   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-3933e86    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-1-v9jp4   localblock  5h1m
       local-pv-8176b2bf   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-0-nvs68   localblock  5h1m
       local-pv-ab7cabb3   2328Gi  RWO     Delete      Available                                               localblock  9s
       local-pv-ac52e8a    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-0-knrgr   localblock  5h1m
       local-pv-b7e6fd37   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-0-rdm7m   localblock  5h1m
       local-pv-cb454338   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-1-h9hfm   localblock  5h1m
       local-pv-da5e3175   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-1-g97lq   localblock  5h
       ...

16. Delete each PV and OSD associated with the failed node using the following steps.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       $ osd_id_to_remove=0
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

    2. Identify the PV associated with the PVC.

       $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-8176b2bf   2328Gi      RWO            localblock     4h49m

       In this example, the associated PV is local-pv-8176b2bf.

    3. Delete the PVC which was identified in earlier steps. In this example, the PVC name is ocs-deviceset-0-0-nvs68.

       $ oc delete pvc ocs-deviceset-0-0-nvs68 -n openshift-storage

       Example output:

       persistentvolumeclaim "ocs-deviceset-0-0-nvs68" deleted

    4. Delete the PV which was identified in earlier steps. In this example, the PV name is local-pv-8176b2bf.

       $ oc delete pv local-pv-8176b2bf

       Example output:

       persistentvolume "local-pv-8176b2bf" deleted

    5. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

    6. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

       # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

       Note

       If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

       # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

    7. Delete the OSD pod deployment.

       $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage

17. Delete crashcollector pod deployment identified in an earlier step.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

18. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    $ oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          5h3m

    1. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD may take several minutes after the operator starts.

19. Delete the ocs-osd-removal job(s).

    $ oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Also, ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-a-64556f7659-c2ngc    1/1     Running     0   5h1m
   rook-ceph-mon-b-7c8b74dc4d-tt6hd    1/1     Running     0   5h1m
   rook-ceph-mon-d-57fb8c657-wg5f2     1/1     Running     0   27m

   OSDs and mon’s might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Identify the node and get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the node as unschedulable.

   $ oc adm cordon <node_name>

5. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

6. Delete the node.

   $ oc delete node <node_name>

7. Log in to vSphere and terminate the identified VM.
8. Create a new VM on VMware with the required infrastructure. See Supported Infrastructure and Platforms.
9. Create a new OpenShift Container Platform worker node using the new VM.

10. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

    $ oc get csr

11. Approve all required OpenShift Container Platform CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

12. Click Compute → Nodes in OpenShift Web Console, confirm if the new node is in Ready state.

13. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

14. Add the local storage devices available in these worker nodes to the OpenShift Container Storage StorageCluster.

    1. Add a new disk entry to LocalVolume CR.

       Edit LocalVolume CR and remove or comment out failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}. In this example, the new device is /dev/disk/by-id/nvme-eui.01000000010000005cd2e490020e5251.

       # oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       # oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4895e0e5251
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4ea2f0f5251
         #   - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4de2f0f5251
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e490020e5251
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3e8964d3                          1490Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-2-0-79j94   localblock                             25h
       local-pv-414755e0                          1490Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-1-0-959rp   localblock                             25h
       local-pv-b481410                           1490Gi      RWO            Delete           Available                                               localblock                             3m24s
       local-pv-d9c5cbd6                          1490Gi      RWO            Delete           Bound     openshift-storage/ocs-deviceset-0-0-nvs68   localblock

15. Delete the PV associated with the failed node.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       # osd_id_to_remove=0
       # oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

       In this example, the PVC name is ocs-deviceset-0-0-nvs68.

    2. Identify the PV associated with the PVC.

       # oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in the previous step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-d9c5cbd6   1490Gi     RWO            localblock     24h

       In this example, the associated PV is local-pv-d9c5cbd6.

    3. Delete the PVC.

       oc delete pvc <pvc-name> -n openshift-storage

    4. Delete the PV.

       # oc delete pv local-pv-d9c5cbd6

       Example output:

       persistentvolume "local-pv-d9c5cbd6" deleted

16. Remove the failed OSD from the cluster.

    # oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

17. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

    # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

    Note

    If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

    # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

18. Delete OSD pod deployment and crashcollector pod deployment.

    $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage
    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

19. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    # oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          1d20h

    1. Delete the rook-ceph-operator.

       # oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       # oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD and mon might take several minutes after the operator restarts.

20. Delete the ocs-osd-removal job.

    # oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-c-64556f7659-c2ngc                                  1/1     Running     0          6h14m
   rook-ceph-mon-d-7c8b74dc4d-tt6hd                                  1/1     Running     0          4h24m
   rook-ceph-mon-e-57fb8c657-wg5f2                                   1/1     Running     0          162m

   OSD and Mon might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.

Procedure

1. Identify the node and get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the node as unschedulable.

   $ oc adm cordon <node_name>

5. Remove the pods which are in Terminating state.

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

6. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

7. Delete the node.

   $ oc delete node <node_name>

8. Log in to vSphere and terminate the identified VM.
9. Create a new VM on VMware with the required infrastructure. See Supported Infrastructure and Platforms.
10. Create a new OpenShift Container Platform worker node using the new VM.

11. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

    $ oc get csr

12. Approve all required OpenShift Container Platform CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

13. Click Compute → Nodes in OpenShift Web Console, confirm if the new node is in Ready state.

14. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

15. Add the local storage devices available in these worker nodes to the OpenShift Container Storage StorageCluster.

    1. Add a new disk entry to LocalVolume CR.

       Edit LocalVolume CR and remove or comment out failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}. In this example, the new device is /dev/disk/by-id/nvme-eui.01000000010000005cd2e490020e5251.

       # oc get -n local-storage localvolume

       Example output:

       NAME          AGE
       local-block   25h

       # oc edit -n local-storage localvolume local-block

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4895e0e5251
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4ea2f0f5251
         #   - /dev/disk/by-id/nvme-eui.01000000010000005cd2e4de2f0f5251
             - /dev/disk/by-id/nvme-eui.01000000010000005cd2e490020e5251
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

    2. Display PVs with localblock.

       $ oc get pv | grep localblock

       Example output:

       local-pv-3e8964d3                          1490Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-2-0-79j94   localblock                             25h
       local-pv-414755e0                          1490Gi      RWO            Delete           Bound       openshift-storage/ocs-deviceset-1-0-959rp   localblock                             25h
       local-pv-b481410                           1490Gi      RWO            Delete           Available                                               localblock                             3m24s
       local-pv-d9c5cbd6                          1490Gi      RWO            Delete           Bound     openshift-storage/ocs-deviceset-0-0-nvs68   localblock

16. Delete the PV associated with the failed node.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       # osd_id_to_remove=0
       # oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

       In this example, the PVC name is ocs-deviceset-0-0-nvs68.

    2. Identify the PV associated with the PVC.

       # oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in the previous step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-d9c5cbd6   1490Gi     RWO            localblock     24h

       In this example, the associated PV is local-pv-d9c5cbd6.

    3. Delete the PVC.

       oc delete pvc <pvc-name> -n openshift-storage

    4. Delete the PV.

       # oc delete pv local-pv-d9c5cbd6

       Example output:

       persistentvolume "local-pv-d9c5cbd6" deleted

17. Remove the failed OSD from the cluster.

    # oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

18. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

    # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

    Note

    If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

    # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

19. Delete OSD pod deployment and crashcollector pod deployment.

    $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage
    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

20. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    # oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          1d20h

    1. Delete the rook-ceph-operator.

       # oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       # oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD and mon might take several minutes after the operator restarts.

21. Delete the`ocs-osd-removal` job.

    # oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-c-64556f7659-c2ngc                                  1/1     Running     0          6h14m
   rook-ceph-mon-d-7c8b74dc4d-tt6hd                                  1/1     Running     0          4h24m
   rook-ceph-mon-e-57fb8c657-wg5f2                                   1/1     Running     0          162m

   OSD and Mon might take several minutes to get to the Running state.

4. If verification steps fail, contact Red Hat Support.