Chapter 5. OpenShift Data Foundation deployed using local storage devices

Procedure

1. Remove the underlying storage device from relevant worker node.

2. Verify that relevant OSD Pod has moved to CrashLoopBackOff state.

   Identify the OSD that needs to be replaced and the OpenShift Container Platform node that has the OSD scheduled on it.

   $ oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide

   Example output:

   rook-ceph-osd-0-6d77d6c7c6-m8xj6    0/1    CrashLoopBackOff    0    24h   10.129.0.16   compute-2   <none>           <none>
   rook-ceph-osd-1-85d99fb95f-2svc7    1/1    Running             0    24h   10.128.2.24   compute-0   <none>           <none>
   rook-ceph-osd-2-6c66cdb977-jp542    1/1    Running             0    24h   10.130.0.18   compute-1   <none>           <none>

   In this example, rook-ceph-osd-0-6d77d6c7c6-m8xj6 needs to be replaced and compute-2 is the OpenShift Container platform node on which the OSD is scheduled.

3. Scale down the rook-ceph-operator deployment.

   $ oc -n openshift-storage scale deployment rook-ceph-operator --replicas=0

4. Scale down the OSD deployment for the OSD to be replaced.

   $ osd_id_to_remove=0

   $ oc scale -n openshift-storage deployment rook-ceph-osd-${osd_id_to_remove} --replicas=0

   where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

   Example output:

   deployment.extensions/rook-ceph-osd-0 scaled

5. Verify that the rook-ceph-osd pod is terminated.

   $ oc get -n openshift-storage pods -l ceph-osd-id=${osd_id_to_remove}

   Example output:

   No resources found in openshift-storage namespace.

   Important

   If the rook-ceph-osd pod is in terminating state for more than a few minutes, use the force option to delete the pod.

   $ oc delete -n openshift-storage pod rook-ceph-osd-0-6d77d6c7c6-m8xj6 --grace-period=0 --force

   Example output:

   warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
     pod "rook-ceph-osd-0-6d77d6c7c6-m8xj6" force deleted

6. Remove the old OSD from the cluster so that you can add a new OSD.

   1. Delete any old ocs-osd-removal jobs.

      $ oc delete -n openshift-storage job ocs-osd-removal-job

      Example output:

      job.batch "ocs-osd-removal-job" deleted

      Note

      The above command must reach Completed state before moving to the next steps. This can take more than ten minutes.

   2. Navigate to the openshift-storage project.

      $ oc project openshift-storage

   3. Remove the old OSD from the cluster.

      $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} FORCE_OSD_REMOVAL=false |oc create -n openshift-storage -f -

      The FORCE_OSD_REMOVAL value must be changed to “true” in clusters that only have three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed.

      Warning

      Proceeding will permanently remove the OSD from the cluster and destroy its data. Ensure osd_id_to_remove is correct. Do not process multiple IDs from the same failure domain at once. Contact Red Hat Technical Support if you need assistance.

7. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal-job pod.

   A status of Completed confirms that the OSD removal job succeeded.

   $ oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage

8. Ensure that the OSD removal is completed.

   $ oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'

   Example output:

   2022-05-10 06:50:04.501511 I | cephosd: completed removal of OSD 0

   Important

   If the ocs-osd-removal-job fails and the pod is not in the expected Completed state, check the pod logs for further debugging.

   For example:

   # oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1

9. If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSDs that are removed from the respective OpenShift Data Foundation nodes.

   1. Get the Persistent Volume Claim (PVC) names of the replaced OSDs from the logs of ocs-osd-removal-job pod.

      $ oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1  |egrep -i 'pvc|deviceset'

      Example output:

      2021-05-12 14:31:34.666000 I | cephosd: removing the OSD PVC "ocs-deviceset-xxxx-xxx-xxx-xxx"

   2. For each of the previously identified nodes, do the following:

      1. Create a debug pod and chroot to the host on the storage node.

         $ oc debug node/<node name>

         <node name>

         Is the name of the node.

         $ chroot /host

      2. Find the relevant device name based on the PVC names identified in the previous step.

         $ dmsetup ls| grep <pvc name>

         <pvc name>

         Is the name of the PVC.

         Example output:

         ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt (253:0)

      3. Remove the mapped device.

         $ cryptsetup luksClose --debug --verbose <ocs-deviceset-name>

         <ocs-deviceset-name>

         Is the name of the relevant device based on the PVC names identified in the previous step.

         Important

         If the above command gets stuck due to insufficient privileges, run the following commands:

         • Press CTRL+Z to exit the above command.

         • Find the PID of the process which was stuck.

           $ ps -ef | grep crypt

         • Terminate the process using the kill command.

           $ kill -9 <PID>

           <PID>

           Is the process ID.

         • Verify that the device name is removed.

           $ dmsetup ls

10. Find the persistent volume (PV) that need to be deleted.

    $ oc get pv -L kubernetes.io/hostname | grep <storageclass-name> | grep Released

    Example output:

    local-pv-d6bf175b           1490Gi       RWO         Delete          Released            openshift-storage/ocs-deviceset-0-data-0-6c5pw      localblock      2d22h       compute-1

11. Delete the PV.

    $ oc delete pv <pv_name>

12. Physically add a new device to the node.

13. Track the provisioning of PVs for the devices that match the deviceInclusionSpec. It can take a few minutes to provision the PVs.

    $ oc -n openshift-local-storage describe localvolumeset <lvs-name>

    Example output:

    [...]
    Status:
      Conditions:
        Last Transition Time:          2020-11-17T05:03:32Z
        Message:                       DiskMaker: Available, LocalProvisioner: Available
        Status:                        True
        Type:                          DaemonSetsAvailable
        Last Transition Time:          2020-11-17T05:03:34Z
        Message:                       Operator reconciled successfully.
        Status:                        True
        Type:                          Available
      Observed Generation:             1
      Total Provisioned Device Count: 4
    Events:
    Type    Reason      Age          From                Message
    ----    ------      ----         ----                -------
    Normal  Discovered  2m30s (x4    localvolumeset-     node.example.com -
            NewDevice   over 2m30s)  symlink-controller  found possible
                                                         matching disk,
                                                         waiting 1m to claim
    
    Normal  FoundMatch  89s (x4      localvolumeset-     node.example.com -
            ingDisk     over 89s)    symlink-controller  symlinking matching
                                                         disk

    Once the PV is provisioned, a new OSD pod is automatically created for the PV.

14. Delete the ocs-osd-removal job(s).

    $ oc delete -n openshift-storage job ocs-osd-removal-job

    Example output:

    job.batch "ocs-osd-removal-job" deleted

    Note

    When using an external key management system (KMS) with data encryption, the old OSD encryption key can be removed from the Vault server as it is now an orphan key.

15. Scale up the rook-ceph-operator deployment.

    $ oc -n openshift-storage scale deployment rook-ceph-operator --replicas=1

16. [Optional] Verify the Ceph cluster health and check if there are any crash reports associated with the previous failed OSD which has now been replaced.

    1. List all crash reports:

       $ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph crash ls -c /var/lib/rook/openshift-storage/openshift-storage.config

       Sample output:

       ID                                ENTITY  NEW
       2025-09-25T07:12:45Z_1234567890   osd.5   *
       2025-09-25T08:20:11Z_abcdef1234   mon.a   *

    2. Filter crashes for the failed OSD from the previous step. For example, if the failed OSD is osd.5, run:

       $ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph crash ls -c /var/lib/rook/openshift-storage/openshift-storage.config | grep osd.5

       Note the specif crash report from the output.

    3. Archive the specific crash. Use the crash ID from the previous step:

       $ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph crash archive <crash-id> -c /var/lib/rook/openshift-storage/openshift-storage.config

       Replace crash-id with the crash report.

    4. Verify the crash has been archived:

       $ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph crash ls -c /var/lib/rook/openshift-storage/openshift-storage.config

Verification steps

1. Verify that there is a new OSD running.

   $ oc get -n openshift-storage pods -l app=rook-ceph-osd

   Example output:

   rook-ceph-osd-0-5f7f4747d4-snshw    1/1     Running     0          4m47s
   rook-ceph-osd-1-85d99fb95f-2svc7    1/1     Running     0          1d20h
   rook-ceph-osd-2-6c66cdb977-jp542    1/1     Running     0          1d20h

   Important

   If the new OSD does not show as Running after a few minutes, restart the rook-ceph-operator pod to force a reconciliation.

   $ oc delete pod -n openshift-storage -l app=rook-ceph-operator

   Example output:

   pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

2. Verify that a new PVC is created.

   $ oc get -n openshift-storage pvc | grep <lvs-name>

   Example output:

   ocs-deviceset-0-0-c2mqb   Bound    local-pv-b481410         1490Gi     RWO            localblock                    5m
   ocs-deviceset-1-0-959rp   Bound    local-pv-414755e0        1490Gi     RWO            localblock                    1d20h
   ocs-deviceset-2-0-79j94   Bound    local-pv-3e8964d3        1490Gi     RWO            localblock                    1d20h

3. Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSDs are encrypted.

   1. Identify the nodes where the new OSD pods are running.

      $ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/<OSD-pod-name>

      <OSD-pod-name>

      Is the name of the OSD pod.

      For example:

      $ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm

      Example output:

      NODE
      compute-1

   2. For each of the nodes identified in the previous step, do the following:

      1. Create a debug pod and open a chroot environment for the selected host(s).

         $ oc debug node/<node name>

         <node name>

         Is the name of the node.

         $ chroot /host

      2. Check for the crypt keyword beside the ocs-deviceset name(s).

         $ lsblk

4. Log in to OpenShift Web Console and check the OSD status on the storage dashboard.

Procedure

1. Identify the OSD that needs to be replaced and the OpenShift Container Platform node that has the OSD scheduled on it.

   $ oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide

   Example output:

   rook-ceph-osd-0-86bf8cdc8-4nb5t   0/1     crashLoopBackOff   0   24h   10.129.2.26     worker-0     <none>       <none>
   rook-ceph-osd-1-7c99657cfb-jdzvz   1/1     Running   0          24h     10.128.2.46     worker-1     <none>       <none>
   rook-ceph-osd-2-5f9f6dfb5b-2mnw9    1/1     Running   0          24h     10.131.0.33    worker-2     <none>       <none>

   In this example, rook-ceph-osd-0-86bf8cdc8-4nb5t needs to be replaced and worker-0 is the RHOCP node on which the OSD is scheduled.

   Note

   The status of the pod is Running if the OSD you want to replace is healthy.

2. Scale down the OSD deployment for the OSD to be replaced.

   $ osd_id_to_remove=0

   $ oc scale -n openshift-storage deployment rook-ceph-osd-${osd_id_to_remove} --replicas=0

   where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

   Example output:

   deployment.extensions/rook-ceph-osd-0 scaled

3. Verify that the rook-ceph-osd pod is terminated.

   $ oc get -n openshift-storage pods -l ceph-osd-id=${osd_id_to_remove}

   Example output:

   No resources found in openshift-storage namespace.

   Important

   If the rook-ceph-osd pod is in terminating state for more than a few minutes, use the force option to delete the pod.

   $ oc delete -n openshift-storage pod rook-ceph-osd-0-86bf8cdc8-4nb5t --grace-period=0 --force

   Example output:

   warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
     pod "rook-ceph-osd-0-86bf8cdc8-4nb5t" force deleted

4. Remove the old OSD from the cluster so that you can add a new OSD.

   1. Identify the DeviceSet associated with the OSD to be replaced.

      $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

      Example output:

      ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl
          ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl

      In this example, the Persistent Volume Claim (PVC) name is ocs-deviceset-localblock-0-data-0-64xjl.

   2. Identify the Persistent Volume (PV) associated with the PVC.

      $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

      where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

      Example output:

      NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      ocs-deviceset-localblock-0-data-0-64xjl   Bound    local-pv-8137c873    256Gi      RWO     localblock     24h

      In this example, the associated PV is local-pv-8137c873.

   3. Identify the name of the device to be replaced.

      $ oc get pv local-pv-<pv-suffix> -o yaml | grep path

      where, pv-suffix is the value in the PV name identified in an earlier step.

      Example output:

      path: /mnt/local-storage/localblock/vdc

      In this example, the device name is vdc.

   4. Identify the prepare-pod associated with the OSD to be replaced.

      $ oc describe -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix> | grep Used

      where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

      Example output:

      Used By:    rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data-0-64knzkc

      In this example, the prepare-pod name is rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data-0-64knzkc.

   5. Delete any old ocs-osd-removal jobs.

      $ oc delete -n openshift-storage job ocs-osd-removal-job

      Example output:

      job.batch "ocs-osd-removal-job" deleted

      Note

      The above command must reach Completed state before moving to the next steps. This can take more than ten minutes.

   6. Change to the openshift-storage project.

      $ oc project openshift-storage

   7. Remove the old OSD from the cluster.

      $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} FORCE_OSD_REMOVAL=false |oc create -n openshift-storage -f -

      The FORCE_OSD_REMOVAL value must be changed to “true” in clusters that only have three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed.

      Warning

      Proceeding will permanently remove the OSD from the cluster and destroy its data. Ensure osd_id_to_remove is correct. Do not process multiple IDs from the same failure domain at once. Contact Red Hat Technical Support if you need assistance.

5. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal-job pod.

   A status of Completed confirms that the OSD removal job succeeded.

   $ oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage

6. Ensure that the OSD removal is completed.

   $ oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'

   Example output:

   2022-05-10 06:50:04.501511 I | cephosd: completed removal of OSD 0

   Important

   If the ocs-osd-removal-job fails and the pod is not in the expected Completed state, check the pod logs for further debugging.

   For example:

   # oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1

7. If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSDs that are removed from the respective OpenShift Data Foundation nodes.

   1. Get the PVC name(s) of the replaced OSD(s) from the logs of ocs-osd-removal-job pod.

      $ oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1  |egrep -i ‘pvc|deviceset’

      Example output:

      2021-05-12 14:31:34.666000 I | cephosd: removing the OSD PVC "ocs-deviceset-xxxx-xxx-xxx-xxx"

   2. For each of the previously identified nodes, do the following:

      1. Create a debug pod and chroot to the host on the storage node.

         $ oc debug node/<node name>

         <node name>

         Is the name of the node.

         $ chroot /host

      2. Find the relevant device name based on the PVC names identified in the previous step.

         $ dmsetup ls| grep <pvc name>

         <pvc name>

         Is the name of the PVC.

         Example output:

         ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt (253:0)

      3. Remove the mapped device.

         $ cryptsetup luksClose --debug --verbose ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt

         Important

         If the above command gets stuck due to insufficient privileges, run the following commands:

         • Press CTRL+Z to exit the above command.

         • Find the PID of the process which was stuck.

           $ ps -ef | grep crypt

         • Terminate the process using the kill command.

           $ kill -9 <PID>

           <PID>

           Is the process ID.

         • Verify that the device name is removed.

           $ dmsetup ls

8. Find the PV that need to be deleted.

   $ oc get pv -L kubernetes.io/hostname | grep localblock | grep Released

   Example output:

   local-pv-d6bf175b           1490Gi       RWO         Delete          Released            openshift-storage/ocs-deviceset-0-data-0-6c5pw      localblock      2d22h       compute-1

9. Delete the PV.

   $ oc delete pv <pv-name>

   <pv-name>

   Is the name of the PV.

10. Replace the old device and use the new device to create a new OpenShift Container Platform PV.

    1. Log in to the OpenShift Container Platform node with the device to be replaced. In this example, the OpenShift Container Platform node is worker-0.

       $ oc debug node/worker-0

       Example output:

       Starting pod/worker-0-debug ...
       To use host binaries, run `chroot /host`
       Pod IP: 192.168.88.21
       If you don't see a command prompt, try pressing enter.
       # chroot /host

    2. Record the /dev/disk that is to be replaced using the device name, vdc, identified earlier.

       # ls -alh /mnt/local-storage/localblock

       Example output:

       total 0
       drwxr-xr-x. 2 root root 17 Nov  18 15:23 .
       drwxr-xr-x. 3 root root 24 Nov  18 15:23 ..
       lrwxrwxrwx. 1 root root  8 Nov  18 15:23 vdc -> /dev/vdc

    3. Find the name of the LocalVolume CR, and remove or comment out the device /dev/disk that is to be replaced.

       $ oc get -n openshift-local-storage localvolume

       Example output:

       NAME          AGE
       localblock   25h

       # oc edit -n openshift-local-storage localvolume localblock

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
          #   - /dev/vdc
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

11. Log in to the OpenShift Container Platform node with the device to be replaced and remove the old symlink.

    $ oc debug node/worker-0

    Example output:

    Starting pod/worker-0-debug ...
    To use host binaries, run `chroot /host`
    Pod IP: 192.168.88.21
    If you don't see a command prompt, try pressing enter.
    # chroot /host

    1. Identify the old symlink for the device name to be replaced. In this example, the device name is vdc.

       # ls -alh /mnt/local-storage/localblock

       Example output:

       total 0
       drwxr-xr-x. 2 root root 17 Nov  18 15:23 .
       drwxr-xr-x. 3 root root 24 Nov  18 15:23 ..
       lrwxrwxrwx. 1 root root  8 Nov  18 15:23 vdc -> /dev/vdc

    2. Remove the symlink.

       # rm /mnt/local-storage/localblock/vdc

    3. Verify that the symlink is removed.

       # ls -alh /mnt/local-storage/localblock

       Example output:

       total 0
       drwxr-xr-x. 2 root root 6 Nov 18 17:11 .
       drwxr-xr-x. 3 root root 24 Nov 18 15:23 ..

12. Replace the old device with the new device.

13. Log back into the correct OpenShift Container Platform node and identify the device name for the new drive. The device name must change unless you are resetting the same device.

    # lsblk

    Example output:

    NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    vda                          252:0    0   40G  0 disk
    |-vda1                       252:1    0    4M  0 part
    |-vda2                       252:2    0  384M  0 part /boot
    `-vda4                       252:4    0 39.6G  0 part
      `-coreos-luks-root-nocrypt 253:0    0 39.6G  0 dm   /sysroot
    vdb                          252:16   0  512B  1 disk
    vdd                          252:32   0  256G  0 disk

    In this example, the new device name is vdd.

14. After the new /dev/disk is available, you can add a new disk entry to the LocalVolume CR.

    1. Edit the LocalVolume CR and add the new /dev/disk.

       In this example, the new device is /dev/vdd.

       # oc edit -n openshift-local-storage localvolume localblock

       Example output:

       [...]
           storageClassDevices:
           - devicePaths:
           #  - /dev/vdc
             - /dev/vdd
             storageClassName: localblock
             volumeMode: Block
       [...]

       Make sure to save the changes after editing the CR.

15. Verify that there is a new PV in Available state and of the correct size.

    $ oc get pv | grep 256Gi

    Example output:

    local-pv-1e31f771   256Gi   RWO    Delete  Bound  openshift-storage/ocs-deviceset-localblock-2-data-0-6xhkf   localblock    24h
    local-pv-ec7f2b80   256Gi   RWO    Delete  Bound  openshift-storage/ocs-deviceset-localblock-1-data-0-hr2fx   localblock    24h
    local-pv-8137c873   256Gi   RWO    Delete  Available                                                          localblock    32m

16. Create a new OSD for the new device.

    Deploy the new OSD. You need to restart the rook-ceph-operator to force operator reconciliation.

    1. Identify the name of the rook-ceph-operator.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-85f6494db4-sg62v   1/1     Running   0          1d20h

    2. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-85f6494db4-sg62v

       Example output:

       pod "rook-ceph-operator-85f6494db4-sg62v" deleted

       In this example, the rook-ceph-operator pod name is rook-ceph-operator-85f6494db4-sg62v.

    3. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-85f6494db4-wx9xx   1/1     Running   0          50s

       Creation of the new OSD may take several minutes after the operator restarts.

17. Delete the ocs-osd-removal job(s).

    $ oc delete -n openshift-storage job ocs-osd-removal-job

    Example output:

    job.batch "ocs-osd-removal-job" deleted

Verfication steps

1. Verify that there is a new OSD running.

   $ oc get -n openshift-storage pods -l app=rook-ceph-osd

   Example output:

   rook-ceph-osd-0-76d8fb97f9-mn8qz   1/1     Running   0          23m
   rook-ceph-osd-1-7c99657cfb-jdzvz   1/1     Running   1          25h
   rook-ceph-osd-2-5f9f6dfb5b-2mnw9   1/1     Running   0          25h

2. Verify that a new PVC is created.

   $ oc get -n openshift-storage pvc | grep localblock

   Example output:

   ocs-deviceset-localblock-0-data-0-q4q6b   Bound    local-pv-8137c873       256Gi     RWO         localblock         10m
   ocs-deviceset-localblock-1-data-0-hr2fx   Bound    local-pv-ec7f2b80       256Gi     RWO         localblock         1d20h
   ocs-deviceset-localblock-2-data-0-6xhkf   Bound    local-pv-1e31f771       256Gi     RWO         localblock         1d20h

3. Optional: If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

   1. Identify the nodes where the new OSD pods are running.

      $ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/<OSD-pod-name>

      <OSD-pod-name>

      Is the name of the OSD pod.

      For example:

      $ oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm

      Example output:

      NODE
      compute-1

   2. For each of the previously identified nodes, do the following:

      1. Create a debug pod and open a chroot environment for the selected host(s).

         $ oc debug node/<node name>

         <node name>

         Is the name of the node.

         $ chroot /host

      2. Check for the crypt keyword beside the ocs-deviceset name(s).

         $ lsblk

4. Log in to OpenShift Web Console and check the status card in the OpenShift Data Foundation dashboard under Storage section.