Chapter 9. Replacing failed storage nodes on Red Hat Virtualization platform

Procedure

1. Identify the node and get labels on the node to be replaced.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and OSDs that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

5. Drain the node.

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Note

   If the failed node is not connected to the network, remove the pods running on it by using the command:

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'
   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

6. Remove the failed node after making a note of the device by-id.

   1. List the nodes.

      $ oc get nodes

      Example output:

      NAME          STATUS   ROLES    AGE     VERSION
      rhvworker01    Ready    worker   6h45m   v1.16.2
      rhvworker02    Ready    worker   6h45m   v1.16.2
      rhvworker03    Ready    worker   6h45m   v1.16.2

   2. Find the unique by-id device name for each of the existing nodes.

      $ oc debug node/<Nodename>

      Example output:

      NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
      sda                            8:0    0   120G  0 disk
      |-sda1                         8:1    0   384M  0 part /boot
      |-sda2                         8:2    0   127M  0 part /boot/efi
      |-sda3                         8:3    0     1M  0 part
      `-sda4                         8:4    0 119.5G  0 part
         -coreos-luks-root-nocrypt 253:0    0 119.5G  0 dm   /sysroot
      sdb                            8:16   0   500G  0 disk
      sr0                           11:0    1   374K  0 rom
      sr1                           11:1    1  1024M  0 rom
      rbd0                         252:0    0    30G  0 disk /var/lib/kubelet/pods/1fe52fb8-ca70-40e1-ac74-28802baf3b80/volumes/kubernetes.io~csi/pvc-705982e5-0a7d-42f5-916f-69340541d52d/mount
      rbd1                         252:16   0    30G  0 disk /var/lib/kubelet/pods/b48b22ba-532f-4d4f-9f58-debfd03c3221/volumes/kubernetes.io~csi/pvc-bb7cdb1c-03fe-4b31-97e3-3c8fa9758f16/mount
      sh-4.4# ls -l /dev/disk/by-id/ | grep sdb
      lrwxrwxrwx. 1 root root  9 Oct  7 14:58 scsi-0QEMU_QEMU_HARDDISK_a01cab90-d3cd-4822-a7eb-a5553d4b619a -> ../../sdb
      lrwxrwxrwx. 1 root root  9 Oct  7 14:58 scsi-SQEMU_QEMU_HARDDISK_a01cab90-d3cd-4822-a7eb-a5553d4b619a -> ../../sdb
      sh-4.4# exit
      exit
      sh-4.2# exit
      exit
      Removing debug pod ...

      Repeat this to identify the device ID for all the existing nodes.

   3. Delete the machine corresponding to the failed node. A new node is automatically added.

      1. Click Compute Machines. Search for the required machine.
      2. Beside the required machine, click the Action menu (⋮) Delete Machine.
      3. Click Delete to confirm the machine deletion. A new machine is automatically created.

      4. Wait for the new machine to start and transition into Running state.

         Important

         This activity may take at least 5-10 minutes or more.

   4. [Optional]: If the failed Red Hat Virtualization instance is not removed automatically, terminate the instance from the console.
7. Click Compute Nodes in the OpenShift web console. Confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) Edit Labels.
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

9. Add the local storage devices available in the new worker node to the OpenShift Container Storage StorageCluster.

   1. Add the new disk entries to LocalVolume CR.

      Edit LocalVolume CR. You can either remove or comment out the failed device /dev/disk/by-id/{id} and add the new /dev/disk/by-id/{id}. For information about identifying the device by-id, see Finding available storage devices.

      $ oc get -n local-storage localvolume

      Example output:

      NAME          AGE
      local-block   25h

      $ oc edit -n local-storage localvolume local-block

      Example output:

      [...]
          storageClassDevices:
          - devicePaths:
            - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_901abab1-b164-4bb4-b9a8-6bdbce2de23b
            - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_a01cab90-d3cd-4822-a7eb-a5553d4b619a
            # SQEMU_QEMU_HARDDISK_a01cab90-d3cd-4822-a7be-a5553d4b619a
            - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_c7388957-3e09-4135-bac1-af7551467d0b
      
            storageClassName: localblock
            volumeMode: Block
      [...]

      Make sure to save the changes after editing the CR.

      You can see that in this CR the following new device using by-id has been added.

      SQEMU_QEMU_HARDDISK_a01cab90-d3cd-4822-a7eb-a5553d4b619a

   2. Display PVs with localblock.

      $ oc get pv | grep localblock

      Example output:

      local-pv-3646185e   2328Gi  RWO     Delete      Available                                               localblock  9s
      local-pv-3933e86    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-1-v9jp4   localblock  5h1m
      local-pv-8176b2bf   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-0-nvs68   localblock  5h1m
      local-pv-ab7cabb3   2328Gi  RWO     Delete      Available                                               localblock  9s
      local-pv-ac52e8a    2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-0-knrgr   localblock  5h1m
      local-pv-b7e6fd37   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-2-0-rdm7m   localblock  5h1m
      local-pv-cb454338   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-0-1-h9hfm   localblock  5h1m
      local-pv-da5e3175   2328Gi  RWO     Delete      Bound       openshift-storage/ocs-deviceset-1-1-g97lq   localblock  5h
      ...

10. Delete each PV and OSD associated with the failed node using the following steps.

    1. Identify the DeviceSet associated with the OSD to be replaced.

       $ osd_id_to_remove=0
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68
           ceph.rook.io/pvc: ocs-deviceset-0-0-nvs68

    2. Identify the PV associated with the PVC.

       $ oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>

       where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.

       Example output:

       NAME                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
       ocs-deviceset-0-0-nvs68   Bound   local-pv-8176b2bf   2328Gi      RWO            localblock     4h49m

       In this example, the associated PV is local-pv-8176b2bf.

    3. Delete the PVC which was identified in earlier steps. In this example, the PVC name is ocs-deviceset-0-0-nvs68.

       $ oc delete pvc ocs-deviceset-0-0-nvs68 -n openshift-storage

       Example output:

       persistentvolumeclaim "ocs-deviceset-0-0-nvs68" deleted

    4. Delete the PV which was identified in earlier steps. In this example, the PV name is local-pv-8176b2bf.

       $ oc delete pv local-pv-8176b2bf

       Example output:

       persistentvolume "local-pv-8176b2bf" deleted

    5. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_ID=${osd_id_to_remove} | oc create -f -

    6. Verify that the OSD is removed successfully by checking the status of the ocs-osd-removal pod. A status of Completed confirms that the OSD removal job succeeded.

       # oc get pod -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage

       Note

       If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

       # oc logs -l job-name=ocs-osd-removal-${osd_id_to_remove} -n openshift-storage --tail=-1

    7. Delete the OSD pod deployment.

       $ oc delete deployment rook-ceph-osd-${osd_id_to_remove} -n openshift-storage

11. Delete crashcollector pod deployment identified in an earlier step.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<old_node_name> -n openshift-storage

12. Deploy the new OSD by restarting the rook-ceph-operator to force operator reconciliation.

    $ oc get -n openshift-storage pod -l app=rook-ceph-operator

    Example output:

    NAME                                  READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-6f74fb5bff-2d982   1/1     Running   0          5h3m

    1. Delete the rook-ceph-operator.

       $ oc delete -n openshift-storage pod rook-ceph-operator-6f74fb5bff-2d982

       Example output:

       pod "rook-ceph-operator-6f74fb5bff-2d982" deleted

    2. Verify that the rook-ceph-operator pod is restarted.

       $ oc get -n openshift-storage pod -l app=rook-ceph-operator

       Example output:

       NAME                                  READY   STATUS    RESTARTS   AGE
       rook-ceph-operator-6f74fb5bff-7mvrq   1/1     Running   0          66s

       Creation of the new OSD may take several minutes after the operator starts.

13. Delete the ocs-osd-removal job(s).

    $ oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify that all other required OpenShift Container Storage pods are in Running state.

   Also, ensure that the new incremental mon is created and is in the Running state.

   $ oc get pod -n openshift-storage | grep mon

   Example output:

   rook-ceph-mon-a-64556f7659-c2ngc    1/1     Running     0   5h1m
   rook-ceph-mon-b-7c8b74dc4d-tt6hd    1/1     Running     0   5h1m
   rook-ceph-mon-d-57fb8c657-wg5f2     1/1     Running     0   27m

   OSDs and mons might take several minutes to get to the Running state.

4. If the verification steps fail, contact Red Hat Support.