Chapter 2. Handling a disk failure

Procedure

1. Check storage cluster health:

   # ceph health

2. Identify the OSD location in the CRUSH hierarchy:

   # ceph osd tree | grep -i down

3. On the OSD node, try to start the OSD:

   # systemctl start ceph-osd@$OSD_ID

   If the command indicates that the OSD is already running, there might be a heartbeat or networking issue. If you cannot restart the OSD, then the drive might have failed.

   Note

   If the OSD is down, then the OSD will eventually get marked out. This is normal behavior for Ceph Storage. When the OSD gets marked out, other OSDs with copies of the failed OSD’s data will begin backfilling to ensure that the required number of copies exist within the storage cluster. While the storage cluster is backfilling, the cluster will be in a degraded state.

4. For containerized deployments of Ceph, try to start the OSD container by referencing the drive associated with the OSD:

   # systemctl start ceph-osd@$OSD_DRIVE

   If the command indicates that the OSD is already running, there might be a heartbeat or networking issue. If you cannot restart the OSD, then the drive might have failed.

   Note

   The drive associated with the OSD can be determined by Mapping a container OSD ID to a drive.

5. Check the failed OSD’s mount point:

   Note

   For containerized deployments of Ceph, if the OSD is down the container will be down and the OSD drive will be unmounted, so you cannot run df to check its mount point. Use another method to determine if the OSD drive has failed. For example, run smartctl on the drive from the container node.

   # df -h

   If you cannot restart the OSD, you can check the mount point. If the mount point no longer appears, then you can try remounting the OSD drive and restarting the OSD. If you cannot restore the mount point, then you might have a failed OSD drive.

   Using the smartctl utility cab help determine if the drive is healthy. For example:

   # yum install smartmontools
   # smartctl -H /dev/$DRIVE

   If the drive has failed, you will need to replace it.

6. Stop the OSD process:

   # systemctl stop ceph-osd@$OSD_ID

   1. If using FileStore, then flush the journal to disk:

      # ceph osd -i $$OSD_ID --flush-journal

7. For containerized deployments of Ceph, stop the OSD container by referencing the drive associated with the OSD:

   # systemctl stop ceph-osd@$OSD_DRIVE

8. Remove the OSD out of the storage cluster:

   # ceph osd out $OSD_ID

9. Ensure the failed OSD is backfilling:

   # ceph -w

10. Remove the OSD from the CRUSH Map:

    # ceph osd crush remove osd.$OSD_ID

    Note

    This step is only needed, if you are permanently removing the OSD and not redeploying it.

11. Remove the OSD’s authentication keys:

    # ceph auth del osd.$OSD_ID

12. Verify that the keys for the OSD are not listed:

    # ceph auth list

13. Remove the OSD from the storage cluster:

    # ceph osd rm osd.$OSD_ID

14. Unmount the failed drive path:

    Note

    For containerized deployments of Ceph, if the OSD is down the container will be down and the OSD drive will be unmounted. In this case there is nothing to unmount and this step can be skipped.

    # umount /var/lib/ceph/osd/$CLUSTER_NAME-$OSD_ID

15. Replace the physical drive. Refer to the hardware vendor’s documentation for the node. If the drive is hot swappable, simply replace the failed drive with a new drive. If the drive is NOT hot swappable and the node contains multiple OSDs, you MIGHT need to bring the node down to replace the physical drive. If you need to bring the node down temporarily, you might set the cluster to noout to prevent backfilling:

    # ceph osd set noout

    Once you replace the drive and you bring the node and its OSDs back online, remove the noout setting:

    # ceph osd unset noout

    Allow the new drive to appear under the /dev/ directory and make a note of the drive path before proceeding further.

16. Find the OSD drive and format the disk.

17. Recreate the OSD:

    1. Using Ansible.
    2. Using the command-line interface.

18. Check the CRUSH hierarchy to ensure it is accurate:

    # ceph osd tree

    If you are not satisfied with the location of the OSD in the CRUSH hierarchy, you might move it with the move command:

    # ceph osd crush move $BUCKET_TO_MOVE $BUCKET_TYPE=$PARENT_BUCKET

19. Verify the OSD is online.

Procedure

1. Destroy the OSD:

   ceph osd destroy $OSD_ID --yes-i-really-mean-it

   Example

   $ ceph osd destroy 1 --yes-i-really-mean-it

2. Optionally, if the replacement disk was used previously, then you need to zap the disk:

   ceph-volume lvm zap $DEVICE

   Example

   $ ceph-volume lvm zap /dev/sdb

3. Create the new OSD with the existing OSD ID:

   ceph-volume lvm create --osd-id $OSD_ID --data $DEVICE

   Example

   $ ceph-volume lvm create --osd-id 1 --data /dev/sdb