Chapter 5. Troubleshooting Ceph OSDs

To Troubleshoot This Problem:

1. Verify that the PG count is sufficient and increase it if needed.
2. Verify that you use CRUSH tunables optimal to the cluster version and adjust them if not.
3. Change the weight of OSDs by utilization.

4. Determine how much space is left on the disks used by OSDs.

   1. To view how much space OSDs use in general:

      [ceph: root@host01 /]# ceph osd df

   2. To view how much space OSDs use on particular nodes. Use the following command from the node containing nearfull OSDs:

      df

   3. If needed, add a new OSD node.

To Troubleshoot This Problem

1. Determine which OSD is down:

   [ceph: root@host01 /]# ceph health detail
   HEALTH_WARN 1/3 in osds are down
   osd.0 is down since epoch 23, last address 192.168.106.220:6800/11080

2. Try to restart the ceph-osd daemon. Replace the OSD_ID with the ID of the OSD that is down:

   Syntax

   systemctl restart ceph-FSID@osd.OSD_ID

   Example

   [root@host01 ~]# systemctl restart ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service

   1. If you are not able start ceph-osd, follow the steps in The ceph-osd daemon cannot start.
   2. If you are able to start the ceph-osd daemon but it is marked as down, follow the steps in The ceph-osd daemon is running but still marked as `down`.

The ceph-osd daemon cannot start

1. If you have a node containing a number of OSDs (generally, more than twelve), verify that the default maximum number of threads (PID count) is sufficient. See Increasing the PID count for details.
2. Verify that the OSD data and journal partitions are mounted properly. You can use the ceph-volume lvm list command to list all devices and volumes associated with the Ceph Storage Cluster and then manually inspect if they are mounted properly. See the mount(8) manual page for details.
3. If you got the ERROR: missing keyring, cannot use cephx for authentication error message, the OSD is a missing keyring.

4. If you got the ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-1 error message, the ceph-osd daemon cannot read the underlying file system. See the following steps for instructions on how to troubleshoot and fix this error.

   1. Check the corresponding log file to determine the cause of the failure. By default, Ceph stores log files in the /var/log/ceph/CLUSTER_FSID/ directory after the logging to files is enabled.
   2. An EIO error message indicates a failure of the underlying disk. To fix this problem replace the underlying OSD disk. See Replacing an OSD drive for details.

   3. If the log includes any other FAILED assert errors, such as the following one, open a support ticket. See Contacting Red Hat Support for service for details.

      FAILED assert(0 == "hit suicide timeout")

5. Check the dmesg output for the errors with the underlying file system or disk:

   dmesg

   1. The error -5 error message similar to the following one indicates corruption of the underlying XFS file system. For details on how to fix this problem, see the What is the meaning of "xfs_log_force: error -5 returned"? solution on the Red Hat Customer Portal.

      xfs_log_force: error -5 returned

   2. If the dmesg output includes any SCSI error error messages, see the SCSI Error Codes Solution Finder solution on the Red Hat Customer Portal to determine the best way to fix the problem.
   3. Alternatively, if you are unable to fix the underlying file system, replace the OSD drive. See Replacing an OSD drive for details.

6. If the OSD failed with a segmentation fault, such as the following one, gather the required information and open a support ticket. See Contacting Red Hat Support for service for details.

   Caught signal (Segmentation fault)

The ceph-osd is running but still marked as down

1. Check the corresponding log file to determine the cause of the failure. By default, Ceph stores log files in the /var/log/ceph/CLUSTER_FSID/ directory after the logging to files is enabled.

   1. If the log includes error messages similar to the following ones, see Flapping OSDs.

      wrongly marked me down
      heartbeat_check: no reply from osd.2 since back

   2. If you see any other errors, open a support ticket. See Contacting Red Hat Support for service for details.

To Troubleshoot This Problem

1. Check the output of the ceph health detail command again. If it includes the slow requests error message, see for details on how to troubleshoot this issue.

   ceph health detail
   HEALTH_WARN 30 requests are blocked > 32 sec; 3 osds have slow requests
   30 ops are blocked > 268435 sec
   1 ops are blocked > 268435 sec on osd.11
   1 ops are blocked > 268435 sec on osd.18
   28 ops are blocked > 268435 sec on osd.39
   3 osds have slow requests

2. Determine which OSDs are marked as down and on what nodes they reside:

   ceph osd tree | grep down

3. On the nodes containing the flapping OSDs, troubleshoot and fix any networking problems.

4. Alternatively, you can temporarily force Monitors to stop marking the OSDs as down and up by setting the noup and nodown flags:

   ceph osd set noup
   ceph osd set nodown

   Important

   Using the noup and nodown flags does not fix the root cause of the problem but only prevents OSDs from flapping. To open a support ticket, see the Contacting Red Hat Support for service section for details.

To Troubleshoot This Problem

1. Determine if the OSDs with slow or block requests share a common piece of hardware, for example, a disk drive, host, rack, or network switch.

2. If the OSDs share a disk:

   1. Use the smartmontools utility to check the health of the disk or the logs to determine any errors on the disk.

      Note

      The smartmontools utility is included in the smartmontools package.

   2. Use the iostat utility to get the I/O wait report (%iowai) on the OSD disk to determine if the disk is under heavy load.

      Note

      The iostat utility is included in the sysstat package.

3. If the OSDs share the node with another service:

   1. Check the RAM and CPU utilization
   2. Use the netstat utility to see the network statistics on the Network Interface Controllers (NICs) and troubleshoot any networking issues.
4. If the OSDs share a rack, check the network switch for the rack. For example, if you use jumbo frames, verify that the NIC in the path has jumbo frames set.
5. If you are unable to determine a common piece of hardware shared by OSDs with slow requests, or to troubleshoot and fix hardware and networking problems, open a support ticket. See Contacting Red Hat support for service for details.