Chapter 1. Initial Troubleshooting

Procedure

1. Certain problems can arise when using unsupported configurations. Ensure that your configuration is supported.

2. Do you know what Ceph component causes the problem?

   1. No. Follow Diagnosing the health of a Ceph storage cluster procedure in the Red Hat Ceph Storage Troubleshooting Guide.
   2. Ceph Monitors. See Troubleshooting Ceph Monitors section in the Red Hat Ceph Storage Troubleshooting Guide.
   3. Ceph OSDs. See Troubleshooting Ceph OSDs section in the Red Hat Ceph Storage Troubleshooting Guide.
   4. Ceph placement groups. See Troubleshooting Ceph placement groups section in the Red Hat Ceph Storage Troubleshooting Guide.
   5. Multi-site Ceph Object Gateway. See Troubleshooting a multi-site Ceph Object Gateway section in the Red Hat Ceph Storage Troubleshooting Guide.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Check the overall status of the storage cluster:

   Example

   [ceph: root@host01 /]# ceph health detail

   Copy to Clipboard Toggle word wrap

   If the command returns HEALTH_WARN or HEALTH_ERR see Understanding Ceph health for details.

3. Monitor the logs of the storage cluster:

   Example

   [ceph: root@host01 /]# ceph -W cephadm

   Copy to Clipboard Toggle word wrap

4. To capture the logs of the cluster to a file, run the following commands:

   Example

   [ceph: root@host01 /]# ceph config set global log_to_file true
   [ceph: root@host01 /]# ceph config set global mon_cluster_log_to_file true

   Copy to Clipboard Toggle word wrap

   The logs are located by default in the /var/log/ceph/CLUSTER_FSID/ directory. Check the Ceph logs for any error messages listed in Understanding Ceph logs.

5. If the logs do not include a sufficient amount of information, increase the debugging level and try to reproduce the action that failed. See Configuring logging for details.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Check the health of the Red Hat Ceph Storage cluster by running the ceph health detail command:

   Example

   [ceph: root@host01 /]# ceph health detail
   
   HEALTH_WARN 1 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
   [WRN] OSD_DOWN: 1 osds down
       osd.1 (root=default,host=host01) is down
   [WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
       osd.1 has flags noup

   Copy to Clipboard Toggle word wrap

   You can see that the storage cluster is in HEALTH_WARN status as one of the OSDs is down.

3. Mute the alert:

   Syntax

   ceph health mute HEALTH_MESSAGE

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN

   Copy to Clipboard Toggle word wrap

4. Optional: A health check mute can have a time to live (TTL) associated with it, such that the mute automatically expires after the specified period of time has elapsed. Specify the TTL as an optional duration argument in the command:

   Syntax

   ceph health mute HEALTH_MESSAGE DURATION

   Copy to Clipboard Toggle word wrap

   DURATION can be specified in s, sec, m, min, h, or hour.

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN 10m

   Copy to Clipboard Toggle word wrap

   In this example, the alert OSD_DOWN is muted for 10 minutes.

5. Verify if the Red Hat Ceph Storage cluster status has changed to HEALTH_OK:

   Example

   [ceph: root@host01 /]# ceph -s
     cluster:
       id:     81a4597a-b711-11eb-8cb8-001a4a000740
       health: HEALTH_OK
               (muted: OSD_DOWN(9m) OSD_FLAGS(9m))
   
     services:
       mon: 3 daemons, quorum host01,host02,host03 (age 33h)
       mgr: host01.pzhfuh(active, since 33h), standbys: host02.wsnngf, host03.xwzphg
       osd: 11 osds: 10 up (since 4m), 11 in (since 5d)
   
     data:
       pools:   1 pools, 1 pgs
       objects: 13 objects, 0 B
       usage:   85 MiB used, 165 GiB / 165 GiB avail
       pgs:     1 active+clean

   Copy to Clipboard Toggle word wrap

   In this example, you can see that the alert OSD_DOWN and OSD_FLAG is muted and the mute is active for nine minutes.

6. Optional: You can retain the mute even after the alert is cleared by making it sticky.

   Syntax

   ceph health mute HEALTH_MESSAGE DURATION --sticky

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN 1h --sticky

   Copy to Clipboard Toggle word wrap

7. You can remove the mute by running the following command:

   Syntax

   ceph health unmute HEALTH_MESSAGE

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph health unmute OSD_DOWN

   Copy to Clipboard Toggle word wrap

Procedure

1. Install the sos package:

   Example

   [root@host01 ~]# dnf install sos

   Copy to Clipboard Toggle word wrap

2. Run the sos report to get the system information of the storage cluster:

   Example

   [root@host01 ~]# sos report -a --all-logs

   Copy to Clipboard Toggle word wrap

   The report is saved in the /var/tmp file.

   Run the following command for specific Ceph daemon information:

   Example

   [root@host01 ~]# sos report --all-logs -e ceph_mgr,ceph_common,ceph_mon,ceph_osd,ceph_ansible,ceph_mds,ceph_rgw

   Copy to Clipboard Toggle word wrap