Troubleshooting Guide

Procedure

1. Certain problems can arise when using unsupported configurations. Ensure that your configuration is supported.

2. Do you know what Ceph component causes the problem?

   1. No. Follow Diagnosing the health of a Ceph storage cluster procedure in the Red Hat Ceph Storage Troubleshooting Guide.
   2. Ceph Monitors. See Troubleshooting Ceph Monitors section in the Red Hat Ceph Storage Troubleshooting Guide.
   3. Ceph OSDs. See Troubleshooting Ceph OSDs section in the Red Hat Ceph Storage Troubleshooting Guide.
   4. Ceph placement groups. See Troubleshooting Ceph placement groups section in the Red Hat Ceph Storage Troubleshooting Guide.
   5. Multi-site Ceph Object Gateway. See Troubleshooting a multi-site Ceph Object Gateway section in the Red Hat Ceph Storage Troubleshooting Guide.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Check the overall status of the storage cluster:

   Example

   [ceph: root@host01 /]# ceph health detail

   If the command returns HEALTH_WARN or HEALTH_ERR see Understanding Ceph health for details.

3. Monitor the logs of the storage cluster:

   Example

   [ceph: root@host01 /]# ceph -W cephadm

4. To capture the logs of the cluster to a file, run the following commands:

   Example

   [ceph: root@host01 /]# ceph config set global log_to_file true
   [ceph: root@host01 /]# ceph config set global mon_cluster_log_to_file true

   The logs are located by default in the /var/log/ceph/CLUSTER_FSID/ directory. Check the Ceph logs for any error messages listed in Understanding Ceph logs.

5. If the logs do not include a sufficient amount of information, increase the debugging level and try to reproduce the action that failed. See Configuring logging for details.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Check the health of the Red Hat Ceph Storage cluster by running the ceph health detail command:

   Example

   [ceph: root@host01 /]# ceph health detail
   
   HEALTH_WARN 1 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
   [WRN] OSD_DOWN: 1 osds down
       osd.1 (root=default,host=host01) is down
   [WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
       osd.1 has flags noup

   You can see that the storage cluster is in HEALTH_WARN status as one of the OSDs is down.

3. Mute the alert:

   Syntax

   ceph health mute HEALTH_MESSAGE

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN

4. Optional: A health check mute can have a time to live (TTL) associated with it, such that the mute automatically expires after the specified period of time has elapsed. Specify the TTL as an optional duration argument in the command:

   Syntax

   ceph health mute HEALTH_MESSAGE DURATION

   DURATION can be specified in s, sec, m, min, h, or hour.

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN 10m

   In this example, the alert OSD_DOWN is muted for 10 minutes.

5. Verify if the Red Hat Ceph Storage cluster status has changed to HEALTH_OK:

   Example

   [ceph: root@host01 /]# ceph -s
     cluster:
       id:     81a4597a-b711-11eb-8cb8-001a4a000740
       health: HEALTH_OK
               (muted: OSD_DOWN(9m) OSD_FLAGS(9m))
   
     services:
       mon: 3 daemons, quorum host01,host02,host03 (age 33h)
       mgr: host01.pzhfuh(active, since 33h), standbys: host02.wsnngf, host03.xwzphg
       osd: 11 osds: 10 up (since 4m), 11 in (since 5d)
   
     data:
       pools:   1 pools, 1 pgs
       objects: 13 objects, 0 B
       usage:   85 MiB used, 165 GiB / 165 GiB avail
       pgs:     1 active+clean

   In this example, you can see that the alert OSD_DOWN and OSD_FLAG is muted and the mute is active for nine minutes.

6. Optional: You can retain the mute even after the alert is cleared by making it sticky.

   Syntax

   ceph health mute HEALTH_MESSAGE DURATION --sticky

   Example

   [ceph: root@host01 /]# ceph health mute OSD_DOWN 1h --sticky

7. You can remove the mute by running the following command:

   Syntax

   ceph health unmute HEALTH_MESSAGE

   Example

   [ceph: root@host01 /]# ceph health unmute OSD_DOWN

Procedure

1. Install the sos package:

   Example

   [root@host01 ~]# dnf install sos

2. Run the sos report to get the system information of the storage cluster:

   Example

   [root@host01 ~]# sosreport -a --all-logs

   The report is saved in the /var/tmp file.

   Run the following command for specific Ceph daemon information:

   Example

   [root@host01 ~]# sos report --all-logs -e ceph_mgr,ceph_common,ceph_mon,ceph_osd,ceph_ansible,ceph_mds,ceph_rgw

Procedure

1. To activate the Ceph debugging output, dout(), at runtime:

   ceph tell TYPE.ID injectargs --debug-SUBSYSTEM VALUE [--NAME VALUE]

2. Replace:

   • TYPE with the type of Ceph daemons (osd, mon, or mds)
   • ID with a specific ID of the Ceph daemon. Alternatively, use * to apply the runtime setting to all daemons of a particular type.
   • SUBSYSTEM with a specific subsystem.

   • VALUE with a number from 1 to 20, where 1 is terse and 20 is verbose.

     For example, to set the log level for the OSD subsystem on the OSD named osd.0 to 0 and the memory level to 5:

     # ceph tell osd.0 injectargs --debug-osd 0/5

1. Log in to the host with a running Ceph daemon, for example, ceph-osd or ceph-mon.

2. Display the configuration:

   Syntax

   ceph daemon NAME config show | less

   Example

   [ceph: root@host01 /]# ceph daemon osd.0 config show | less

Procedure

1. To activate Ceph debugging output, dout() at boot time, add the debugging settings to the Ceph configuration file.

   1. For subsystems common to each daemon, add the settings under the [global] section.

   2. For subsystems for particular daemons, add the settings under a daemon section, such as [mon], [osd], or [mds].

      Example

      [global]
              debug_ms = 1/5
      
      [mon]
              debug_mon = 20
              debug_paxos = 1/5
              debug_auth = 2
      
      [osd]
              debug_osd = 1/5
              debug_monc = 5/20
      
      [mds]
              debug_mds = 1

Procedure

1. Add the size setting after the rotation frequency to the log rotation file:

   rotate 7
   weekly
   size SIZE
   compress
   sharedscripts

   For example, to rotate a log file when it reaches 500 MB:

   rotate 7
   weekly
   size 500 MB
   compress
   sharedscripts
   size 500M

   Note

   The size value can be expressed as '500 MB' or '500M'.

2. Open the crontab editor:

   [root@mon ~]# crontab -e

3. Add an entry to check the /etc/logrotate.d/ceph-<fsid> file. For example, to instruct Cron to check /etc/logrotate.d/ceph-<fsid> every 30 minutes:

   30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph-d3bb5396-c404-11ee-9e65-002590fc2a2e >/dev/null 2>&1

Procedure

1. Find where the logs are located:

   Syntax

   logrotate -f

   Example

   [root@host01 ~]# logrotate -f
   /etc/logrotate.d/ceph-12ab345c-1a2b-11ed-b736-fa163e4f6220

2. List the logs within the specified location:

   Syntax

   ll LOG_LOCATION

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    -rw-r--r--. 1 ceph ceph    412 Sep 28 09:26 opslog.log.1.gz

3. List the current buckets:

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd ls

4. Create a bucket:

   Syntax

   /usr/local/bin/s3cmd mb s3://NEW_BUCKET_NAME

   Example

   [root@host01 ~]# /usr/local/bin/s3cmd mb s3://bucket1
   Bucket `s3://bucket1` created

5. List the current logs:

   Syntax

   ll LOG_LOCATION

   Example

   [root@host01 ~]# ll /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220
    total 852
    ...
    -rw-r--r--. 1 ceph ceph    920 Jun 29 02:17 opslog.log
    -rw-r--r--. 1 ceph ceph    412 Jun 28 09:26 opslog.log.1.gz

6. Collect the logs:

   Syntax

   tail -f LOG_LOCATION/opslog.log

   Example

   [root@host01 ~]# tail -f /var/log/ceph/12ab345c-1a2b-11ed-b736-fa163e4f6220/opslog.log
   
   {"bucket":"","time":"2022-09-29T06:17:03.133488Z","time_local":"2022-09-
   29T06:17:03.133488+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"list_buckets","uri":"GET /
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":232,
   "bytes_received":0,"object_size":0,"total_time":9,"user_agent":"","referrer":
   "","trans_id":"tx00000c80881a9acd2952a-006335385f-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}
   
   {"bucket":"cn1","time":"2022-09-29T06:17:10.521156Z","time_local":"2022-09-
   29T06:17:10.521156+0000","remote_addr":"10.0.211.66","user":"test1",
   "operation":"create_bucket","uri":"PUT /cn1/
   HTTP/1.1","http_status":"200","error_code":"","bytes_sent":0,
   "bytes_received":0,"object_size":0,"total_time":106,"user_agent":"",
   "referrer":"","trans_id":"tx0000058d60c593632c017-0063353866-175e5-primary",
   "authentication_type":"Local","access_key_id":"1234","temp_url":false}

Procedure

1. Installing the net-tools and telnet packages can help when troubleshooting network issues that can occur in a Ceph storage cluster:

   Example

   [root@host01 ~]# dnf install net-tools
   [root@host01 ~]# dnf install telnet

2. Log into the cephadm shell and verify that the public_network parameters in the Ceph configuration file include the correct values:

   Example

   [ceph: root@host01 /]# cat /etc/ceph/ceph.conf
   # minimal ceph.conf for 57bddb48-ee04-11eb-9962-001a4a000672
   [global]
   	fsid = 57bddb48-ee04-11eb-9962-001a4a000672
   	mon_host = [v2:10.74.249.26:3300/0,v1:10.74.249.26:6789/0] [v2:10.74.249.163:3300/0,v1:10.74.249.163:6789/0] [v2:10.74.254.129:3300/0,v1:10.74.254.129:6789/0]
   [mon.host01]
   public network = 10.74.248.0/21

3. Exit the shell and verify that the network interfaces are up:

   Example

   [root@host01 ~]# ip link list
   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether 00:1a:4a:00:06:72 brd ff:ff:ff:ff:ff:ff

4. Verify that the Ceph nodes are able to reach each other using their short host names. Verify this on each node in the storage cluster:

   Syntax

   ping SHORT_HOST_NAME

   Example

   [root@host01 ~]# ping host02

5. If you use a firewall, ensure that Ceph nodes are able to reach each other on their appropriate ports. The firewall-cmd and telnet tools can validate the port status, and if the port is open respectively:

   Syntax

   firewall-cmd --info-zone=ZONE
   telnet IP_ADDRESS PORT

   Example

   [root@host01 ~]# firewall-cmd --info-zone=public
   public (active)
     target: default
     icmp-block-inversion: no
     interfaces: ens3
     sources:
     services: ceph ceph-mon cockpit dhcpv6-client ssh
     ports: 9283/tcp 8443/tcp 9093/tcp 9094/tcp 3000/tcp 9100/tcp 9095/tcp
     protocols:
     masquerade: no
     forward-ports:
     source-ports:
     icmp-blocks:
     rich rules:
   
   [root@host01 ~]# telnet 192.168.0.22 9100

6. Verify that there are no errors on the interface counters. Verify that the network connectivity between nodes has expected latency, and that there is no packet loss.

   1. Using the ethtool command:

      Syntax

      ethtool -S INTERFACE

      Example

      [root@host01 ~]# ethtool -S ens3 | grep errors
      NIC statistics:
           rx_fcs_errors: 0
           rx_align_errors: 0
           rx_frame_too_long_errors: 0
           rx_in_length_errors: 0
           rx_out_length_errors: 0
           tx_mac_errors: 0
           tx_carrier_sense_errors: 0
           tx_errors: 0
           rx_errors: 0

   2. Using the ifconfig command:

      Example

      [root@host01 ~]# ifconfig
      ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
              inet 10.74.249.26  netmask 255.255.248.0  broadcast 10.74.255.255
              inet6 fe80::21a:4aff:fe00:672  prefixlen 64  scopeid 0x20<link>
              inet6 2620:52:0:4af8:21a:4aff:fe00:672  prefixlen 64  scopeid 0x0<global>
              ether 00:1a:4a:00:06:72  txqueuelen 1000  (Ethernet)
              RX packets 150549316  bytes 56759897541 (52.8 GiB)
              RX errors 0  dropped 176924  overruns 0  frame 0
              TX packets 55584046  bytes 62111365424 (57.8 GiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
      
      lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
              inet 127.0.0.1  netmask 255.0.0.0
              inet6 ::1  prefixlen 128  scopeid 0x10<host>
              loop  txqueuelen 1000  (Local Loopback)
              RX packets 9373290  bytes 16044697815 (14.9 GiB)
              RX errors 0  dropped 0  overruns 0  frame 0
              TX packets 9373290  bytes 16044697815 (14.9 GiB)
              TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

   3. Using the netstat command:

      Example

      [root@host01 ~]# netstat -ai
      Kernel Interface table
      Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
      ens3             1500 311847720      0 364903 0      114341918      0      0      0 BMRU
      lo              65536 19577001      0      0 0      19577001      0      0      0 LRU

7. For performance issues, in addition to the latency checks and to verify the network bandwidth between all nodes of the storage cluster, use the iperf3 tool. The iperf3 tool does a simple point-to-point network bandwidth test between a server and a client.

   1. Install the iperf3 package on the Red Hat Ceph Storage nodes you want to check the bandwidth:

      Example

      [root@host01 ~]# dnf install iperf3

   2. On a Red Hat Ceph Storage node, start the iperf3 server:

      Example

      [root@host01 ~]# iperf3 -s
      -----------------------------------------------------------
      Server listening on 5201
      -----------------------------------------------------------

      Note

      The default port is 5201, but can be set using the -P command argument.

   3. On a different Red Hat Ceph Storage node, start the iperf3 client:

      Example

      [root@host02 ~]# iperf3 -c mon
      Connecting to host mon, port 5201
      [  4] local xx.x.xxx.xx port 52270 connected to xx.x.xxx.xx port 5201
      [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
      [  4]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    409 KBytes
      [  4]   1.00-2.00   sec   113 MBytes   945 Mbits/sec    0    409 KBytes
      [  4]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0    454 KBytes
      [  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    471 KBytes
      [  4]   4.00-5.00   sec   112 MBytes   940 Mbits/sec    0    471 KBytes
      [  4]   5.00-6.00   sec   113 MBytes   945 Mbits/sec    0    471 KBytes
      [  4]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    488 KBytes
      [  4]   7.00-8.00   sec   113 MBytes   947 Mbits/sec    0    520 KBytes
      [  4]   8.00-9.00   sec   112 MBytes   939 Mbits/sec    0    520 KBytes
      [  4]   9.00-10.00  sec   112 MBytes   939 Mbits/sec    0    520 KBytes
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bandwidth       Retr
      [  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
      [  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver
      
      iperf Done.

      This output shows a network bandwidth of 1.1 Gbits/second between the Red Hat Ceph Storage nodes, along with no retransmissions (Retr) during the test.

      Red Hat recommends you validate the network bandwidth between all the nodes in the storage cluster.

8. Ensure that all nodes have the same network interconnect speed. Slower attached nodes might slow down the faster connected ones. Also, ensure that the inter switch links can handle the aggregated bandwidth of the attached nodes:

   Syntax

   ethtool INTERFACE

   Example

   [root@host01 ~]# ethtool ens3
   Settings for ens3:
   Supported ports: [ TP ]
   Supported link modes:   10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Supported pause frame use: No
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes:  10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                        100baseT/Half 100baseT/Full
                                        1000baseT/Full
   Link partner advertised pause frame use: Symmetric
   Link partner advertised auto-negotiation: Yes
   Link partner advertised FEC modes: Not reported
   Speed: 1000Mb/s 1
   Duplex: Full 2
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   Auto-negotiation: on
   MDI-X: off
   Supports Wake-on: g
   Wake-on: d
   Current message level: 0x000000ff (255)
          drv probe link timer ifdown ifup rx_err tx_err
   Link detected: yes 3

Procedure

1. Verify that the chronyd daemon is running on the Ceph Monitor hosts:

   Example

   [root@mon ~]# systemctl status chronyd

2. If chronyd is not running, enable and start it:

   Example

   [root@mon ~]# systemctl enable chronyd
   [root@mon ~]# systemctl start chronyd

3. Ensure that chronyd is synchronizing the clocks correctly:

   Example

   [root@mon ~]# chronyc sources
   [root@mon ~]# chronyc sourcestats
   [root@mon ~]# chronyc tracking

Procedure

1. If the remaining Ceph Monitors are able to form a quorum, get the Ceph Monitor map by using the ceph mon getmap command:

   Example

   [ceph: root@host01 /]# ceph mon getmap -o /tmp/monmap

2. If the remaining Ceph Monitors are not able to form the quorum and you have at least one Ceph Monitor with a correct Ceph Monitor map, copy it from that Ceph Monitor:

   1. Stop the Ceph Monitor which you want to copy the Ceph Monitor map from:

      Syntax

      systemctl stop ceph-FSID@DAEMON_NAME

      Example

      [root@mon ~]# systemctl stop ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

   2. Copy the Ceph Monitor map:

      Syntax

      ceph-mon -i ID --extract-monmap /tmp/monmap

      Replace ID with the ID of the Ceph Monitor which you want to copy the Ceph Monitor map from:

      Example

      [ceph: root@host01 /]#  ceph-mon -i mon.a  --extract-monmap /tmp/monmap

3. Stop the Ceph Monitor with the corrupted or outdated Ceph Monitor map:

   Syntax

   systemctl stop ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl stop ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

4. Inject the Ceph Monitor map:

   Syntax

   ceph-mon -i ID --inject-monmap /tmp/monmap

   Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map:

   Example

   [root@mon ~]# ceph-mon -i mon.host01 --inject-monmap /tmp/monmap

5. Start the Ceph Monitor:

   Syntax

   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

   If you copied the Ceph Monitor map from another Ceph Monitor, start that Ceph Monitor, too:

   Syntax

   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

Procedure

1. From the Monitor host, remove the Monitor store by default located at /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME:

   rm -rf /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME

   Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor store of a Monitor running on host1 from a cluster called remote:

   [root@mon ~]# rm -rf /var/lib/ceph/mon/remote-host1

2. Remove the Monitor from the Monitor map (monmap):

   ceph mon remove SHORT_HOST_NAME --cluster CLUSTER_NAME

   Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor running on host1 from a cluster called remote:

   [ceph: root@host01 /]# ceph mon remove host01 --cluster remote

3. Troubleshoot and fix any problems related to the underlying file system or hardware of the Monitor host.

Procedure

1. To compact the Monitor store when the ceph-mon daemon is running:

   Syntax

   ceph tell mon.HOST_NAME compact

2. Replace HOST_NAME with the short host name of the host where the ceph-mon is running. Use the hostname -s command when unsure.

   Example

   [ceph: root@host01 /]# ceph tell mon.host01 compact

3. Add the following parameter to the Ceph configuration under the [mon] section:

   [mon]
   mon_compact_on_start = true

4. Restart the ceph-mon daemon:

   Syntax

   systemctl restart ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl restart ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

5. Ensure that Monitors have formed a quorum:

   [ceph: root@host01 /]# ceph mon stat

6. Repeat these steps on other Monitors if needed.

   Note

   Before you start, ensure that you have the ceph-test package installed.

7. Verify that the ceph-mon daemon with the large store is not running. Stop the daemon if needed.

   Syntax

   systemctl status ceph-FSID@DAEMON_NAME
   systemctl stop ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service
   [root@mon ~]# systemctl stop ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

8. Compact the Monitor store:

   Syntax

   ceph-monstore-tool /var/lib/ceph/CLUSTER_FSID/mon.HOST_NAME compact

   Replace HOST_NAME with a short host name of the Monitor host.

   Example

   [ceph: root@host01 /]# ceph-monstore-tool /var/lib/ceph/b404c440-9e4c-11ec-a28a-001a4a0001df/mon.host01 compact

9. Start ceph-mon again:

   Syntax

   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

Procedure

1. To resolve this situation, for each host running ceph-mgr daemons, open ports 6800-7300.

   Example

   [root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp
   [root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp --permanent

2. Restart the ceph-mgr daemons.

Procedure

1. Log in to the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the noout flag before stopping the OSD:

   Example

   [ceph: root@host01 /]# ceph osd set noout

3. When you finish troubleshooting or maintenance, unset the noout flag to start rebalancing:

   Example

   [ceph: root@host01 /]# ceph osd unset noout

1. Remove the OSD from the Ceph cluster. For details, see the Removing an OSD from the Ceph Cluster procedure.
2. Replace the drive. For details, see Replacing the physical drive section.
3. Add the OSD to the cluster. For details, see Adding an OSD to the Ceph Cluster procedure.

Removing an OSD from the Ceph Cluster

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Determine which OSD is down.

   Example

   [ceph: root@host01 /]# ceph osd tree | grep -i down
   ID  CLASS  WEIGHT   TYPE NAME           STATUS  REWEIGHT  PRI-AFF
    0   hdd 0.00999        osd.0     down  1.00000          1.00000

3. Mark the OSD as out for the cluster to rebalance and copy its data to other OSDs.

   Syntax

   ceph osd out OSD_ID.

   Example

   [ceph: root@host01 /]# ceph osd out osd.0
   marked out osd.0.

   Note

   If the OSD is down, Ceph marks it as out automatically after 600 seconds when it does not receive any heartbeat packet from the OSD based on the mon_osd_down_out_interval parameter. When this happens, other OSDs with copies of the failed OSD data begin backfilling to ensure that the required number of copies exists within the cluster. While the cluster is backfilling, the cluster will be in a degraded state.

4. Ensure that the failed OSD is backfilling.

   Example

   [ceph: root@host01 /]# ceph -w | grep backfill
   2022-05-02 04:48:03.403872 mon.0 [INF] pgmap v10293282: 431 pgs: 1 active+undersized+degraded+remapped+backfilling, 28 active+undersized+degraded, 49 active+undersized+degraded+remapped+wait_backfill, 59 stale+active+clean, 294 active+clean; 72347 MB data, 101302 MB used, 1624 GB / 1722 GB avail; 227 kB/s rd, 1358 B/s wr, 12 op/s; 10626/35917 objects degraded (29.585%); 6757/35917 objects misplaced (18.813%); 63500 kB/s, 15 objects/s recovering
   2022-05-02 04:48:04.414397 mon.0 [INF] pgmap v10293283: 431 pgs: 2 active+undersized+degraded+remapped+backfilling, 75 active+undersized+degraded+remapped+wait_backfill, 59 stale+active+clean, 295 active+clean; 72347 MB data, 101398 MB used, 1623 GB / 1722 GB avail; 969 kB/s rd, 6778 B/s wr, 32 op/s; 10626/35917 objects degraded (29.585%); 10580/35917 objects misplaced (29.457%); 125 MB/s, 31 objects/s recovering
   2022-05-02 04:48:00.380063 osd.1 [INF] 0.6f starting backfill to osd.0 from (0'0,0'0] MAX to 2521'166639
   2022-05-02 04:48:00.380139 osd.1 [INF] 0.48 starting backfill to osd.0 from (0'0,0'0] MAX to 2513'43079
   2022-05-02 04:48:00.380260 osd.1 [INF] 0.d starting backfill to osd.0 from (0'0,0'0] MAX to 2513'136847
   2022-05-02 04:48:00.380849 osd.1 [INF] 0.71 starting backfill to osd.0 from (0'0,0'0] MAX to 2331'28496
   2022-05-02 04:48:00.381027 osd.1 [INF] 0.51 starting backfill to osd.0 from (0'0,0'0] MAX to 2513'87544

   You should see the placement group states change from active+clean to active, some degraded objects, and finally active+clean when migration completes.

5. Stop the OSD:

   Syntax

   ceph orch daemon stop OSD_ID

   Example

   [ceph: root@host01 /]# ceph orch daemon stop osd.0

6. Remove the OSD from the storage cluster:

   Syntax

   ceph orch osd rm OSD_ID --replace

   Example

   [ceph: root@host01 /]# ceph orch osd rm 0 --replace

   The OSD_ID is preserved.

1. If the drive is hot-swappable, replace the failed drive with a new one.
2. If the drive is not hot-swappable and the node contains multiple OSDs, you might have to shut down the whole node and replace the physical drive. Consider preventing the cluster from backfilling. See the Stopping and Starting Rebalancing chapter in the Red Hat Ceph Storage Troubleshooting Guide for details.
3. When the drive appears under the /dev/ directory, make a note of the drive path.
4. If you want to add the OSD manually, find the OSD drive and format the disk.

Adding an OSD to the Ceph Cluster

1. Once the new drive is inserted, you can use the following options to deploy the OSDs:

   • The OSDs are deployed automatically by the Ceph Orchestrator if the --unmanaged parameter is not set.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

   • Deploy the OSDs on all the available devices with the unmanaged parameter set to true.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices --unmanaged=true

   • Deploy the OSDs on specific devices and hosts.

     Example

     [ceph: root@host01 /]# ceph orch daemon add osd host02:/dev/sdb

2. Ensure that the CRUSH hierarchy is accurate:

   Example

   [ceph: root@host01 /]# ceph osd tree

Procedure

1. Log in to the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Determine the current value of full_ratio, by default it is set to 0.95:

   [ceph: root@host01 /]# ceph osd dump | grep -i full
   full_ratio 0.95

3. Temporarily increase the value of set-full-ratio to 0.97:

   [ceph: root@host01 /]# ceph osd set-full-ratio 0.97

   Important

   Red Hat strongly recommends to not set the set-full-ratio to a value higher than 0.97. Setting this parameter to a higher value makes the recovery process harder. As a consequence, you might not be able to recover full OSDs at all.

4. Verify that you successfully set the parameter to 0.97:

   [ceph: root@host01 /]# ceph osd dump | grep -i full
   full_ratio 0.97

5. Monitor the cluster state:

   [ceph: root@host01 /]# ceph -w

   As soon as the cluster changes its state from full to nearfull, delete any unnecessary data.

6. Set the value of full_ratio back to 0.95:

   [ceph: root@host01 /]# ceph osd set-full-ratio 0.95

7. Verify that you successfully set the parameter to 0.95:

   [ceph: root@host01 /]# ceph osd dump | grep -i full
   full_ratio 0.95

Procedure

1. Check the sync status between the sites:

   Example

   [ceph: host04 /]# radosgw-admin sync status
             realm d713eec8-6ec4-4f71-9eaf-379be18e551b (india)
         zonegroup ccf9e0b2-df95-4e0a-8933-3b17b64c52b7 (shared)
              zone 04daab24-5bbd-4c17-9cf5-b1981fd7ff79 (primary)
      current time 2022-09-15T06:53:52Z
   zonegroup features enabled: resharding
     metadata sync no sync (zone is master)
         data sync source: 596319d2-4ffe-4977-ace1-8dd1790db9fb (secondary)
                           syncing
                           full sync: 0/128 shards
                           incremental sync: 128/128 shards
                           data is caught up with source

2. Synchronize data from the secondary zone:

   Example

   [ceph: root@host04 /]# radosgw-admin data sync init --source-zone primary

3. Restart all the Ceph Object Gateway daemons at the site:

   Example

   [ceph: root@host04 /]# ceph orch restart rgw.myrgw

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the stuck PGs:

   Example

   [ceph: root@host01 /]# ceph pg dump_stuck inactive
   [ceph: root@host01 /]# ceph pg dump_stuck unclean
   [ceph: root@host01 /]# ceph pg dump_stuck stale

Procedure

1. Reduce the impact of data redistribution and recovery on individual OSDs and OSD hosts:

   1. Lower the value of the osd max backfills, osd_recovery_max_active, and osd_recovery_op_priority parameters:

      [ceph: root@host01 /]# ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1 --osd_recovery_op_priority 1'

   2. Disable the shallow and deep scrubbing:

      [ceph: root@host01 /]# ceph osd set noscrub
      [ceph: root@host01 /]# ceph osd set nodeep-scrub

2. Use the Ceph Placement Groups (PGs) per Pool Calculator to calculate the optimal value of the pg_num and pgp_num parameters.

3. Increase the pg_num value in small increments until you reach the desired value.

   1. Determine the starting increment value. Use a very low value that is a power of two, and increase it when you determine the impact on the cluster. The optimal value depends on the pool size, OSD count, and client I/O load.

   2. Increment the pg_num value:

      Syntax

      ceph osd pool set POOL pg_num VALUE

      Specify the pool name and the new value, for example:

      Example

      [ceph: root@host01 /]# ceph osd pool set data pg_num 4

   3. Monitor the status of the cluster:

      Example

      [ceph: root@host01 /]# ceph -s

      The PGs state will change from creating to active+clean. Wait until all PGs are in the active+clean state.

4. Increase the pgp_num value in small increments until you reach the desired value:

   1. Determine the starting increment value. Use a very low value that is a power of two, and increase it when you determine the impact on the cluster. The optimal value depends on the pool size, OSD count, and client I/O load.

   2. Increment the pgp_num value:

      Syntax

      ceph osd pool set POOL pgp_num VALUE

      Specify the pool name and the new value, for example:

      [ceph: root@host01 /]# ceph osd pool set data pgp_num 4

   3. Monitor the status of the cluster:

      [ceph: root@host01 /]# ceph -s

      The PGs state will change through peering, wait_backfill, backfilling, recover, and others. Wait until all PGs are in the active+clean state.

5. Repeat the previous steps for all pools with insufficient PG count.

6. Set osd max backfills, osd_recovery_max_active, and osd_recovery_op_priority to their default values:

   [ceph: root@host01 /]# ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 3 --osd_recovery_op_priority 3'

7. Enable the shallow and deep scrubbing:

   [ceph: root@host01 /]# ceph osd unset noscrub
   [ceph: root@host01 /]# ceph osd unset nodeep-scrub

Procedure

1. Disable automated monitor deployment:

   Example

   [ceph: root@host01 /]# ceph orch apply mon --unmanaged
   
   Scheduled mon update…

2. View the monitors in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 5 daemons, quorum host01, host02, host04, host05 (age 30s), out of quorum: host07

3. Set the monitor in quorum as a new tiebreaker:

   Syntax

   ceph mon set_new_tiebreaker NEW_HOST

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host02

   Important

   You get an error message if the monitor is in the same location as existing non-tiebreaker monitors:

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host02
   
   Error EINVAL: mon.host02 has location DC1, which matches mons host02 on the datacenter dividing bucket for stretch mode.

   If that happens, change the location of the monitor:

   Syntax

   ceph mon set_location HOST datacenter=DATACENTER

   Example

   [ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC3

4. Remove the failed tiebreaker monitor:

   Syntax

   ceph orch daemon rm FAILED_TIEBREAKER_MONITOR --force

   Example

   [ceph: root@host01 /]# ceph orch daemon rm mon.host07 --force
   
   Removed mon.host07 from host 'host07'

5. Once the monitor is removed from the host, redeploy the monitor:

   Syntax

   ceph mon add HOST IP_ADDRESS datacenter=DATACENTER
   ceph orch daemon add mon HOST

   Example

   [ceph: root@host01 /]# ceph mon add host07 213.222.226.50 datacenter=DC1
   [ceph: root@host01 /]# ceph orch daemon add mon host07

6. Ensure there are five monitors in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 5 daemons, quorum host01, host02, host04, host05, host07 (age 15s)

7. Verify that everything is configured properly:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host02
   disallowed_leaders host02
   0: [v2:132.224.169.63:3300/0,v1:132.224.169.63:6789/0] mon.host02; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host07; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host03; crush_location {datacenter=DC2}
   dumped monmap epoch 19

8. Redeploy the monitors:

   Syntax

   ceph orch apply mon --placement="HOST_1, HOST_2, HOST_3, HOST_4, HOST_5”

   Example

   [ceph: root@host01 /]# ceph orch apply mon --placement="host01, host02, host04, host05, host07"
   
   Scheduled mon update...

Procedure

1. Add a new monitor to the cluster:

   1. Manually add the crush_location to the new monitor:

      Syntax

      ceph mon add NEW_HOST IP_ADDRESS datacenter=DATACENTER

      Example

      [ceph: root@host01 /]# ceph mon add host06 213.222.226.50 datacenter=DC3
      
      adding mon.host06 at [v2:213.222.226.50:3300/0,v1:213.222.226.50:6789/0]

      Note

      The new monitor has to be in a different location than existing non-tiebreaker monitors.

   2. Disable automated monitor deployment:

      Example

      [ceph: root@host01 /]# ceph orch apply mon --unmanaged
      
      Scheduled mon update…

   3. Deploy the new monitor:

      Syntax

      ceph orch daemon add mon NEW_HOST

      Example

      [ceph: root@host01 /]# ceph orch daemon add mon host06

2. Ensure there are 6 monitors, from which 5 are in quorum:

   Example

   [ceph: root@host01 /]# ceph -s
   
   mon: 6 daemons, quorum host01, host02, host04, host05, host06 (age 30s), out of quorum: host07

3. Set the new monitor as a new tiebreaker:

   Syntax

   ceph mon set_new_tiebreaker NEW_HOST

   Example

   [ceph: root@host01 /]# ceph mon set_new_tiebreaker host06

4. Remove the failed tiebreaker monitor:

   Syntax

   ceph orch daemon rm FAILED_TIEBREAKER_MONITOR --force

   Example

   [ceph: root@host01 /]# ceph orch daemon rm mon.host07 --force
   
   Removed mon.host07 from host 'host07'

5. Verify that everything is configured properly:

   Example

   [ceph: root@host01 /]# ceph mon dump
   
   epoch 19
   fsid 1234ab78-1234-11ed-b1b1-de456ef0a89d
   last_changed 2023-01-17T04:12:05.709475+0000
   created 2023-01-16T05:47:25.631684+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   stretch_mode_enabled 1
   tiebreaker_mon host06
   disallowed_leaders host06
   0: [v2:213.222.226.50:3300/0,v1:213.222.226.50:6789/0] mon.host06; crush_location {datacenter=DC3}
   1: [v2:220.141.179.34:3300/0,v1:220.141.179.34:6789/0] mon.host04; crush_location {datacenter=DC2}
   2: [v2:40.90.220.224:3300/0,v1:40.90.220.224:6789/0] mon.host01; crush_location {datacenter=DC1}
   3: [v2:60.140.141.144:3300/0,v1:60.140.141.144:6789/0] mon.host02; crush_location {datacenter=DC1}
   4: [v2:186.184.61.92:3300/0,v1:186.184.61.92:6789/0] mon.host05; crush_location {datacenter=DC2}
   dumped monmap epoch 19

6. Redeploy the monitors:

   Syntax

   ceph orch apply mon --placement="HOST_1, HOST_2, HOST_3, HOST_4, HOST_5”

   Example

   [ceph: root@host01 /]# ceph orch apply mon --placement="host01, host02, host04, host05, host06"
   
   Scheduled mon update…

Procedure

1. Force the stretch cluster into the recovery mode:

   Example

   [ceph: root@host01 /]#  ceph osd force_recovery_stretch_mode --yes-i-really-mean-it

   Note

   The recovery state puts the cluster in the HEALTH_WARN state.

2. When in recovery mode, the cluster should go back into normal stretch mode after the placement groups are healthy. If that does not happen, you can force the stretch cluster into the healthy mode:

   Example

   [ceph: root@host01 /]#  ceph osd force_healthy_stretch_mode --yes-i-really-mean-it

   Note

   You can also run this command if you want to force the cross-data-center peering early and you are willing to risk data downtime, or you have verified separately that all the placement groups can peer, even if they are not fully recovered.

   You might also wish to invoke the healthy mode to remove the HEALTH_WARN state, which is generated by the recovery state.

   Note

   The force_recovery_stretch_mode and force_recovery_healthy_mode commands should not be necessary, as they are included in the process of managing unanticipated situations.

Procedure

1. Open a support ticket on the Red Hat Customer Portal.
2. Ideally, attach an sosreport to the ticket. See the What is a sosreport and how to create one in Red Hat Enterprise Linux? solution for details.
3. If the Ceph daemons fail with a segmentation fault, consider generating a human-readable core dump file. See Generating readable core dump files for details.

Prerequisites

1. Install the debuginfo packages if they are not installed already.

   1. Enable the following repositories to install the required debuginfo packages.

      Example

      [root@host01 ~]# subscription-manager repos --enable=rhceph-6-tools-for-rhel-9-x86_64-rpms
      [root@host01 ~]# yum --enable=rhceph-6-tools-for-rhel-9-x86_64-debug-rpms

      Once the repository is enabled, you can install the debug info packages that you need from this list of supported packages:

      ceph-base-debuginfo
      ceph-common-debuginfo
      ceph-debugsource
      ceph-fuse-debuginfo
      ceph-immutable-object-cache-debuginfo
      ceph-mds-debuginfo
      ceph-mgr-debuginfo
      ceph-mon-debuginfo
      ceph-osd-debuginfo
      ceph-radosgw-debuginfo
      cephfs-mirror-debuginfo

2. Ensure that the gdb package is installed and if it is not, install it:

   Example

   [root@host01 ~]# dnf install gdb

   • Section 10.2.1, “Generating readable core dump files in containerized deployments”