Troubleshooting Guide

Procedure

1. Certain problems can arise when using unsupported configurations. Ensure that your configuration is supported.

2. Do you know what Ceph component causes the problem?

   1. No. Follow Diagnosing the health of a Ceph storage cluster procedure in the Red Hat Ceph Storage Troubleshooting Guide.
   2. Ceph Monitors. See Troubleshooting Ceph Monitors section in the Red Hat Ceph Storage Troubleshooting Guide.
   3. Ceph OSDs. See Troubleshooting Ceph OSDs section in the Red Hat Ceph Storage Troubleshooting Guide.
   4. Ceph placement groups. See Troubleshooting Ceph placement groups section in the Red Hat Ceph Storage Troubleshooting Guide.
   5. Multi-site Ceph Object Gateway. See Troubleshooting a multi-site Ceph Object Gateway section in the Red Hat Ceph Storage Troubleshooting Guide.

Procedure

1. Log into the Ansible administration node as an ansible user.

   Note

   Ensure the node has adequate space to collect the logs from the hosts.

2. Navigate to /usr/share/ceph-ansible directory:

   Example

   [ansible@admin ~]# cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

3. Run the Ansible playbook to gather the logs:

   Example

   ansible@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/gather-ceph-logs.yml -i hosts

   Copy to Clipboard Toggle word wrap

   The logs are stored in the /tmp directory of the Ansible node.

Procedure

1. To activate the Ceph debugging output, dout(), at runtime:

   ceph tell TYPE.ID injectargs --debug-SUBSYSTEM VALUE [--NAME VALUE]

   Copy to Clipboard Toggle word wrap

2. Replace:

   • TYPE with the type of Ceph daemons (osd, mon, or mds)
   • ID with a specific ID of the Ceph daemon. Alternatively, use * to apply the runtime setting to all daemons of a particular type.
   • SUBSYSTEM with a specific subsystem.

   • VALUE with a number from 1 to 20, where 1 is terse and 20 is verbose.

     For example, to set the log level for the OSD subsystem on the OSD named osd.0 to 0 and the memory level to 5:

     # ceph tell osd.0 injectargs --debug-osd 0/5

     Copy to Clipboard Toggle word wrap

1. Log in to the host with a running Ceph daemon, for example ceph-osd or ceph-mon.

2. Display the configuration:

   ceph daemon NAME config show | less

   Copy to Clipboard Toggle word wrap

   Example

   # ceph daemon osd.0 config show | less

   Copy to Clipboard Toggle word wrap

Procedure

1. To activate Ceph debugging output, dout() at boot time, add the debugging settings to the Ceph configuration file.

   1. For subsystems common to each daemon, add the settings under the [global] section.

   2. For subsystems for particular daemons, add the settings under a daemon section, such as [mon], [osd], or [mds].

      Example

      [global]
              debug_ms = 1/5
      
      [mon]
              debug_mon = 20
              debug_paxos = 1/5
              debug_auth = 2
      
      [osd]
              debug_osd = 1/5
              debug_monc = 5/20
      
      [mds]
              debug_mds = 1

      Copy to Clipboard Toggle word wrap

Procedure

1. Add the size setting after the rotation frequency to the log rotation file:

   rotate 7
   weekly
   size SIZE
   compress
   sharedscripts

   Copy to Clipboard Toggle word wrap

   For example, to rotate a log file when it reaches 500 MB:

   rotate 7
   weekly
   size 500 MB
   compress
   sharedscripts
   size 500M

   Copy to Clipboard Toggle word wrap

2. Open the crontab editor:

   [root@mon ~]# crontab -e

   Copy to Clipboard Toggle word wrap

3. Add an entry to check the /etc/logrotate.d/ceph file. For example, to instruct Cron to check /etc/logrotate.d/ceph every 30 minutes:

   30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1

   Copy to Clipboard Toggle word wrap

Procedure

1. Installing the net-tools and telnet packages can help when troubleshooting network issues that can occur in a Ceph storage cluster:

   Red Hat Enterprise Linux 7

   [root@mon ~]# yum install net-tools
   [root@mon ~]# yum install telnet

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 8

   [root@mon ~]# dnf install net-tools
   [root@mon ~]# dnf install telnet

   Copy to Clipboard Toggle word wrap

2. Verify that the cluster_network and public_network parameters in the Ceph configuration file include the correct values:

   Example

   [root@mon ~]# cat /etc/ceph/ceph.conf | grep net
   cluster_network = 192.168.1.0/24
   public_network = 192.168.0.0/24

   Copy to Clipboard Toggle word wrap

3. Verify that the network interfaces are up:

   Example

   [root@mon ~]# ip link list
   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   2: enp22s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether 40:f2:e9:b8:a0:48 brd ff:ff:ff:ff:ff:ff

   Copy to Clipboard Toggle word wrap

4. Verify that the Ceph nodes are able to reach each other using their short host names. Verify this on each node in the storage cluster:

   Syntax

   ping SHORT_HOST_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ping osd01

   Copy to Clipboard Toggle word wrap

5. If you use a firewall, ensure that Ceph nodes are able to reach other on their appropriate ports. The firewall-cmd and telnet tools can validate the port status, and if the port is open respectively:

   Syntax

   firewall-cmd --info-zone=ZONE
   telnet IP_ADDRESS PORT

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# firewall-cmd --info-zone=public
   public (active)
     target: default
     icmp-block-inversion: no
     interfaces: enp1s0
     sources: 192.168.0.0/24
     services: ceph ceph-mon cockpit dhcpv6-client ssh
     ports: 9100/tcp 8443/tcp 9283/tcp 3000/tcp 9092/tcp 9093/tcp 9094/tcp 9094/udp
     protocols:
     masquerade: no
     forward-ports:
     source-ports:
     icmp-blocks:
     rich rules:
   
   [root@mon ~]# telnet 192.168.0.22 9100

   Copy to Clipboard Toggle word wrap

6. Verify that there are no errors on the interface counters. Verify that the network connectivity between nodes has expected latency, and that there is no packet loss.

   1. Using the ethtool command:

      Syntax

      ethtool -S INTERFACE

      Copy to Clipboard Toggle word wrap

      Example

      [root@mon ~]# ethtool -S enp22s0f0 | grep errors
      NIC statistics:
           rx_fcs_errors: 0
           rx_align_errors: 0
           rx_frame_too_long_errors: 0
           rx_in_length_errors: 0
           rx_out_length_errors: 0
           tx_mac_errors: 0
           tx_carrier_sense_errors: 0
           tx_errors: 0
           rx_errors: 0

      Copy to Clipboard Toggle word wrap

   2. Using the ifconfig command:

      Example

      [root@mon ~]# ifconfig
      enp22s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet 10.8.222.13  netmask 255.255.254.0  broadcast 10.8.223.255
      inet6 2620:52:0:8de:42f2:e9ff:feb8:a048  prefixlen 64  scopeid 0x0<global>
      inet6 fe80::42f2:e9ff:feb8:a048  prefixlen 64  scopeid 0x20<link>
      ether 40:f2:e9:b8:a0:48  txqueuelen 1000  (Ethernet)
      RX packets 4219130  bytes 2704255777 (2.5 GiB)
      RX errors 0  dropped 0  overruns 0  frame 0 

      1

      
      TX packets 1418329  bytes 738664259 (704.4 MiB)
      TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 

      2

      
      device interrupt 16

      Copy to Clipboard Toggle word wrap

   3. Using the netstat command:

      Example

      [root@mon ~]# netstat -ai
      Kernel Interface table
      Iface          MTU   RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR Flg
      docker0       1500       0      0      0 0           0      0      0      0 BMU
      eno2          1500       0      0      0 0           0      0      0      0 BMU
      eno3          1500       0      0      0 0           0      0      0      0 BMU
      eno4          1500       0      0      0 0           0      0      0      0 BMU
      enp0s20u13u5  1500  253277      0      0 0           0      0      0      0 BMRU
      enp22s0f0     9000  234160      0      0 0      432326      0      0      0 BMRU 

      1

      
      lo           65536   10366      0      0 0       10366      0      0      0 LRU

      Copy to Clipboard Toggle word wrap

7. For performance issues, in addition to the latency checks and to verify the network bandwidth between all nodes of the storage cluster, use the iperf3 tool. The iperf3 tool does a simple point-to-point network bandwidth test between a server and a client.

   1. Install the iperf3 package on the Red Hat Ceph Storage nodes you want to check the bandwidth:

      Red Hat Enterprise Linux 7

      [root@mon ~]# yum install iperf3

      Copy to Clipboard Toggle word wrap

      Red Hat Enterprise Linux 8

      [root@mon ~]# dnf install iperf3

      Copy to Clipboard Toggle word wrap

   2. On a Red Hat Ceph Storage node, start the iperf3 server:

      Example

      [root@mon ~]# iperf3 -s
      -----------------------------------------------------------
      Server listening on 5201
      -----------------------------------------------------------

      Copy to Clipboard Toggle word wrap

      Note

      The default port is 5201, but can be set using the -P command argument.

   3. On a different Red Hat Ceph Storage node, start the iperf3 client:

      Example

      [root@osd ~]# iperf3 -c mon
      Connecting to host mon, port 5201
      [  4] local xx.x.xxx.xx port 52270 connected to xx.x.xxx.xx port 5201
      [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
      [  4]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    409 KBytes
      [  4]   1.00-2.00   sec   113 MBytes   945 Mbits/sec    0    409 KBytes
      [  4]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0    454 KBytes
      [  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    471 KBytes
      [  4]   4.00-5.00   sec   112 MBytes   940 Mbits/sec    0    471 KBytes
      [  4]   5.00-6.00   sec   113 MBytes   945 Mbits/sec    0    471 KBytes
      [  4]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    488 KBytes
      [  4]   7.00-8.00   sec   113 MBytes   947 Mbits/sec    0    520 KBytes
      [  4]   8.00-9.00   sec   112 MBytes   939 Mbits/sec    0    520 KBytes
      [  4]   9.00-10.00  sec   112 MBytes   939 Mbits/sec    0    520 KBytes
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bandwidth       Retr
      [  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
      [  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver
      
      iperf Done.

      Copy to Clipboard Toggle word wrap

      This output shows a network bandwidth of 1.1 Gbits/second between the Red Hat Ceph Storage nodes, along with no retransmissions (Retr) during the test.

      Red Hat recommends you validate the network bandwidth between all the nodes in the storage cluster.

8. Ensure that all nodes have the same network interconnect speed. Slower attached nodes might slow down the faster connected ones. Also, ensure that the inter switch links can handle the aggregated bandwidth of the attached nodes:

   Syntax

   ethtool INTERFACE

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ethtool enp22s0f0
   Settings for enp22s0f0:
   Supported ports: [ TP ]
   Supported link modes:   10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Supported pause frame use: No
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes:  10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                        100baseT/Half 100baseT/Full
                                        1000baseT/Full
   Link partner advertised pause frame use: Symmetric
   Link partner advertised auto-negotiation: Yes
   Link partner advertised FEC modes: Not reported
   Speed: 1000Mb/s 

   1

   
   Duplex: Full 

   2

   
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   Auto-negotiation: on
   MDI-X: off
   Supports Wake-on: g
   Wake-on: d
   Current message level: 0x000000ff (255)
          drv probe link timer ifdown ifup rx_err tx_err
   Link detected: yes 

   3

   Copy to Clipboard Toggle word wrap

Procedure

1. Verify that the chronyd daemon is running on the Ceph Monitor hosts:

   Example

   [root@mon ~]# systemctl status chronyd

   Copy to Clipboard Toggle word wrap

2. If chronyd is not running, enable and start it:

   Example

   [root@mon ~]# systemctl enable chronyd
   [root@mon ~]# systemctl start chronyd

   Copy to Clipboard Toggle word wrap

3. Ensure that chronyd is synchronizing the clocks correctly:

   Example

   [root@mon ~]# chronyc sources
   [root@mon ~]# chronyc sourcestats
   [root@mon ~]# chronyc tracking

   Copy to Clipboard Toggle word wrap

Procedure

1. Verify that the ntpd daemon is running on the Ceph Monitor hosts:

   Example

   [root@mon ~]# systemctl status ntpd

   Copy to Clipboard Toggle word wrap

2. If ntpd is not running, enable and start it:

   Example

   [root@mon ~]# systemctl enable ntpd
   [root@mon ~]# systemctl start ntpd

   Copy to Clipboard Toggle word wrap

3. Ensure that ntpd is synchronizing the clocks correctly:

   Example

   [root@mon ~]# ntpq -p

   Copy to Clipboard Toggle word wrap

Procedure

1. If the remaining Ceph Monitors are able to form a quorum, get the Ceph Monitor map by using the ceph mon getmap command:

   [root@mon ~]# ceph mon getmap -o /tmp/monmap

   Copy to Clipboard Toggle word wrap

2. If the remaining Ceph Monitors are not able to form the quorum and you have at least one Ceph Monitor with a correct Ceph Monitor map, copy it from that Ceph Monitor:

   1. Stop the Ceph Monitor which you want to copy the Ceph Monitor map from:

      [root@mon ~]# systemctl stop ceph-mon@<host-name>

      Copy to Clipboard Toggle word wrap

      For example, to stop the Ceph Monitor running on a host with the host1 short host name:

      [root@mon ~]# systemctl stop ceph-mon@host1

      Copy to Clipboard Toggle word wrap

   2. Copy the Ceph Monitor map:

      [root@mon ~]# ceph-mon -i ID --extract-monmap /tmp/monmap

      Copy to Clipboard Toggle word wrap

      Replace ID with the ID of the Ceph Monitor which you want to copy the Ceph Monitor map from:

      [root@mon ~]# ceph-mon -i mon.a  --extract-monmap /tmp/monmap

      Copy to Clipboard Toggle word wrap

3. Stop the Ceph Monitor with the corrupted or outdated Ceph Monitor map:

   [root@mon ~]# systemctl stop ceph-mon@HOST_NAME

   Copy to Clipboard Toggle word wrap

   For example, to stop a Ceph Monitor running on a host with the host2 short host name:

   [root@mon ~]# systemctl stop ceph-mon@host2

   Copy to Clipboard Toggle word wrap

4. You can inject the Ceph Monitor map as a ceph user in two different ways:

   • Run the command as a ceph user:

     Syntax

     su - ceph -c 'ceph-mon -i ID --inject-monmap /tmp/monmap'

     Copy to Clipboard Toggle word wrap

     Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map:

     Example

     [root@mon ~]# su - ceph -c 'ceph-mon -i mon.c --inject-monmap /tmp/monmap'

     Copy to Clipboard Toggle word wrap

   • Run the command as a root user and then run chown to change the permissions:

     1. Run the command as a root user:

        Syntax

        ceph-mon -i ID --inject-monmap /tmp/monmap

        Copy to Clipboard Toggle word wrap

        Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map:

        Example

        [root@mon ~]# ceph-mon -i mon.c --inject-monmap /tmp/monmap

        Copy to Clipboard Toggle word wrap

     2. Change the file permissions:

        Example

        [root@mon ~]# chown -R ceph:ceph /var/lib/ceph/mon/ceph-c/

        Copy to Clipboard Toggle word wrap

5. Start the Ceph Monitor:

   [root@mon ~]# systemctl start ceph-mon@host2

   Copy to Clipboard Toggle word wrap

   If you copied the Ceph Monitor map from another Ceph Monitor, start that Ceph Monitor, too:

   [root@mon ~]# systemctl start ceph-mon@host1

   Copy to Clipboard Toggle word wrap

Procedure

1. From the Monitor host, remove the Monitor store by default located at /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME:

   rm -rf /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME

   Copy to Clipboard Toggle word wrap

   Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor store of a Monitor running on host1 from a cluster called remote:

   [root@mon ~]# rm -rf /var/lib/ceph/mon/remote-host1

   Copy to Clipboard Toggle word wrap

2. Remove the Monitor from the Monitor map (monmap):

   ceph mon remove SHORT_HOST_NAME --cluster CLUSTER_NAME

   Copy to Clipboard Toggle word wrap

   Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor running on host1 from a cluster called remote:

   [root@mon ~]# ceph mon remove host1 --cluster remote

   Copy to Clipboard Toggle word wrap
3. Troubleshoot and fix any problems related to the underlying file system or hardware of the Monitor host.

4. From the Ansible administration node, redeploy the Monitor by running the ceph-ansible playbook:

   $ /usr/share/ceph-ansible/ansible-playbook site.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. To compact the Monitor store when the ceph-mon daemon is running:

   ceph tell mon.HOST_NAME compact

   Copy to Clipboard Toggle word wrap

2. Replace HOST_NAME with the short host name of the host where the ceph-mon is running. Use the hostname -s command when unsure.

   # ceph tell mon.host1 compact

   Copy to Clipboard Toggle word wrap

3. Add the following parameter to the Ceph configuration under the [mon] section:

   [mon]
   mon_compact_on_start = true

   Copy to Clipboard Toggle word wrap

4. Restart the ceph-mon daemon:

   [root@mon ~]# systemctl restart ceph-mon@_HOST_NAME_

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.

   [root@mon ~]# systemctl restart ceph-mon@host1

   Copy to Clipboard Toggle word wrap

5. Ensure that Monitors have formed a quorum:

   [root@mon ~]# ceph mon stat

   Copy to Clipboard Toggle word wrap

6. Repeat these steps on other Monitors if needed.

   Note

   Before you start, ensure that you have the ceph-test package installed.

7. Verify that the ceph-mon daemon with the large store is not running. Stop the daemon if needed.

   [root@mon ~]# systemctl status ceph-mon@HOST_NAME
   [root@mon ~]# systemctl stop ceph-mon@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.

   [root@mon ~]# systemctl status ceph-mon@host1
   [root@mon ~]# systemctl stop ceph-mon@host1

   Copy to Clipboard Toggle word wrap

8. Compact the Monitor store as a ceph user in two different ways:

   • Run the command as a ceph user:

     Syntax

     su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon.HOST_NAME compact'

     Copy to Clipboard Toggle word wrap

     Example

     [root@mon ~]# su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact'

     Copy to Clipboard Toggle word wrap

   • Run the command as a root user and then run chown to change the permissions:

     1. Run the command as a root user:

        Syntax

        ceph-monstore-tool /var/lib/ceph/mon/mon.HOST_NAME compact

        Copy to Clipboard Toggle word wrap

        Example

        [root@mon ~]# ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact

        Copy to Clipboard Toggle word wrap

     2. Change the file permissions:

        Example

        [root@mon ~]# chown -R ceph:ceph /var/lib/ceph/mon/mon.node1

        Copy to Clipboard Toggle word wrap

9. Start ceph-mon again:

   [root@mon ~]# systemctl start ceph-mon@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# systemctl start ceph-mon@host1

   Copy to Clipboard Toggle word wrap

Procedure

1. To resolve this situation, for each host running ceph-mgr daemons, open ports 6800-7300.

   Example

   [root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp
   [root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp --permanent

   Copy to Clipboard Toggle word wrap

2. Restart the ceph-mgr daemons.

Procedure

1. Log into the monitor node.

2. From the monitor nodes, if you are unable to access the OSD nodes without being the root user, then copy the public key pair to the OSD nodes:

   1. Generate the SSH key pair, accept the default file name, and leave the passphrase empty:

      Example

      [root@mons-1 ~]# ssh-keygen

      Copy to Clipboard Toggle word wrap

   2. Copy the public key to all the OSD nodes in the storage cluster:

      Example

      [root@mons-1 ~]#  ssh-copy-id root@osds-1
      [root@mons-1 ~]#  ssh-copy-id root@osds-2
      [root@mons-1 ~]#  ssh-copy-id root@osds-3

      Copy to Clipboard Toggle word wrap

3. Stop the OSD daemon service on all the OSD nodes:

   Example

   [root@osds-1 ~]#  sudo systemctl stop ceph-osd\*.service ceph-osd.target

   Copy to Clipboard Toggle word wrap

4. To collect the cluster map from all the OSD nodes, create the recovery file and execute the script:

   1. Create the recovery file:

      Example

      [root@mons-1 ~]# touch recover.sh

      Copy to Clipboard Toggle word wrap

   2. Add the following content to the file and replace the OSD_NODES with either the IP addresses of all the OSD nodes or the hostname of the all OSD nodes in the Red Hat Ceph Storage cluster:

      Syntax

       --------------------------------------------------------------------------  NOTE: The directory names specified by 'ms', 'db', and 'db_slow' must end
       with a trailing / otherwise rsync will not operate properly.  --------------------------------------------------------------------------
      ms=/tmp/monstore/
      db=/root/db/
      db_slow=/root/db.slow/
      
      mkdir -p $ms $db $db_slow
      
       --------------------------------------------------------------------------  NOTE: Replace the contents inside double quotes for 'osd_nodes' below with
       the list of OSD nodes in the environment.  --------------------------------------------------------------------------
      osd_nodes="OSD_NODES_1 OSD_NODES_2 OSD_NODES_3..."
      
      for osd_node in $osd_nodes; do
      echo "Operating on $osd_node"
      rsync -avz --delete $ms $osd_node:$ms
      rsync -avz --delete $db $osd_node:$db
      rsync -avz --delete $db_slow $osd_node:$db_slow
      
      ssh -t $osd_node <<EOF
      for osd in /var/lib/ceph/osd/ceph-*; do
          ceph-objectstore-tool --type bluestore --data-path \$osd --op update-mon-db --no-mon-config --mon-store-path $ms
          if [ -e \$osd/keyring ]; then
              cat \$osd/keyring >> $ms/keyring
              echo '    caps mgr = "allow profile osd"' >> $ms/keyring
              echo '    caps mon = "allow profile osd"' >> $ms/keyring
              echo '    caps osd = "allow *"' >> $ms/keyring
          else
              echo WARNING: \$osd on $osd_node does not have a local keyring.
          fi
      done
      EOF
      
      rsync -avz --delete --remove-source-files $osd_node:$ms $ms
      rsync -avz --delete --remove-source-files $osd_node:$db $db
      rsync -avz --delete --remove-source-files $osd_node:$db_slow $db_slow
      done
       --------------------------------------------------------------------------  End of script
      ## --------------------------------------------------------------------------

      Copy to Clipboard Toggle word wrap

   3. Provide executable permission on the file:

      Example

      [root@mons-1 ~]# chmod 755 recover.sh

      Copy to Clipboard Toggle word wrap

   4. Execute the file to gather the keyrings of all the OSDs from all the OSD nodes in the storage cluster:

      Example

      [root@mons-1 ~]# ./recovery.sh

      Copy to Clipboard Toggle word wrap

5. Fetch the keyrings of other daemons from the respective nodes:

   1. For Ceph Monitor, the keyring is the same for all the Ceph monitors.

      Syntax

      cat /var/lib/ceph/mon/ceph-MONITOR_NODE/keyring

      Copy to Clipboard Toggle word wrap

      Example

      [root@mons-1 ~]# cat /var/lib/ceph/mon/ceph-mons-1/keyring

      Copy to Clipboard Toggle word wrap

   2. For Ceph Manager, fetch the keyring from all the manager nodes:

      Syntax

      cat /var/lib/ceph/mgr/ceph-MANAGER_NODE/keyring

      Copy to Clipboard Toggle word wrap

      Example

      [root@mons-1 ~]# cat /var/lib/ceph/mgr/ceph-mons-1/keyring

      Copy to Clipboard Toggle word wrap

   3. For Ceph OSDs, the keyrings are generated from the above script and stored in a temporary path:

      In this example, the OSD keyrings are stored in the /tmp/monstore/keyring file.

   4. For the client, fetch the keyring from all the client nodes:

      Syntax

      cat /etc/ceph/CLIENT_KEYRING

      Copy to Clipboard Toggle word wrap

      Example

      [root@client ~]# cat /etc/ceph/ceph.client.admin.keyring

      Copy to Clipboard Toggle word wrap

   5. For meta data server (MDS), fetch the keyring from all the Ceph MDS nodes:

      Syntax

      cat /var/lib/ceph/mds/ceph-MDS_NODE/keyring

      Copy to Clipboard Toggle word wrap

      Example

      [root@mons-2 ~]# cat /var/lib/ceph/mds/ceph-mds-1/keyring

      Copy to Clipboard Toggle word wrap

      For this key ring append the following caps if not exist:

      caps mds =  "allow"
      caps mon = "allow profile mds"
      caps osd = "allow *"

      Copy to Clipboard Toggle word wrap

   6. For Ceph Object Gateway, fetch the keyring from all the Ceph Object Gateway nodes:

      Syntax

      cat /var/lib/ceph/radosgw/ceph-CEPH_OBJECT_GATEWAY_NODE/keyring

      Copy to Clipboard Toggle word wrap

      Example

      [root@mons-3 ~]# cat /var/lib/ceph/radosgw/ceph-rgw-1/keyring

      Copy to Clipboard Toggle word wrap

      For this key ring append the following caps if not exist:

      caps mon = "allow rw"
      caps osd = "allow *"

      Copy to Clipboard Toggle word wrap

6. On the Ansible administration node, create a file with all the keyrings fetched from the previous step:

   Example

   [root@admin ~]# cat /tmp/monstore/keyring
   
   [mon.]
   	key = AQAa+RxhAAAAABAApmwud0GQHX0raMBc9zTAYQ==
   	caps mon = "allow *"
   [client.admin]
   	key = AQAo+RxhcYWtGBAAiY4kKltMGnAXqPLM2A+B8w==
   	caps mds = "allow *"
   	caps mgr = "allow *"
   	caps mon = "allow *"
   	caps osd = "allow *"
   [mgr.mons-1]
   	key = AQA++RxhAAAAABAAKdG1ETTEMR8KPa9ZpfcIzw==
   	caps mds = "allow *"
   	caps mon = "allow profile mgr"
   	caps osd = "allow *"
   [mgr.mons-2]
   	key = AQA9+RxhAAAAABAAcCBxsoaIl0sdHTz3dqX4SQ==
   	caps mds = "allow *"
   	caps mon = "allow profile mgr"
   	caps osd = "allow *"
   [mgr.mons-3]
   	key = AQA/+RxhAAAAABAAwe/mwv0hS79fWP+00W6ypQ==
   	caps mds = "allow *"
   	caps mon = "allow profile mgr"
   	caps osd = "allow *"
   [osd.1]
   key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.5]
   key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.8]
   key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.2]
   key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.4]
   key = AQCE+Rxh0mDxDRAApAeqKOJycW5bpP3IuAhSMw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.7]
   key = AQCJ+Rxhn+RAIhAAp1ImK1jiazBsDpmTQvVEVw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.0]
   key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.3]
   key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.6]
   key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.1]
   key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.5]
   key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.8]
   key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.2]
   key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.0]
   key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.3]
   key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [osd.6]
   key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ==
       caps mgr = "allow profile osd"
       caps mon = "allow profile osd"
       caps osd = "allow *"
   [mds.mds-1]
   	key = AQDs+RxhAF9vERAAdn6ArdUJ31RLr2sBVkzp3A==
           caps mds = "allow"
           caps mon = "allow profile mds"
           caps osd = "allow *"
   [mds.mds-2]
   	key = AQDs+RxhROoAFxAALAgMfM45wC5ht/vSFN2EzQ==
           caps mds = "allow"
           caps mon = "allow profile mds"
           caps osd = "allow *"
   [mds.mds-3]
   	key = AQDs+Rxhd092FRAArXLIHAhMp2z9zcWDCSoIDQ==
           caps mds = "allow"
           caps mon = "allow profile mds"
           caps osd = "allow *"
   [client.rgw.rgws-1.rgw0]
   	key = AQD9+Rxh0iP2MxAAYY76Js1AaZhzFG44cvcyOw==
           caps mon = "allow rw"
           caps osd = "allow *"

   Copy to Clipboard Toggle word wrap

7. Optional: On each Ceph Monitor node, ensure that the monitor map is not available:

   Example

   [root@mons-1 ~]# ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap
   [root@mons-1 ~]# monmaptool /tmp/monmap --print
   
   monmaptool: monmap file /tmp/monmap
   monmaptool: couldn't open /tmp/monmap: (2) No such file or directory
   
   Notice theNo such file or directory  error message if monmap is missed
   
   Notice that the “No such file or directory”  error message if monmap is missed

   Copy to Clipboard Toggle word wrap

8. From each Ceph Monitor node, fetch the MONITOR_ID, IP_ADDRESS_OF_MONITOR, and FSID from the etc/ceph/ceph.conf file:

   Example

   [global]
   cluster network = 10.0.208.0/22
   fsid = 9877bde8-ccb2-4758-89c3-90ca9550ffea
   mon host = [v2:10.0.211.00:3300,v1:10.0.211.00:6789],[v2:10.0.211.13:3300,v1:10.0.211.13:6789],[v2:10.0.210.13:3300,v1:10.0.210.13:6789]
   mon initial members = ceph-mons-1, ceph-mons-2, ceph-mons-3

   Copy to Clipboard Toggle word wrap

9. On the Ceph Monitor node, rebuild the monitor map:

   Syntax

   monmaptool --create --addv MONITOR_ID IP_ADDRESS_OF_MONITOR --enable-all-features --clobber PATH_OF_MONITOR_MAP --fsid FSID

   Copy to Clipboard Toggle word wrap

   Example

   [root@mons-1 ~]# monmaptool --create --addv mons-1 [v2:10.74.177.30:3300,v1:10.74.177.30:6789] --addv mons-2 [v2:10.74.179.197:3300,v1:10.74.179.197:6789] --addv mons-3 [v2:10.74.182.123:3300,v1:10.74.182.123:6789] --enable-all-features --clobber /root/monmap.mons-1 --fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8
   
   monmaptool: monmap file /root/monmap.mons-1
   monmaptool: set fsid to 6c01cb34-33bf-44d0-9aec-3432276f6be8
   monmaptool: writing epoch 0 to /root/monmap.mon-a (3 monitors)

   Copy to Clipboard Toggle word wrap

10. On the Ceph Monitor node, check the generated monitor map:

    Syntax

    monmaptool PATH_OF_MONITOR_MAP --print

    Copy to Clipboard Toggle word wrap

    Example

    [root@mons-1 ~]# monmaptool /root/monmap.mons-1 --print
    
    monmaptool: monmap file /root/monmap.mons-1
    epoch 0
    fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8
    last_changed 2021-11-23 02:57:23.235505
    created 2021-11-23 02:57:23.235505
    min_mon_release 0 (unknown)
    election_strategy: 1
    0: [v2:10.74.177.30:3300/0,v1:10.74.177.30:6789/0] mon.mons-1
    1: [v2:10.74.179.197:3300/0,v1:10.74.179.197:6789/0] mon.mons-2
    2: [v2:10.74.182.123:3300/0,v1:10.74.182.123:6789/0] mon.mons-3

    Copy to Clipboard Toggle word wrap

11. On the Ceph Monitor node where we are recovering the monitors, rebuild the Ceph Monitor store from the collected map:

    Syntax

    ceph-monstore-tool /tmp/monstore rebuild -- --keyring KEYRING_PATH  --monmap PATH_OF_MONITOR_MAP

    Copy to Clipboard Toggle word wrap

    In this example, the recovery is run on the mons-1 node.

    Example

    [root@mons-1 ~]# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/monstore/keyring  --monmap /root/monmap.mons-1

    Copy to Clipboard Toggle word wrap

12. Change the ownership of monstore directory to ceph:

    Example

    [root@mons-1 ~]# chown -R ceph:ceph /tmp/monstore

    Copy to Clipboard Toggle word wrap

13. On all the Ceph Monitor nodes, take a back-up of the corrupted store:

    Example

    [root@mons-1 ~]# mv /var/lib/ceph/mon/ceph-mons-1/store.db /var/lib/ceph/mon/ceph-mons-1/store.db.corrupted

    Copy to Clipboard Toggle word wrap

14. On all the Ceph Monitor nodes, replace the corrupted store:

    Example

    [root@mons-1 ~]# scp -r /tmp/monstore/store.db mons-1:/var/lib/ceph/mon/ceph-mons-1/

    Copy to Clipboard Toggle word wrap

15. On all the Ceph Monitor nodes, change the owner of the new store:

    Example

    [root@mons-1 ~]# chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db

    Copy to Clipboard Toggle word wrap

16. On all the Ceph OSD nodes, start the OSDs:

    Example

    [root@osds-1 ~]# sudo systemctl start ceph-osd.target

    Copy to Clipboard Toggle word wrap

17. On all the Ceph Monitor nodes, start the monitors

    Example

    [root@mons-1 ~]# sudo systemctl start ceph-mon.target

    Copy to Clipboard Toggle word wrap