Chapter 4. Troubleshooting Ceph Monitors

This chapter contains information on how to fix the most common errors related to the Ceph Monitors.

4.1. Prerequisites

Verify the network connection.

4.2. Most common Ceph Monitor errors

The following tables list the most common error messages that are returned by the ceph health detail command, or included in the Ceph logs. The tables provide links to corresponding sections that explain the errors and point to specific procedures to fix the problems.

4.2.1. Prerequisites

A running Red Hat Ceph Storage cluster.

4.2.2. Ceph Monitor error messages

A table of common Ceph Monitor error messages, and a potential fix.

Error message	See
`HEALTH_WARN`
`mon.X is down (out of quorum)`	Ceph Monitor is out of quorum
`clock skew`	Clock skew
`store is getting too big!`	The Ceph Monitor store is getting too big

4.2.3. Common Ceph Monitor error messages in the Ceph logs

A table of common Ceph Monitor error messages found in the Ceph logs, and a link to a potential fix.

Error message	Log file	See
`clock skew`	Main cluster log	Clock skew
`clocks not synchronized`	Main cluster log	Clock skew
`Corruption: error in middle of record`	Monitor log	Ceph Monitor is out of quorum Recovering the Ceph Monitor store
`Corruption: 1 missing files`	Monitor log	Ceph Monitor is out of quorum Recovering the Ceph Monitor store
`Caught signal (Bus error)`	Monitor log	Ceph Monitor is out of quorum

4.2.4. Ceph Monitor is out of quorum

One or more Ceph Monitors are marked as down but the other Ceph Monitors are still able to form a quorum. In addition, the ceph health detail command returns an error message similar to the following one:

HEALTH_WARN 1 mons down, quorum 1,2 mon.b,mon.c
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)

What This Means

Ceph marks a Ceph Monitor as down due to various reasons.

If the ceph-mon daemon is not running, it might have a corrupted store or some other error is preventing the daemon from starting. Also, the /var/ partition might be full. As a consequence, ceph-mon is not able to perform any operations to the store located by default at /var/lib/ceph/mon-SHORT_HOST_NAME/store.db and terminates.

If the ceph-mon daemon is running but the Ceph Monitor is out of quorum and marked as down, the cause of the problem depends on the Ceph Monitor state:

If the Ceph Monitor is in the probing state longer than expected, it cannot find the other Ceph Monitors. This problem can be caused by networking issues, or the Ceph Monitor can have an outdated Ceph Monitor map (monmap) and be trying to reach the other Ceph Monitors on incorrect IP addresses. Alternatively, if the monmap is up-to-date, Ceph Monitor’s clock might not be synchronized.
If the Ceph Monitor is in the electing state longer than expected, the Ceph Monitor’s clock might not be synchronized.
If the Ceph Monitor changes its state from synchronizing to electing and back, the cluster state is advancing. This means that it is generating new maps faster than the synchronization process can handle.
If the Ceph Monitor marks itself as the leader or a peon, then it believes to be in a quorum, while the remaining cluster is sure that it is not. This problem can be caused by failed clock synchronization.

To Troubleshoot This Problem

Verify that the ceph-mon daemon is running. If not, start it:
```
[root@mon ~]# systemctl status ceph-mon@HOST_NAME
[root@mon ~]# systemctl start ceph-mon@HOST_NAME
```
Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.
If you are not able to start ceph-mon, follow the steps in The ceph-mon daemon cannot start.
If you are able to start the ceph-mon daemon but is is marked as down, follow the steps in The ceph-mon daemon is running, but marked as `down`.

The ceph-mon Daemon Cannot Start

Check the corresponding Ceph Monitor log, by default located at /var/log/ceph/ceph-mon.HOST_NAME.log.
If the log contains error messages similar to the following ones, the Ceph Monitor might have a corrupted store.
```
Corruption: error in middle of record
Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb
```
To fix this problem, replace the Ceph Monitor. See Replacing a failed monitor.
If the log contains an error message similar to the following one, the /var/ partition might be full. Delete any unnecessary data from /var/.
```
Caught signal (Bus error)
```
Important
Do not delete any data from the Monitor directory manually. Instead, use the ceph-monstore-tool to compact it. See Compacting the Ceph Monitor store for details.
If you see any other error messages, open a support ticket. See Contacting Red Hat Support for service for details.

The ceph-mon Daemon Is Running, but Still Marked as down

From the Ceph Monitor host that is out of the quorum, use the mon_status command to check its state:
```
[root@mon ~]# ceph daemon ID mon_status
```
Replace ID with the ID of the Ceph Monitor, for example:
```
[root@mon ~]# ceph daemon mon.a mon_status
```
If the status is probing, verify the locations of the other Ceph Monitors in the mon_status output.
1. If the addresses are incorrect, the Ceph Monitor has incorrect Ceph Monitor map (monmap). To fix this problem, see Injecting a Ceph Monitor map.
2. If the addresses are correct, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. In addition, troubleshoot any networking issues, see Troubleshooting Networking issues for details.
If the status is electing, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details.
If the status changes from electing to synchronizing, open a support ticket. See Contacting Red Hat Support for service for details.
If the Ceph Monitor is the leader or a peon, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. Open a support ticket if synchronizing the clocks does not solve the problem. See Contacting Red Hat Support for service for details.

Additional Resources

See Understanding Ceph Monitor status
The Starting, Stopping, Restarting the Ceph daemons by instance section in the Administration Guide for Red Hat Ceph Storage 4
The Using the Ceph Administration Socket section in the Administration Guide for Red Hat Ceph Storage 4

4.2.5. Clock skew

A Ceph Monitor is out of quorum, and the ceph health detail command output contains error messages similar to these:

mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
mon.a addr 127.0.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s)

In addition, Ceph logs contain error messages similar to these:

2015-06-04 07:28:32.035795 7f806062e700 0 log [WRN] : mon.a 127.0.0.1:6789/0 clock skew 0.14s > max 0.05s
2015-06-04 04:31:25.773235 7f4997663700 0 log [WRN] : message from mon.1 was stamped 0.186257s in the future, clocks not synchronized

What This Means

The clock skew error message indicates that Ceph Monitors' clocks are not synchronized. Clock synchronization is important because Ceph Monitors depend on time precision and behave unpredictably if their clocks are not synchronized.

The mon_clock_drift_allowed parameter determines what disparity between the clocks is tolerated. By default, this parameter is set to 0.05 seconds.

Important

Do not change the default value of mon_clock_drift_allowed without previous testing. Changing this value might affect the stability of the Ceph Monitors and the Ceph Storage Cluster in general.

Possible causes of the clock skew error include network problems or problems with Network Time Protocol (NTP) synchronization if that is configured. In addition, time synchronization does not work properly on Ceph Monitors deployed on virtual machines.

To Troubleshoot This Problem

Verify that your network works correctly. For details, see Troubleshooting networking issues. In particular, troubleshoot any problems with NTP clients if you use NTP.
1. If you use chrony for NTP, see Basic chrony NTP troubleshooting section for more information.
2. If you use ntpd, see Basic NTP troubleshooting.
If you use a remote NTP server, consider deploying your own NTP server on your network.
1. For details, see the Using the Chrony suite to configure NTP chapter in the Configuring basic system settings for Red Hat Enterprise Linux 8.
2. See the Configuring NTP Using ntpd chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

Note

Ceph evaluates time synchronization every five minutes only so there will be a delay between fixing the problem and clearing the clock skew messages.

Additional Resources

4.2.6. The Ceph Monitor store is getting too big

The ceph health command returns an error message similar to the following one:

mon.ceph1 store is getting too big! 48031 MB >= 15360 MB -- 62% avail

What This Means

Ceph Monitors store is in fact a LevelDB database that stores entries as key–values pairs. The database includes a cluster map and is located by default at /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME/store.db.

Querying a large Monitor store can take time. As a consequence, the Ceph Monitor can be delayed in responding to client queries.

In addition, if the /var/ partition is full, the Ceph Monitor cannot perform any write operations to the store and terminates. See Ceph Monitor is out of quorum for details on troubleshooting this issue.

To Troubleshoot This Problem

Check the size of the database:

du -sch /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME/store.db

Specify the name of the cluster and the short host name of the host where the ceph-mon is running.

Example

# du -sch /var/lib/ceph/mon/ceph-host1/store.db
47G     /var/lib/ceph/mon/ceph-ceph1/store.db/
47G     total

Compact the Ceph Monitor store. For details, see Compacting the Ceph Monitor Store.

Additional Resources

Ceph Monitor is out of quorum

4.2.7. Understanding Ceph Monitor status

The mon_status command returns information about a Ceph Monitor, such as:

State
Rank
Elections epoch
Monitor map (monmap)

If Ceph Monitors are able to form a quorum, use mon_status with the ceph command-line utility.

If Ceph Monitors are not able to form a quorum, but the ceph-mon daemon is running, use the administration socket to execute mon_status.

An example output of mon_status

{
    "name": "mon.3",
    "rank": 2,
    "state": "peon",
    "election_epoch": 96,
    "quorum": [
        1,
        2
    ],
    "outside_quorum": [],
    "extra_probe_peers": [],
    "sync_provider": [],
    "monmap": {
        "epoch": 1,
        "fsid": "d5552d32-9d1d-436c-8db1-ab5fc2c63cd0",
        "modified": "0.000000",
        "created": "0.000000",
        "mons": [
            {
                "rank": 0,
                "name": "mon.1",
                "addr": "172.25.1.10:6789\/0"
            },
            {
                "rank": 1,
                "name": "mon.2",
                "addr": "172.25.1.12:6789\/0"
            },
            {
                "rank": 2,
                "name": "mon.3",
                "addr": "172.25.1.13:6789\/0"
            }
        ]
    }
}

Ceph Monitor States

Leader: During the electing phase, Ceph Monitors are electing a leader. The leader is the Ceph Monitor with the highest rank, that is the rank with the lowest value. In the example above, the leader is mon.1.
Peon: Peons are the Ceph Monitors in the quorum that are not leaders. If the leader fails, the peon with the highest rank becomes a new leader.
Probing: A Ceph Monitor is in the probing state if it is looking for other Ceph Monitors. For example after you start the Ceph Monitors, they are probing until they find enough Ceph Monitors specified in the Ceph Monitor map (monmap) to form a quorum.
Electing: A Ceph Monitor is in the electing state if it is in the process of electing the leader. Usually, this status changes quickly.
Synchronizing: A Ceph Monitor is in the synchronizing state if it is synchronizing with the other Ceph Monitors to join the quorum. The smaller the Ceph Monitor store it, the faster the synchronization process. Therefore, if you have a large store, synchronization takes longer time.

Additional Resources

For details, see the Using the Ceph Administration Socket section in the Administration Guide for Red Hat Ceph Storage 4.

4.2.8. Additional Resources

See the Section 4.2.2, “Ceph Monitor error messages” in the Red Hat Ceph Storage Troubleshooting Guide.
See the Section 4.2.3, “Common Ceph Monitor error messages in the Ceph logs” in the Red Hat Ceph Storage Troubleshooting Guide.

4.3. Injecting a `monmap`

If a Ceph Monitor has an outdated or corrupted Ceph Monitor map (monmap), it cannot join a quorum because it is trying to reach the other Ceph Monitors on incorrect IP addresses.

The safest way to fix this problem is to obtain and inject the actual Ceph Monitor map from other Ceph Monitors.

Note

This action overwrites the existing Ceph Monitor map kept by the Ceph Monitor.

This procedure shows how to inject the Ceph Monitor map when the other Ceph Monitors are able to form a quorum, or when at least one Ceph Monitor has a correct Ceph Monitor map. If all Ceph Monitors have corrupted store and therefore also the Ceph Monitor map, see Recovering the Ceph Monitor store.

Prerequisites

Access to the Ceph Monitor Map.
Root-level access to the Ceph Monitor node.

Procedure

If the remaining Ceph Monitors are able to form a quorum, get the Ceph Monitor map by using the ceph mon getmap command:
```
[root@mon ~]# ceph mon getmap -o /tmp/monmap
```
If the remaining Ceph Monitors are not able to form the quorum and you have at least one Ceph Monitor with a correct Ceph Monitor map, copy it from that Ceph Monitor:
1. Stop the Ceph Monitor which you want to copy the Ceph Monitor map from:
```
[root@mon ~]# systemctl stop ceph-mon@<host-name>
```
  For example, to stop the Ceph Monitor running on a host with the host1 short host name:
```
[root@mon ~]# systemctl stop ceph-mon@host1
```
2. Copy the Ceph Monitor map:
```
[root@mon ~]# ceph-mon -i ID --extract-monmap /tmp/monmap
```
  Replace ID with the ID of the Ceph Monitor which you want to copy the Ceph Monitor map from:
```
[root@mon ~]# ceph-mon -i mon.a  --extract-monmap /tmp/monmap
```
Stop the Ceph Monitor with the corrupted or outdated Ceph Monitor map:
```
[root@mon ~]# systemctl stop ceph-mon@HOST_NAME
```
For example, to stop a Ceph Monitor running on a host with the host2 short host name:
```
[root@mon ~]# systemctl stop ceph-mon@host2
```
You can inject the Ceph Monitor map as a ceph user in two different ways:
- Run the command as a ceph user:
  Syntax
```
su - ceph -c 'ceph-mon -i ID --inject-monmap /tmp/monmap'
```
  Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map:
  Example
```
[root@mon ~]# su - ceph -c 'ceph-mon -i mon.c --inject-monmap /tmp/monmap'
```
- Run the command as a root user and then run chown to change the permissions:
  1. Run the command as a root user:
    Syntax
    
    ceph-mon -i ID --inject-monmap /tmp/monmap
    
    Replace ID with the ID of the Ceph Monitor with the corrupted or outdated Ceph Monitor map:
    Example
    
    [root@mon ~]# ceph-mon -i mon.c --inject-monmap /tmp/monmap
  2. Change the file permissions:
    Example
    
    [root@mon ~]# chown -R ceph:ceph /var/lib/ceph/mon/ceph-c/
Start the Ceph Monitor:
```
[root@mon ~]# systemctl start ceph-mon@host2
```
If you copied the Ceph Monitor map from another Ceph Monitor, start that Ceph Monitor, too:
```
[root@mon ~]# systemctl start ceph-mon@host1
```

Additional Resources

4.4. Replacing a failed Monitor

When a Monitor has a corrupted store, the recommended way to fix this problem is to replace the Monitor by using the Ansible automation application.

Prerequisites

A running Red Hat Ceph Storage cluster.
Able to form a quroum.
Root-level access to Ceph Monitor node.

Procedure

From the Monitor host, remove the Monitor store by default located at /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME:
```
rm -rf /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME
```
Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor store of a Monitor running on host1 from a cluster called remote:
```
[root@mon ~]# rm -rf /var/lib/ceph/mon/remote-host1
```
Remove the Monitor from the Monitor map (monmap):
```
ceph mon remove SHORT_HOST_NAME --cluster CLUSTER_NAME
```
Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor running on host1 from a cluster called remote:
```
[root@mon ~]# ceph mon remove host1 --cluster remote
```
Troubleshoot and fix any problems related to the underlying file system or hardware of the Monitor host.
From the Ansible administration node, redeploy the Monitor by running the ceph-ansible playbook:
```
$ /usr/share/ceph-ansible/ansible-playbook site.yml
```

Additional Resources

See the Ceph Monitor is out of quorum for details.
The Managing the storage cluster size chapter in the Red Hat Ceph Storage Operations Guide.
The Deploying Red Hat Ceph Storage chapter in the Red Hat Ceph Storage 4 Installation Guide.

4.5. Compacting the monitor store

When the Monitor store has grown big in size, you can compact it:

Dynamically by using the ceph tell command.
Upon the start of the ceph-mon daemon.
By using the ceph-monstore-tool when the ceph-mon daemon is not running. Use this method when the previously mentioned methods fail to compact the Monitor store or when the Monitor is out of quorum and its log contains the Caught signal (Bus error) error message.

Important

Monitor store size changes when the cluster is not in the active+clean state or during the rebalancing process. For this reason, compact the Monitor store when rebalancing is completed. Also, ensure that the placement groups are in the active+clean state.

Prerequisites

A running Red Hat Ceph Storage cluster.
Root-level access to the Ceph Monitor node.

Procedure

To compact the Monitor store when the ceph-mon daemon is running:
```
ceph tell mon.HOST_NAME compact
```
Replace HOST_NAME with the short host name of the host where the ceph-mon is running. Use the hostname -s command when unsure.
```
# ceph tell mon.host1 compact
```
Add the following parameter to the Ceph configuration under the [mon] section:
```
[mon]
mon_compact_on_start = true
```
Restart the ceph-mon daemon:
```
[root@mon ~]# systemctl restart ceph-mon@_HOST_NAME_
```
Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.
```
[root@mon ~]# systemctl restart ceph-mon@host1
```
Ensure that Monitors have formed a quorum:
```
[root@mon ~]# ceph mon stat
```
Repeat these steps on other Monitors if needed.
Note
Before you start, ensure that you have the ceph-test package installed.
Verify that the ceph-mon daemon with the large store is not running. Stop the daemon if needed.
```
[root@mon ~]# systemctl status ceph-mon@HOST_NAME
[root@mon ~]# systemctl stop ceph-mon@HOST_NAME
```
Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.
```
[root@mon ~]# systemctl status ceph-mon@host1
[root@mon ~]# systemctl stop ceph-mon@host1
```

Compact the Monitor store as a ceph user in two different ways:

Run the command as a ceph user:

Syntax

su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon.HOST_NAME compact'

Example

[root@mon ~]# su - ceph -c 'ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact'

Run the command as a root user and then run chown to change the permissions:

Run the command as a root user:

Syntax

ceph-monstore-tool /var/lib/ceph/mon/mon.HOST_NAME compact

Example

[root@mon ~]# ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact

Change the file permissions:

Example

[root@mon ~]# chown -R ceph:ceph /var/lib/ceph/mon/mon.node1

Start ceph-mon again:

[root@mon ~]# systemctl start ceph-mon@HOST_NAME

Example

[root@mon ~]# systemctl start ceph-mon@host1

Additional Resources

4.6. Opening port for Ceph manager

The ceph-mgr daemons receive placement group information from OSDs on the same range of ports as the ceph-osd daemons. If these ports are not open, a cluster will devolve from HEALTH_OK to HEALTH_WARN and will indicate that PGs are unknown with a percentage count of the PGs unknown.

Prerequisites

A running Red Hat Ceph Storage cluster.
Root-level access to Ceph manager.

Procedure

To resolve this situation, for each host running ceph-mgr daemons, open ports 6800-7300.

Example

[root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp
[root@ceph-mgr] # firewall-cmd --add-port 6800-7300/tcp --permanent

Restart the ceph-mgr daemons.

4.7. Recovering the Ceph Monitor for bare-metal deployments

If all the monitors are down in your Red Hat Ceph Storage cluster, and the ceph -s command does not execute as expected, you can recover the monitors using the monmaptool command. The monmaptool command rebuilds the Ceph monitor store from the keyring files of the daemons.

Note

This procedure is for bare-metal Red Hat Ceph Storage deployments only. For containerized Red Hat Ceph Storage deployments, see the Knowledgebase article MON recovery procedure for Red Hat Ceph Storage containerized deployment when all the three mon are down..

Prerequisites

Bare-metal deployed Red Hat Ceph Storage cluster.
Root-level access to all the nodes.
All the Ceph monitors are down.

Procedure

Log into the monitor node.
From the monitor nodes, if you are unable to access the OSD nodes without being the root user, then copy the public key pair to the OSD nodes:
1. Generate the SSH key pair, accept the default file name, and leave the passphrase empty:
  Example
```
[root@mons-1 ~]# ssh-keygen
```
2. Copy the public key to all the OSD nodes in the storage cluster:
  Example
```
[root@mons-1 ~]#  ssh-copy-id root@osds-1
[root@mons-1 ~]#  ssh-copy-id root@osds-2
[root@mons-1 ~]#  ssh-copy-id root@osds-3
```

Stop the OSD daemon service on all the OSD nodes:

Example

[root@osds-1 ~]#  sudo systemctl stop ceph-osd\*.service ceph-osd.target

To collect the cluster map from all the OSD nodes, create the recovery file and execute the script:

Create the recovery file:
Example
```
[root@mons-1 ~]# touch recover.sh
```

Add the following content to the file and replace the OSD_NODES with either the IP addresses of all the OSD nodes or the hostname of the all OSD nodes in the Red Hat Ceph Storage cluster:

Syntax

 --------------------------------------------------------------------------  NOTE: The directory names specified by 'ms', 'db', and 'db_slow' must end
 with a trailing / otherwise rsync will not operate properly.  --------------------------------------------------------------------------
ms=/tmp/monstore/
db=/root/db/
db_slow=/root/db.slow/

mkdir -p $ms $db $db_slow

 --------------------------------------------------------------------------  NOTE: Replace the contents inside double quotes for 'osd_nodes' below with
 the list of OSD nodes in the environment.  --------------------------------------------------------------------------
osd_nodes="OSD_NODES_1 OSD_NODES_2 OSD_NODES_3..."

for osd_node in $osd_nodes; do
echo "Operating on $osd_node"
rsync -avz --delete $ms $osd_node:$ms
rsync -avz --delete $db $osd_node:$db
rsync -avz --delete $db_slow $osd_node:$db_slow

ssh -t $osd_node <<EOF
for osd in /var/lib/ceph/osd/ceph-*; do
    ceph-objectstore-tool --type bluestore --data-path \$osd --op update-mon-db --no-mon-config --mon-store-path $ms
    if [ -e \$osd/keyring ]; then
        cat \$osd/keyring >> $ms/keyring
        echo '    caps mgr = "allow profile osd"' >> $ms/keyring
        echo '    caps mon = "allow profile osd"' >> $ms/keyring
        echo '    caps osd = "allow *"' >> $ms/keyring
    else
        echo WARNING: \$osd on $osd_node does not have a local keyring.
    fi
done
EOF

rsync -avz --delete --remove-source-files $osd_node:$ms $ms
rsync -avz --delete --remove-source-files $osd_node:$db $db
rsync -avz --delete --remove-source-files $osd_node:$db_slow $db_slow
done
 --------------------------------------------------------------------------  End of script
## --------------------------------------------------------------------------

Provide executable permission on the file:
Example
```
[root@mons-1 ~]# chmod 755 recover.sh
```
Execute the file to gather the keyrings of all the OSDs from all the OSD nodes in the storage cluster:
Example
```
[root@mons-1 ~]# ./recovery.sh
```

Fetch the keyrings of other daemons from the respective nodes:
1. For Ceph Monitor, the keyring is the same for all the Ceph monitors.
  Syntax
```
cat /var/lib/ceph/mon/ceph-MONITOR_NODE/keyring
```
  Example
```
[root@mons-1 ~]# cat /var/lib/ceph/mon/ceph-mons-1/keyring
```
2. For Ceph Manager, fetch the keyring from all the manager nodes:
  Syntax
```
cat /var/lib/ceph/mgr/ceph-MANAGER_NODE/keyring
```
  Example
```
[root@mons-1 ~]# cat /var/lib/ceph/mgr/ceph-mons-1/keyring
```
3. For Ceph OSDs, the keyrings are generated from the above script and stored in a temporary path:
  In this example, the OSD keyrings are stored in the /tmp/monstore/keyring file.
4. For the client, fetch the keyring from all the client nodes:
  Syntax
```
cat /etc/ceph/CLIENT_KEYRING
```
  Example
```
[root@client ~]# cat /etc/ceph/ceph.client.admin.keyring
```
5. For meta data server (MDS), fetch the keyring from all the Ceph MDS nodes:
  Syntax
```
cat /var/lib/ceph/mds/ceph-MDS_NODE/keyring
```
  Example
```
[root@mons-2 ~]# cat /var/lib/ceph/mds/ceph-mds-1/keyring
```
  For this key ring append the following caps if not exist:
```
caps mds =  "allow"
caps mon = "allow profile mds"
caps osd = "allow *"
```
6. For Ceph Object Gateway, fetch the keyring from all the Ceph Object Gateway nodes:
  Syntax
```
cat /var/lib/ceph/radosgw/ceph-CEPH_OBJECT_GATEWAY_NODE/keyring
```
  Example
```
[root@mons-3 ~]# cat /var/lib/ceph/radosgw/ceph-rgw-1/keyring
```
  For this key ring append the following caps if not exist:
```
caps mon = "allow rw"
caps osd = "allow *"
```

On the Ansible administration node, create a file with all the keyrings fetched from the previous step:

Example

[root@admin ~]# cat /tmp/monstore/keyring

[mon.]
	key = AQAa+RxhAAAAABAApmwud0GQHX0raMBc9zTAYQ==
	caps mon = "allow *"
[client.admin]
	key = AQAo+RxhcYWtGBAAiY4kKltMGnAXqPLM2A+B8w==
	caps mds = "allow *"
	caps mgr = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
[mgr.mons-1]
	key = AQA++RxhAAAAABAAKdG1ETTEMR8KPa9ZpfcIzw==
	caps mds = "allow *"
	caps mon = "allow profile mgr"
	caps osd = "allow *"
[mgr.mons-2]
	key = AQA9+RxhAAAAABAAcCBxsoaIl0sdHTz3dqX4SQ==
	caps mds = "allow *"
	caps mon = "allow profile mgr"
	caps osd = "allow *"
[mgr.mons-3]
	key = AQA/+RxhAAAAABAAwe/mwv0hS79fWP+00W6ypQ==
	caps mds = "allow *"
	caps mon = "allow profile mgr"
	caps osd = "allow *"
[osd.1]
key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.5]
key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.8]
key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.2]
key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.4]
key = AQCE+Rxh0mDxDRAApAeqKOJycW5bpP3IuAhSMw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.7]
key = AQCJ+Rxhn+RAIhAAp1ImK1jiazBsDpmTQvVEVw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.0]
key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.3]
key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.6]
key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.1]
key = AQB/+RxhlH8rFxAAL3mb8Kdb+QuWWdJi+RvwGw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.5]
key = AQCE+RxhKSNsHRAAIyLO5g75tqFVsl6MEEzwXw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.8]
key = AQCJ+Rxhc0wHJhAA5Bb2kU9Nadpm3UCLASnCfw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.2]
key = AQB/+RxhhrQCGRAAUhh77gIVhN8zsTbaKMJuHw==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.0]
key = AQB/+RxhPhh+FRAAc5b0nwiuK6o1AIbjVc6tQg==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.3]
key = AQCE+RxhJv8PARAAqCzH2br1xJmMTNnqH3I9mA==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[osd.6]
key = AQCI+RxhAt4eIhAAYQEJqSNRT7l2WNl/rYQcKQ==
    caps mgr = "allow profile osd"
    caps mon = "allow profile osd"
    caps osd = "allow *"
[mds.mds-1]
	key = AQDs+RxhAF9vERAAdn6ArdUJ31RLr2sBVkzp3A==
        caps mds = "allow"
        caps mon = "allow profile mds"
        caps osd = "allow *"
[mds.mds-2]
	key = AQDs+RxhROoAFxAALAgMfM45wC5ht/vSFN2EzQ==
        caps mds = "allow"
        caps mon = "allow profile mds"
        caps osd = "allow *"
[mds.mds-3]
	key = AQDs+Rxhd092FRAArXLIHAhMp2z9zcWDCSoIDQ==
        caps mds = "allow"
        caps mon = "allow profile mds"
        caps osd = "allow *"
[client.rgw.rgws-1.rgw0]
	key = AQD9+Rxh0iP2MxAAYY76Js1AaZhzFG44cvcyOw==
        caps mon = "allow rw"
        caps osd = "allow *"

Optional: On each Ceph Monitor node, ensure that the monitor map is not available:

Example

[root@mons-1 ~]# ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap
[root@mons-1 ~]# monmaptool /tmp/monmap --print

monmaptool: monmap file /tmp/monmap
monmaptool: couldn't open /tmp/monmap: (2) No such file or directory

Notice theNo such file or directory  error message if monmap is missed

Notice that the “No such file or directory”  error message if monmap is missed

From each Ceph Monitor node, fetch the MONITOR_ID, IP_ADDRESS_OF_MONITOR, and FSID from the etc/ceph/ceph.conf file:

Example

[global]
cluster network = 10.0.208.0/22
fsid = 9877bde8-ccb2-4758-89c3-90ca9550ffea
mon host = [v2:10.0.211.00:3300,v1:10.0.211.00:6789],[v2:10.0.211.13:3300,v1:10.0.211.13:6789],[v2:10.0.210.13:3300,v1:10.0.210.13:6789]
mon initial members = ceph-mons-1, ceph-mons-2, ceph-mons-3

On the Ceph Monitor node, rebuild the monitor map:

Syntax

monmaptool --create --addv MONITOR_ID IP_ADDRESS_OF_MONITOR --enable-all-features --clobber PATH_OF_MONITOR_MAP --fsid FSID

Example

[root@mons-1 ~]# monmaptool --create --addv mons-1 [v2:10.74.177.30:3300,v1:10.74.177.30:6789] --addv mons-2 [v2:10.74.179.197:3300,v1:10.74.179.197:6789] --addv mons-3 [v2:10.74.182.123:3300,v1:10.74.182.123:6789] --enable-all-features --clobber /root/monmap.mons-1 --fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8

monmaptool: monmap file /root/monmap.mons-1
monmaptool: set fsid to 6c01cb34-33bf-44d0-9aec-3432276f6be8
monmaptool: writing epoch 0 to /root/monmap.mon-a (3 monitors)

On the Ceph Monitor node, check the generated monitor map:

Syntax

monmaptool PATH_OF_MONITOR_MAP --print

Example

[root@mons-1 ~]# monmaptool /root/monmap.mons-1 --print

monmaptool: monmap file /root/monmap.mons-1
epoch 0
fsid 6c01cb34-33bf-44d0-9aec-3432276f6be8
last_changed 2021-11-23 02:57:23.235505
created 2021-11-23 02:57:23.235505
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:10.74.177.30:3300/0,v1:10.74.177.30:6789/0] mon.mons-1
1: [v2:10.74.179.197:3300/0,v1:10.74.179.197:6789/0] mon.mons-2
2: [v2:10.74.182.123:3300/0,v1:10.74.182.123:6789/0] mon.mons-3

On the Ceph Monitor node where we are recovering the monitors, rebuild the Ceph Monitor store from the collected map:

Syntax

ceph-monstore-tool /tmp/monstore rebuild -- --keyring KEYRING_PATH  --monmap PATH_OF_MONITOR_MAP

In this example, the recovery is run on the mons-1 node.

Example

[root@mons-1 ~]# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/monstore/keyring  --monmap /root/monmap.mons-1

Change the ownership of monstore directory to ceph:
Example
```
[root@mons-1 ~]# chown -R ceph:ceph /tmp/monstore
```

On all the Ceph Monitor nodes, take a back-up of the corrupted store:

Example

[root@mons-1 ~]# mv /var/lib/ceph/mon/ceph-mons-1/store.db /var/lib/ceph/mon/ceph-mons-1/store.db.corrupted

On all the Ceph Monitor nodes, replace the corrupted store:

Example

[root@mons-1 ~]# scp -r /tmp/monstore/store.db mons-1:/var/lib/ceph/mon/ceph-mons-1/

On all the Ceph Monitor nodes, change the owner of the new store:

Example

[root@mons-1 ~]# chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db

On all the Ceph OSD nodes, start the OSDs:

Example

[root@osds-1 ~]# sudo systemctl start ceph-osd.target

On all the Ceph Monitor nodes, start the monitors
Example
```
[root@mons-1 ~]# sudo systemctl start ceph-mon.target
```

4.8. Recovering the Ceph Monitor store

Ceph Monitors store the cluster map in a key–value store such as LevelDB. If the store is corrupted on a Monitor, the Monitor terminates unexpectedly and fails to start again. The Ceph logs might include the following errors:

Corruption: error in middle of record
Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb

Production Red Hat Ceph Storage clusters use at least three Ceph Monitors so that if one fails, it can be replaced with another one. However, under certain circumstances, all Ceph Monitors can have corrupted stores. For example, when the Ceph Monitor nodes have incorrectly configured disk or file system settings, a power outage can corrupt the underlying file system.

If there is corruption on all Ceph Monitors, you can recover it with information stored on the OSD nodes by using utilities called ceph-monstore-tool and ceph-objectstore-tool.

Important

These procedures cannot recover the following information:

Metadata Daemon Server (MDS) keyrings and maps
Placement Group settings:
- full ratio set by using the ceph pg set_full_ratio command
- nearfull ratio set by using the ceph pg set_nearfull_ratio command

Important

Never restore the Ceph Monitor store from an old backup. Rebuild the Ceph Monitor store from the current cluster state using the following steps and restore from that.

4.8.1. Recovering the Ceph Monitor store when using BlueStore

Follow this procedure if the Ceph Monitor store is corrupted on all Ceph Monitors and you use the BlueStore back end.

In containerized environments, this method requires attaching Ceph repositories and restoring to a non-containerized Ceph Monitor first.

Warning

This procedure can cause data loss. If you are unsure about any step in this procedure, contact the Red Hat Technical Support for an assistance with the recovering process.

Prerequisites

Bare-metal deployments
- The rsync and ceph-test packages are installed.
Container deployments
- All OSDs containers are stopped.
- Enable Ceph repositories on the Ceph nodes based on their roles.
- The ceph-test and rsync packages are installed on the OSD and Monitor nodes.
- The ceph-mon package is installed on the Monitor nodes.
- The ceph-osd package is installed on the OSD nodes.

Procedure

If you use Ceph in containers, mount all disk with Ceph data to a temporary location. Repeat this step for all OSD nodes.
1. List the data partitions. Use ceph-volume or ceph-disk depending on which utility you used to set up the devices:
```
[root@osd ~]# ceph-volume lvm list
```
  or
```
[root@osd ~]# ceph-disk list
```
2. Mount the data partitions to temporary location:
```
mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-$i
```
3. Restore the SELinux context:
```
for i in {OSD_ID}; do restorecon /var/lib/ceph/osd/ceph-$i; done
```
  Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.
4. Change the owner and group to ceph:ceph:
```
for i in {OSD_ID}; do chown -R ceph:ceph /var/lib/ceph/osd/ceph-$i; done
```
  Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.
  Important
  Due to a bug that causes the update-mon-db command to use additional db and db.slow directories for the Monitor database, you must also copy these directories. To do so:
  Prepare a temporary location outside the container to mount and access the OSD database and extract the OSD maps needed to restore the Ceph Monitor:
  ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev OSD-DATA --path /var/lib/ceph/osd/ceph-OSD-ID
  Replace OSD-DATA with the Volume Group (VG) or Logical Volume (LV) path to the to the OSD data and OSD-ID with the ID of the OSD.
  Create a symbolic link between the BlueStore database and block.db:
  ln -snf BLUESTORE DATABASE /var/lib/ceph/osd/ceph-OSD-ID/block.db
  Replace BLUESTORE-DATABASE with the Volume Group (VG) or Logical Volume (LV) path to the BlueStore database and OSD-ID with the ID of the OSD.

Use the following commands from the Ceph Monitor node with the corrupted store. Repeat them for all OSDs on all nodes.

Collect the cluster map from all OSD nodes:

[root@ mon~]# cd /root/
[root@mon ~]# ms=/tmp/monstore/
[root@mon ~]# db=/root/db/
[root@mon ~]# db_slow=/root/db.slow/

[root@mon ~]# mkdir $ms
[root@mon ~]# for host in $osd_nodes; do
                echo "$host"
                rsync -avz $ms $host:$ms
                rsync -avz $db $host:$db
                rsync -avz $db_slow $host:$db_slow

                rm -rf $ms
                rm -rf $db
                rm -rf $db_slow

                sh -t $host <<EOF
                  for osd in /var/lib/ceph/osd/ceph-*; do
                    ceph-objectstore-tool --type bluestore --data-path \$osd --op update-mon-db --mon-store-path $ms

                   done
                EOF

                      rsync -avz $host:$ms $ms
                      rsync -avz $host:$db $db
                      rsync -avz $host:$db_slow $db_slow
                done

Set the appropriate capabilities:

[root@mon ~]# ceph-authtool /etc/ceph/ceph.client.admin.keyring -n mon. --cap mon 'allow *' --gen-key
[root@mon ~]# cat /etc/ceph/ceph.client.admin.keyring
  [mon.]
    key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
    caps mon = "allow *"
  [client.admin]
    key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
    auid = 0
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"

Move all sst file from the db and db.slow directories to the temporary location:

[root@mon ~]# mv /root/db/*.sst /root/db.slow/*.sst /tmp/monstore/store.db

Rebuild the Monitor store from the collected map:
```
[root@mon ~]# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /etc/ceph/ceph.client.admin
```
Note
After using this command, only keyrings extracted from the OSDs and the keyring specified on the ceph-monstore-tool command line are present in Ceph’s authentication database. You have to recreate or import all other keyrings, such as clients, Ceph Manager, Ceph Object Gateway, and others, so those clients can access the cluster.
Back up the corrupted store. Repeat this step for all Ceph Monitor nodes:
```
mv /var/lib/ceph/mon/ceph-HOSTNAME/store.db /var/lib/ceph/mon/ceph-HOSTNAME/store.db.corrupted
```
Replace HOSTNAME with the host name of the Ceph Monitor node.
Replace the corrupted store. Repeat this step for all Ceph Monitor nodes:
```
scp -r /tmp/monstore/store.db HOSTNAME:/var/lib/ceph/mon/ceph-HOSTNAME/
```
Replace HOSTNAME with the host name of the Monitor node.
Change the owner of the new store. Repeat this step for all Ceph Monitor nodes:
```
chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db
```
Replace HOSTNAME with the host name of the Ceph Monitor node.

If you use Ceph in containers, then unmount all the temporary mounted OSDs on all nodes:
```
[root@osd ~]# umount /var/lib/ceph/osd/ceph-*
```

Start all the Ceph Monitor daemons:

[root@mon ~]# systemctl start ceph-mon *

Ensure that the Monitors are able to form a quorum:
- Bare-metal deployments
```
[root@mon ~]# ceph -s
```
- Containers
```
[user@admin ~]$ docker exec ceph-mon-_HOSTNAME_ ceph -s
```
  Replace HOSTNAME with the host name of the Ceph Monitor node.
Import the Ceph Manager keyring and start all Ceph Manager processes:
```
ceph auth import -i /etc/ceph/ceph.mgr.HOSTNAME.keyring
systemctl start ceph-mgr@HOSTNAME
```
Replace HOSTNAME with the host name of the Ceph Manager node.
Start all OSD processes across all OSD nodes:
```
[root@osd ~]# systemctl start ceph-osd *
```
Ensure that the OSDs are returning to service:
- Bare-metal deployments
```
[root@mon ~]# ceph -s
```
- Containers
```
[user@admin ~]$ docker exec ceph-mon-_HOSTNAME_ ceph -s
```
  Replace HOSTNAME with the host name of the Ceph Monitor node.

Additional Resources

For details on registering Ceph nodes to the Content Delivery Network (CDN), see Registering Red Hat Ceph Storage Nodes to the CDN and Attaching Subscriptions section in the Red Hat Ceph Storage Installation Guide.
For details on enabling repositories, see the Enabling the Red Hat Ceph Storage Repositories section in the Red Hat Ceph Storage Installation Guide.

4.9. Additional Resources

See Chapter 3, Troubleshooting networking issues in the Red Hat Ceph Storage Troubleshooting Guide for network-related problems.

Chapter 4. Troubleshooting Ceph Monitors

4.1. Prerequisites

4.2. Most common Ceph Monitor errors

4.2.1. Prerequisites

4.2.2. Ceph Monitor error messages

4.2.3. Common Ceph Monitor error messages in the Ceph logs

4.2.4. Ceph Monitor is out of quorum

4.2.5. Clock skew

4.2.6. The Ceph Monitor store is getting too big

4.2.7. Understanding Ceph Monitor status

4.2.8. Additional Resources

4.3. Injecting a `monmap`

4.4. Replacing a failed Monitor

4.5. Compacting the monitor store

4.6. Opening port for Ceph manager

4.7. Recovering the Ceph Monitor for bare-metal deployments

4.8. Recovering the Ceph Monitor store

4.8.1. Recovering the Ceph Monitor store when using BlueStore

4.9. Additional Resources

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 4. Troubleshooting Ceph Monitors

4.1. Prerequisites

4.2. Most common Ceph Monitor errors

4.2.1. Prerequisites

4.2.2. Ceph Monitor error messages

4.2.3. Common Ceph Monitor error messages in the Ceph logs

4.2.4. Ceph Monitor is out of quorum

4.2.5. Clock skew

4.2.6. The Ceph Monitor store is getting too big

4.2.7. Understanding Ceph Monitor status

4.2.8. Additional Resources

4.3. Injecting a monmap

4.4. Replacing a failed Monitor

4.5. Compacting the monitor store

4.6. Opening port for Ceph manager

4.7. Recovering the Ceph Monitor for bare-metal deployments

4.8. Recovering the Ceph Monitor store

4.8.1. Recovering the Ceph Monitor store when using BlueStore

4.9. Additional Resources

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links

4.3. Injecting a `monmap`