Chapter 4. Troubleshooting Monitors

If the remaining Monitors are able to form a quorum, get the Monitor map by using the ceph mon getmap command:

# ceph mon getmap -o /tmp/monmap

If the remaining Monitors are not able to form the quorum and you have at least one Monitor with a correct Monitor map, copy it from that Monitor:

Stop the Monitor with the corrupted or outdated Monitor map:

systemctl stop ceph-mon@<host-name>

For example, to stop a Monitor running on a host with the host2 short host name:

# systemctl stop ceph-mon@host2

Inject the Monitor map:

ceph-mon -i <id> --inject-monmap /tmp/monmap

Replace <id> with the ID of the Monitor with the corrupted or outdated Monitor map, for example:

# ceph-mon -i mon.c --inject-monmap /tmp/monmap

Start the Monitor, for example:

# systemctl start ceph-mon@host2

If you copied the Monitor map from another Monitor, start that Monitor, too, for example:

# systemctl start ceph-mon@host1

Collect the cluster map from all OSD nodes:

ms=<directory>
mkdir $ms

for host in $host_list; do
  rsync -avz "$ms" root@$host:"$ms"; rm -rf "$ms"
  ssh root@$host <<EOF
  for osd in  /var/lib/ceph/osd/ceph-*; do
    ceph-objectstore-tool --data-path \$osd --op update-mon-db --mon-store-path $ms
  done
EOF
rsync -avz root@$host:$ms $ms; done

Replace <directory> with a temporary directory to store the collected cluster map, for example:

$ ms=/tmp/monstore/
$ mkdir $ms
$ for host in $host_list; do
  rsync -avz "$ms" root@$host:"$ms"; rm -rf "$ms"
  ssh root@$host <<EOF
  for osd in  /var/lib/ceph/osd/ceph-*; do
    ceph-objectstore-tool --data-path \$osd --op update-mon-db --mon-store-path $ms
  done
EOF
rsync -avz root@$host:$ms $ms; done

Set appropriate capabilities:

ceph-authtool <keyring>  -n mon. --cap mon 'allow *'
ceph-authtool <keyring>  -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'

Replace <keyring> with the path to the client administration keyring, for example:

$ ceph-authtool /etc/ceph/ceph.client.admin.keyring  -n mon. --cap mon 'allow *'
$ ceph-authtool /etc/ceph/ceph.client.admin.keyring  -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'

Rebuild the Monitor store from the collected map:

ceph-monstore-tool <directory> rebuild -- --keyring <keyring>

Replace <directory> with the temporary directory from the first step and <keyring> with the path to the client administration keyring, for example:

$ ceph-monstore-tool /tmp/mon-store rebuild -- --keyring /etc/ceph/ceph.client.admin.keyring

$ ceph-monstore-tool /tmp/mon-store rebuild

Back up the corrupted store:

mv /var/lib/ceph/mon/<mon-ID>/store.db \
   /var/lib/ceph/mon/<mon-ID>/store.db.corrupted

Replace <mon-ID> with the Monitor ID, for example <mon.0>:

# mv /var/lib/ceph/mon/mon.0/store.db \
     /var/lib/ceph/mon/mon.0/store.db.corrupted

Replace the corrupted store:

mv /tmp/mon-store/store.db /var/lib/ceph/mon/<mon-ID>/store.db

Replace <mon-ID> with the Monitor ID, for example <mon.0>:

# mv /tmp/mon-store/store.db /var/lib/ceph/mon/mon.0/store.db

Repeat this step for all Monitors with corrupted store.

Change the owner of the new store:

chown -R ceph:ceph /var/lib/ceph/mon/<mon-ID>/store.db

Replace <mon-ID> with the Monitor ID, for example <mon.0>:

# chown -R ceph:ceph /var/lib/ceph/mon/mon.0/store.db

Repeat this step for all Monitors with corrupted store.

From the Monitor host, remove the Monitor store by default located at /var/lib/ceph/mon/<cluster-name>-<short-host-name>:

rm -rf /var/lib/ceph/mon/<cluster-name>-<short-host-name>

Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor store of a Monitor running on host1 from a cluster called remote:

# rm -rf /var/lib/ceph/mon/remote-host1

Remove the Monitor from the Monitor map (monmap):

ceph mon remove <short-host-name> --cluster <cluster-name>

Specify the short host name of the Monitor host and the cluster name. For example, to remove the Monitor running on host1 from a cluster called remote:

# ceph mon remove host1 --cluster remote

From the Ansible administration node, redeploy the Monitor by running the ceph-ansible playbook:

# /usr/share/ceph-ansible/ansible-playbook site.yml

Add the following parameter to the Ceph configuration under the [mon] section:

[mon]
mon_compact_on_start = true

Restart the ceph-mon daemon:

systemctl restart ceph-mon@<host-name>

Replace <host-name> with the short name of the host where the daemon is running. Use the hostname -s command when unsure.

# systemctl restart ceph-mon@host1

Ensure that Monitors have formed a quorum:

# ceph mon stat

Verify that the ceph-mon daemon with the large store is not running. Stop the daemon if needed.

systemctl status ceph-mon@<host-name>
systemctl stop ceph-mon@<host-name>

Replace <host-name> with the short name of the host where the daemon is running. Use the hostname -s command when unsure.

# systemctl status ceph-mon@host1
# systemctl stop ceph-mon@host1

Compact the Monitor store:

ceph-monstore-tool /var/lib/ceph/mon/mon.<host-name> compact

Replace <host-name> with a short host name of the Monitor host.

# ceph-monstore-tool /var/lib/ceph/mon/mon.node1 compact

Start ceph-mon again:

systemctl start ceph-mon@<host-name>

For example:

# systemctl start ceph-mon@host1