Chapter 6. Troubleshooting Ceph MDSs

Procedure

1. Log in to the Ansible administration node.

2. Change to the /usr/share/ceph-ansible directory:

   Example

   [ansible@admin ~]$ cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

3. Run the Ansible shrink-mds.yml playbook, and when prompted, type yes to confirm shrinking the cluster:

   Syntax

   ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=ID -i hosts

   Copy to Clipboard Toggle word wrap

   Replace ID with the ID of the MDS node you want to remove. You can remove only one Ceph MDS each time the playbook runs.

   Example

   [ansible @admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=node02 -i hosts

   Copy to Clipboard Toggle word wrap

4. As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and remove the MDS node under the [mdss] section:

   Syntax

   [mdss]
   MDS_NODE_NAME
   MDS_NODE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [mdss]
   node01
   node03

   Copy to Clipboard Toggle word wrap

   In this example, node02 was removed from the [mdss] list.

Procedure

1. Log into the Ceph MDS node that you want to remove the MDS daemon from.

2. Stop the Ceph MDS service:

   Syntax

   sudo systemctl stop ceph-mds@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host where the daemon is running.

   Example

   [admin@node02 ~]$ sudo systemctl stop ceph-mds@node02

   Copy to Clipboard Toggle word wrap

3. Disable the MDS service if you are not redeploying MDS to this node:

   Syntax

   sudo systemctl disable ceph-mds@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host to disable the daemon.

   Example

   [admin@node02 ~]$ sudo systemctl disable ceph-mds@node02

   Copy to Clipboard Toggle word wrap

4. Remove the /var/lib/ceph/mds/ceph-MDS_ID directory on the MDS node:

   Syntax

   sudo rm -fr /var/lib/ceph/mds/ceph-MDS_ID

   Copy to Clipboard Toggle word wrap

   Replace MDS_ID with the ID of the MDS node that you want to remove the MDS daemon from.

   Example

   [admin@node02 ~]$ sudo rm -fr /var/lib/ceph/mds/ceph-node02

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the Ansible administration node

2. Change to the /usr/share/ceph-ansible directory:

   Example

   [ansible@admin ~]$ cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

3. As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and add the MDS node under the [mdss] section:

   Syntax

   [mdss]
   MDS_NODE_NAME
   NEW_MDS_NODE_NAME

   Copy to Clipboard Toggle word wrap

   Replace NEW_MDS_NODE_NAME with the host name of the node where you want to install the MDS server.

   Alternatively, you can colocate the MDS daemon with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections.

   Example

   [mdss]
   node01
   node03

   Copy to Clipboard Toggle word wrap

4. As the ansible user, run the Ansible playbook to provision the MDS node:

   • Bare-metal deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit mdss -i hosts

     Copy to Clipboard Toggle word wrap

   • Container deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit mdss -i hosts

     Copy to Clipboard Toggle word wrap

     After the Ansible playbook has finished running, the new Ceph MDS node appears in the storage cluster.

Procedure

1. Add a new MDS node by logging into the node and creating an MDS mount point:

   Syntax

   sudo mkdir /var/lib/ceph/mds/ceph-MDS_ID

   Copy to Clipboard Toggle word wrap

   Replace MDS_ID with the ID of the MDS node that you want to add the MDS daemon to.

   Example

   [admin@node03 ~]$ sudo mkdir /var/lib/ceph/mds/ceph-node03

   Copy to Clipboard Toggle word wrap

2. If this is a new MDS node, create the authentication key if you are using Cephx authentication:

   Syntax

   sudo ceph auth get-or-create mds.MDS_ID mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-MDS_ID/keyring

   Copy to Clipboard Toggle word wrap

   Replace MDS_ID with the ID of the MDS node to deploy the MDS daemon on.

   Example

   [admin@node03 ~]$ sudo ceph auth get-or-create mds.node03 mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-node03/keyring

   Copy to Clipboard Toggle word wrap

   Note

   Cephx authentication is enabled by default. See the Cephx authentication link in the Additional Resources section for more information about Cephx authentication.

3. Start the MDS daemon:

   Syntax

   sudo systemctl start ceph-mds@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host to start the daemon.

   Example

   [admin@node03 ~]$ sudo systemctl start ceph-mds@node03

   Copy to Clipboard Toggle word wrap

4. Enable the MDS service:

   Syntax

   systemctl enable ceph-mds@HOST_NAME

   Copy to Clipboard Toggle word wrap

   Replace HOST_NAME with the short name of the host to enable the service.

   Example

   [admin@node03 ~]$ sudo systemctl enable ceph-mds@node03

   Copy to Clipboard Toggle word wrap