Chapter 4. Ceph File System administration

Procedure

1. To unmount a Ceph File System mounted as a kernel client:

   Syntax

   umount MOUNT_POINT

   Example

   [root@client ~]# umount /mnt/cephfs

Procedure

1. To unmount a Ceph File System mounted in FUSE:

   Syntax

   fusermount -u MOUNT_POINT

   Example

   [root@client ~]# fusermount -u /mnt/cephfs

Procedure

1. Add the p flag to the Ceph user’s capabilities:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME client.CLIENT_NAME /DIRECTORY CAPABILITY [/DIRECTORY CAPABILITY] ...

   Example

   [user@client ~]$ ceph fs authorize cephfs_a client.1 /temp rwp
   
   client.1
     key: AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==
     caps: [mds] allow r, allow rwp path=/temp
     caps: [mon] allow r
     caps: [osd] allow rw tag cephfs data=cephfs_a

2. Set the ceph.dir.pin extended attribute on a directory:

   Syntax

   setfattr -n ceph.dir.pin -v RANK DIRECTORY

   Example

   [user@client ~]$ setfattr -n ceph.dir.pin -v 2 /temp

   This example assigns the /temp directory and all of its subdirectories to rank 2.

Procedure

1. Set the ceph.dir.pin extended attribute to -1 on a directory:

   Syntax

   setfattr -n ceph.dir.pin -v -1 DIRECTORY

   Example

   [user@client ~]$ serfattr -n ceph.dir.pin -v -1 /home/ceph-user

   Note

   Any separately mapped subdirectories of /home/ceph-user/ are not affected.

Procedure

1. Create a new data pool:

   Syntax

   ceph osd pool create POOL_NAME PG_NUMBER

   Replace:

   • POOL_NAME with the name of the pool.
   • PG_NUMBER with the number of placement groups (PGs).

   Example

   [root@mon ~]# ceph osd pool create cephfs_data_ssd 64
   pool 'cephfs_data_ssd' created

2. Add the newly created pool under the control of the Metadata Servers:

   Syntax

   ceph fs add_data_pool FS_NAME POOL_NAME

   Replace:

   • FS_NAME with the name of the file system.
   • POOL_NAME with the name of the pool.

   Example:

   [root@mon ~]# ceph fs add_data_pool cephfs cephfs_data_ssd
   added data pool 6 to fsmap

3. Verify that the pool was successfully added:

   Example

   [root@mon ~]# ceph fs ls
   name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ssd]

4. If you use the cephx authentication, make sure that clients can access the new pool.

Procedure

1. To mark the CephFS cluster down:

   Syntax

   ceph fs set FS_NAME down true

   Exmaple

   [root@mon]# ceph fs set cephfs down true

   1. To bring the CephFS cluster back up:

      Syntax

      ceph fs set FS_NAME down false

      Exmaple

      [root@mon]# ceph fs set cephfs down false

   or

1. To quickly take down a CephFS cluster:

   Syntax

   ceph fs fail FS_NAME

   Exmaple

   [root@mon]# ceph fs fail cephfs

Procedure

1. Display the CephFS status to determine the MDS ranks.

   Syntax

   ceph fs status

   Example

   [root@mon ~]# ceph fs status
   cephfs - 0 clients
   ======
   +------+--------+----------------+---------------+-------+-------+
   | Rank | State  |      MDS       |    Activity   |  dns  |  inos |
   +------+--------+----------------+---------------+-------+-------+
   |  0   | active | cluster1-node6 | Reqs:    0 /s |   10  |   13  |
   +------+--------+----------------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 2688k | 15.0G |
   |   cephfs_data   |   data   |    0  | 15.0G |
   +-----------------+----------+-------+-------+
   +----------------+
   |  Standby MDS   |
   +----------------+
   | cluster1-node5 |
   +----------------+

   In the example above, the rank is 0.

2. Mark the CephFS as down:

   Syntax

   ceph fs set FS_NAME down true

   Replace FS_NAME with the name of the CephFS you want to remove.

   Example

   [root@mon]# ceph fs set cephfs down true
   marked down

3. Display the status of the CephFS to determine it has stopped:

   Syntax

   ceph fs status

   Example

   [root@mon ~]# ceph fs status
   cephfs - 0 clients
   ======
   +------+----------+----------------+----------+-------+-------+
   | Rank |  State   |      MDS       | Activity |  dns  |  inos |
   +------+----------+----------------+----------+-------+-------+
   |  0   | stopping | cluster1-node6 |          |   10  |   12  |
   +------+----------+----------------+----------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 2688k | 15.0G |
   |   cephfs_data   |   data   |    0  | 15.0G |
   +-----------------+----------+-------+-------+
   +----------------+
   |  Standby MDS   |
   +----------------+
   | cluster1-node5 |
   +----------------+

   After some time, the MDS is no longer listed:

   Example

   [root@mon ~]# ceph fs status
   cephfs - 0 clients
   ======
   +------+-------+-----+----------+-----+------+
   | Rank | State | MDS | Activity | dns | inos |
   +------+-------+-----+----------+-----+------+
   +------+-------+-----+----------+-----+------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 2688k | 15.0G |
   |   cephfs_data   |   data   |    0  | 15.0G |
   +-----------------+----------+-------+-------+
   +----------------+
   |  Standby MDS   |
   +----------------+
   | cluster1-node5 |
   +----------------+

4. Fail all MDS ranks shown in the status of step one:

   Syntax

   ceph mds fail RANK

   Replace RANK with the rank of the MDS daemons to fail.

   Example

   [root@mon]# ceph mds fail 0

5. Remove the CephFS:

   Syntax

   ceph fs rm FS_NAME --yes-i-really-mean-it

   Replace FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [root@mon]# ceph fs rm cephfs --yes-i-really-mean-it

6. Verify that the file system is removed:

   Syntax

   ceph fs ls

   Example

   [root@mon ~]# ceph fs ls
   No filesystems enabled

7. Optional: Remove the pools that were used by CephFS.

   1. On a Ceph Monitor node, list the pools:

      Syntax

      ceph osd pool ls

      Example

      [root@mon ~]# ceph osd pool ls
      rbd
      cephfs_data
      cephfs_metadata

      In the example output, cephfs_metadata and cephfs_data are the pools that were used by CephFS.

   2. Remove the metadata pool:

      Syntax

      ceph osd pool delete CEPH_METADATA_POOL CEPH_METADATA_POOL --yes-i-really-really-mean-it

      Replace CEPH_METADATA_POOL with the pool CephFS used for metadata storage by including the pool name twice.

      Example

      [root@mon ~]# ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
      pool 'cephfs_metadata' removed

   3. Remove the data pool:

      Syntax

      ceph osd pool delete CEPH_DATA_POOL CEPH_DATA_POOL --yes-i-really-really-mean-it

      Replace CEPH_DATA_POOL with the pool CephFS used for data storage by including the pool name twice.

      Example

      [root@mon ~]# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
      pool 'cephfs_data' removed

Procedure

1. Navigate to the /usr/share/ceph-ansible/ directory:

   [admin@admin ~]$ cd /usr/share/ceph-ansible

2. Identify the Ceph Metadata Server (MDS) nodes by reviewing the [mdss] section in the Ansible inventory file. On the Ansible administration node, open /usr/share/ceph-ansible/hosts:

   Example

   [mdss]
   cluster1-node5
   cluster1-node6

   In the example, cluster1-node5 and cluster1-node6 are the MDS nodes.

3. Set the max_mds parameter to 1:

   Syntax

   ceph fs set NAME max_mds NUMBER

   Example

   [root@mon ~]# ceph fs set cephfs max_mds 1

4. Run the shrink-mds.yml playbook, specifying the Metadata Server (MDS) to remove:

   Syntax

   ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=MDS_NODE -i hosts

   Replace MDS_NODE with the Metadata Server node you want to remove. The Ansible playbook will ask you if you want to shrink the cluster. Type yes and press the enter key.

   Example

   [admin@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=cluster1-node6 -i hosts

5. Optional: Repeat the process for any additional MDS nodes:

   Syntax

   ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=MDS_NODE -i hosts

   Replace MDS_NODE with the Metadata Server node you want to remove. The Ansible playbook will ask you if you want to shrink the cluster. Type yes and press the enter key.

   Example

   [admin@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=cluster1-node5 -i hosts

6. Check the status of the CephFS:

   Syntax

   ceph fs status

   Example

   [root@mon ~]# ceph fs status
   cephfs - 0 clients
   ======
   +------+--------+----------------+---------------+-------+-------+
   | Rank | State  |      MDS       |    Activity   |  dns  |  inos |
   +------+--------+----------------+---------------+-------+-------+
   |  0   | failed | cluster1-node6 | Reqs:    0 /s |   10  |   13  |
   +------+--------+----------------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 2688k | 15.0G |
   |   cephfs_data   |   data   |    0  | 15.0G |
   +-----------------+----------+-------+-------+
   +----------------+
   |  Standby MDS   |
   +----------------+
   | cluster1-node5 |
   +----------------+

7. Remove the [mdss] section and the nodes in it from the Ansible inventory file so they will not be reprovisioned as metadata servers on future runs of the site.yml or site-container.yml playbooks. Open for editing the Ansible inventory file, /usr/share/ceph-ansible/hosts:

   Example

   [mdss]
   cluster1-node5
   cluster1-node6

   Remove the [mdss] section and all nodes under it.

8. Remove the CephFS:

   Syntax

   ceph fs rm FS_NAME --yes-i-really-mean-it

   Replace FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [root@mon]# ceph fs rm cephfs --yes-i-really-mean-it

9. Optional: Remove the pools that were used by CephFS.

   1. On a Ceph Monitor node, list the pools:

      Syntax

      ceph osd pool ls

      Find the pools that were used by CephFS.

      Example

      [root@mon ~]# ceph osd pool ls
      rbd
      cephfs_data
      cephfs_metadata

      In the example output, cephfs_metadata and cephfs_data are the pools that were used by CephFS.

   2. Remove the metadata pool:

      Syntax

      ceph osd pool delete CEPH_METADATA_POOL CEPH_METADATA_POOL --yes-i-really-really-mean-it

      Replace CEPH_METADATA_POOL with the pool CephFS used for metadata storage by including the pool name twice.

      Example

      [root@mon ~]# ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
      pool 'cephfs_metadata' removed

   3. Remove the data pool:

      Syntax

      ceph osd pool delete CEPH_DATA_POOL CEPH_DATA_POOL --yes-i-really-really-mean-it

      Replace CEPH_METADATA_POOL with the pool CephFS used for metadata storage by including the pool name twice.

      Example

      [root@mon ~]# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
      pool 'cephfs_data' removed

   4. Verify the pools no longer exist:

      Example

      [root@mon ~]# ceph osd pool ls
      rbd

      The cephfs_metadata and cephfs_data pools are no longer listed.

Procedure

1. Set the minimum client version:

   ceph fs set name min_compat_client release

   Replace name with the name of the Ceph File System and release with the minimum client version. For example to restrict clients to use the nautilus upstream version at minimum on the cephfs Ceph File System:

   $ ceph fs set cephfs min_compat_client nautilus

   See Table 4.1, “min_compat_client values” for the full list of available values and how they correspond with Red Hat Ceph Storage versions.

Procedure

1. To fail a daemon:

   Syntax

   ceph mds fail MDS_NAME

   Where MDS_NAME is the name of the standby-replay MDS node.

   Example

   [root@mds ~]# ceph mds fail example01

   Note

   You can find the Ceph MDS name from the ceph fs status command.

Procedure

1. Review the client list:

   Syntax

   ceph tell DAEMON_NAME client ls

   Exmaple

   [root@mon]# ceph tell mds.0 client ls
   [
       {
           "id": 4305,
           "num_leases": 0,
           "num_caps": 3,
           "state": "open",
           "replay_requests": 0,
           "completed_requests": 0,
           "reconnecting": false,
           "inst": "client.4305 172.21.9.34:0/422650892",
           "client_metadata": {
               "ceph_sha1": "ae81e49d369875ac8b569ff3e3c456a31b8f3af5",
               "ceph_version": "ceph version 12.0.0-1934-gae81e49 (ae81e49d369875ac8b569ff3e3c456a31b8f3af5)",
               "entity_id": "0",
               "hostname": "senta04",
               "mount_point": "/tmp/tmpcMpF1b/mnt.0",
               "pid": "29377",
               "root": "/"
           }
       }
   ]

2. Evict the specified CephFS client:

   Syntax

   ceph tell DAEMON_NAME client evict id=ID_NUMBER

   Exmaple

   [root@mon]# ceph tell mds.0 client evict id=4305

Procedure

1. Review the blacklist:

   Exmaple

   [root@mon]# ceph osd blacklist ls
   listed 1 entries
   127.0.0.1:0/3710147553 2020-03-19 11:32:24.716146

2. Remove the CephFS client from the blacklist:

   Syntax

   ceph osd blacklist rm CLIENT_NAME_OR_IP_ADDR

   Exmaple

   [root@mon]# ceph osd blacklist rm 127.0.0.1:0/3710147553
   un-blacklisting 127.0.0.1:0/3710147553

3. Optionally, to have FUSE-based CephFS clients trying automatically to reconnect when removing them from the blacklist. On the FUSE client, set the following option to true:

   client_reconnect_stale = true