Chapter 5. Ceph File System administration

Procedure

1. Enable the Red Hat Ceph Storage 8 tools repository, if it is not already enabled:

   Red Hat Enterprise Linux 9

   [root@client ~]# subscription-manager repos --enable=rhceph-8-tools-for-rhel-9-x86_64-rpms

   Copy to Clipboard Toggle word wrap

2. Install the cephfs-top package:

   Example

   [root@client ~]# dnf install cephfs-top

   Copy to Clipboard Toggle word wrap

3. Enable the Ceph Manager stats plugin:

   Example

   [root@client ~]# ceph mgr module enable stats

   Copy to Clipboard Toggle word wrap

4. Create the client.fstop Ceph user:

   Example

   [root@client ~]# ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring

   Copy to Clipboard Toggle word wrap

   Note

   Optionally, use the --id argument to specify a different Ceph user, other than client.fstop.

5. Start the cephfs-top utility:

   Example

   [root@client ~]# cephfs-top
   cephfs-top - Wed Nov 30 15:26:05 2022
   
   All Filesystem Info
   Total Client(s): 4 - 3 FUSE, 1 kclient, 0 libcephfs
   COMMANDS: m - select a filesystem | s - sort menu | l - limit number of clients | r - reset to default | q - quit
   
     client_id mount_root chit(%) dlease(%) ofiles oicaps oinodes rtio(MB) raio(MB) rsp(MB/s) wtio(MB) waio(MB) wsp(MB/s) rlatavg(ms) rlatsd(ms) wlatavg(ms) wlatsd(ms) mlatavg(ms) mlatsd(ms) mount_point@host/addr
   
   Filesystem: cephfs1 - 2 client(s)
   
   
     4500     /          100.0   100.0	     0	    751    0       0.0	    0.0	     0.0	   578.13   0.03     0.0       N/A         N/A        N/A         N/A       N/A        N/A       N/A@example/192.168.1.4
     4501     /          100.0   0.0      0	    1    0       0.0	    0.0	     0.0	   0.0   0.0     0.0       0.0         0.0        0.0         0.0        0.41        0.0       /mnt/cephfs2@example/192.168.1.4
   
   Filesystem: cephfs2 - 2 client(s)
   
   
     4512     /          100.0   0.0	     0	    1      0       0.0	    0.0	     0.0	   0.0      0.0      0.0       0.0         0.0        0.0         0.0        0.4        0.0        /mnt/cephfs3@example/192.168.1.4
     4518     /          100.0   0.0	     0	    1      0       0.0	    0.0	     0.0	   0.0      0.0      0.0       0.0         0.0        0.0         0.0        0.52        0.0        /mnt/cephfs4@example/192.168.1.4

   Copy to Clipboard Toggle word wrap

Procedure

1. Add the p flag to the Ceph user’s capabilities:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME client.CLIENT_NAME /DIRECTORY CAPABILITY [/DIRECTORY CAPABILITY] ...

   Copy to Clipboard Toggle word wrap

   Example

   [user@client ~]$ ceph fs authorize cephfs_a client.1 /temp rwp
   
   client.1
     key: AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==
     caps: [mds] allow r, allow rwp path=/temp
     caps: [mon] allow r
     caps: [osd] allow rw tag cephfs data=cephfs_a

   Copy to Clipboard Toggle word wrap

2. Set the ceph.dir.pin extended attribute on a directory:

   Syntax

   setfattr -n ceph.dir.pin -v RANK DIRECTORY

   Copy to Clipboard Toggle word wrap

   Example

   [user@client ~]$ setfattr -n ceph.dir.pin -v 2 /temp

   Copy to Clipboard Toggle word wrap

   This example assigns the /temp directory and all of its subdirectories to rank 2.

Procedure

1. Create a new data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Copy to Clipboard Toggle word wrap

   Replace:

   • POOL_NAME with the name of the pool.

   Example

   [ceph: root@host01 /]# ceph osd pool create cephfs_data_ssd
   
   pool 'cephfs_data_ssd' created

   Copy to Clipboard Toggle word wrap

2. Add the newly created pool under the control of the Metadata Servers:

   Syntax

   ceph fs add_data_pool FS_NAME POOL_NAME

   Copy to Clipboard Toggle word wrap

   Replace:

   • FS_NAME with the name of the file system.
   • POOL_NAME with the name of the pool.

   Example:

   [ceph: root@host01 /]# ceph fs add_data_pool cephfs cephfs_data_ssd
   
   added data pool 6 to fsmap

   Copy to Clipboard Toggle word wrap

3. Verify that the pool was successfully added:

   Example

   [ceph: root@host01 /]# ceph fs ls
   
   name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ssd]

   Copy to Clipboard Toggle word wrap

4. Optional: Remove a data pool from the file system:

   Syntax

   ceph fs rm_data_pool FS_NAME POOL_NAME

   Copy to Clipboard Toggle word wrap

   Example:

   [ceph: root@host01 /]# ceph fs rm_data_pool cephfs cephfs_data_ssd
   
   removed data pool 6 from fsmap

   Copy to Clipboard Toggle word wrap

   1. Verify that the pool was successfully removed:

      Example

      [ceph: root@host01 /]# ceph fs ls
      
      name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs.cephfs.data]

      Copy to Clipboard Toggle word wrap

5. If you use the cephx authentication, make sure that clients can access the new pool.

Procedure

1. To mark the CephFS cluster down:

   Syntax

   ceph fs set FS_NAME down true

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph fs set cephfs down true

   Copy to Clipboard Toggle word wrap

   1. To bring the CephFS cluster back up:

      Syntax

      ceph fs set FS_NAME down false

      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph fs set cephfs down false

      Copy to Clipboard Toggle word wrap

   or

1. To quickly take down a CephFS cluster:

   Syntax

   ceph fs fail FS_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph fs fail cephfs

   Copy to Clipboard Toggle word wrap

   Note

   To get the CephFS cluster back up, set cephfs to joinable:

   Syntax

   ceph fs set FS_NAME joinable true

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true
   
   cephfs marked joinable; MDS may join as newly active.

   Copy to Clipboard Toggle word wrap

Procedure

1. Mark the storage cluster as down:

   Syntax

   ceph fs set FS_NAME down true

   Copy to Clipboard Toggle word wrap

   Replace

   • FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [ceph: root@host01 /]# ceph fs set cephfs down true
   
   cephfs marked down.

   Copy to Clipboard Toggle word wrap

2. Display the status of the Ceph File System:

   ceph fs status

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph fs status
   
   cephfs - 0 clients
   ======
   +-------------------+----------+-------+-------+
   |       POOL        |   TYPE   |  USED | AVAIL |
   +-----------------+------------+-------+-------+
   |cephfs.cephfs.meta | metadata | 31.5M |  52.6G|
   |cephfs.cephfs.data |   data   |    0  |  52.6G|
   +-----------------+----------+-------+---------+
                  STANDBY MDS
   cephfs.ceph-host01
   cephfs.ceph-host02
   cephfs.ceph-host03

   Copy to Clipboard Toggle word wrap

3. Remove the Ceph File System:

   Syntax

   ceph fs rm FS_NAME --yes-i-really-mean-it

   Copy to Clipboard Toggle word wrap

   Replace

   • FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [ceph: root@host01 /]# ceph fs rm cephfs --yes-i-really-mean-it

   Copy to Clipboard Toggle word wrap

4. Verify that the file system has been successfully removed:

   Example

   [ceph: root@host01 /]# ceph fs ls

   Copy to Clipboard Toggle word wrap

5. Optional. Remove data and metadata pools associated with the removed file system.

Procedure

1. Review the client list:

   Syntax

   ceph tell DAEMON_NAME client ls

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph tell mds.0 client ls
   [
       {
           "id": 4305,
           "num_leases": 0,
           "num_caps": 3,
           "state": "open",
           "replay_requests": 0,
           "completed_requests": 0,
           "reconnecting": false,
           "inst": "client.4305 172.21.9.34:0/422650892",
           "client_metadata": {
               "ceph_sha1": "79f0367338897c8c6d9805eb8c9ad24af0dcd9c7",
               "ceph_version": "ceph version 16.2.8-65.el8cp (79f0367338897c8c6d9805eb8c9ad24af0dcd9c7)",
               "entity_id": "0",
               "hostname": "senta04",
               "mount_point": "/tmp/tmpcMpF1b/mnt.0",
               "pid": "29377",
               "root": "/"
           }
       }
   ]

   Copy to Clipboard Toggle word wrap

2. Evict the specified CephFS client:

   Syntax

   ceph tell DAEMON_NAME client evict id=ID_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph tell mds.0 client evict id=4305

   Copy to Clipboard Toggle word wrap

Procedure

1. Review the blocklist:

   Example

   [ceph: root@host01 /]# ceph osd blocklist ls
   
   listed 1 entries
   127.0.0.1:0/3710147553 2022-05-09 11:32:24.716146

   Copy to Clipboard Toggle word wrap

2. Remove the CephFS client from the blocklist:

   Syntax

   ceph osd blocklist rm CLIENT_NAME_OR_IP_ADDR

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph osd blocklist rm 127.0.0.1:0/3710147553
   
   un-blocklisting 127.0.0.1:0/3710147553

   Copy to Clipboard Toggle word wrap

3. Optionally, you can have kernel-based CephFS clients automatically reconnect when removing them from the blocklist. On the kernel-based CephFS client, set the following option to clean either when doing a manual mount, or automatically mounting with an entry in the /etc/fstab file:

   recover_session=clean

   Copy to Clipboard Toggle word wrap

4. Optionally, you can have FUSE-based CephFS clients automatically reconnect when removing them from the blocklist. On the FUSE client, set the following option to true either when doing a manual mount, or automatically mounting with an entry in the /etc/fstab file:

   client_reconnect_stale=true

   Copy to Clipboard Toggle word wrap