Ceph File System Guide

1. On a node with administration capabilities, set the max_mds parameter to the desired number of active MDS daemons. Note that Ceph only increases the actual number of ranks in the Ceph File Systems if a spare MDS daemon is available to take the new rank.

   ceph fs set <name> max_mds <number>

   For example, to increase the number of active MDS daemons to two in the Ceph File System called cephfs:

   [root@monitor ~]# ceph fs set cephfs max_mds 2

2. Verify the number of active MDS daemons.

   ceph fs status <name>

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   |  1   | active | node2 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   +-------------+

1. On a node with administration capabilities, change the max_mds parameter to the desired number of active MDS daemons.

   ceph fs set <name> max_mds <number>

   For example, to decrease the number of active MDS daemons to one in the Ceph File System called cephfs:

   [root@monitor ~]# ceph fs set cephfs max_mds 1

2. Deactivate the active MDS daemon:

   ceph mds deactivate <role>

   Replace <role> with "name of the Ceph File System:rank", "FSID:rank", or just rank. For example, to deactivate the MDS daemon with rank 1 on the Ceph File System named cephfs:

   [root@monitor ~]# ceph mds deactivate cephfs:1
   telling mds.1:1 127.0.0.1:6800/3187061458 to deactivate

3. Verify the number of active MDS daemons.

   ceph fs status <name>

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+

1. Create two pools, one for storing data and one for storing metadata:

   ceph osd pool create <name> <pg_num>

   Specify the pool name and the number of placement groups (PGs), for example:

   [root@monitor ~]# ceph osd pool create cephfs-data 64
   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

   Typically, the metadata pool can start with a conservative number of PGs as it will generally have far fewer objects than the data pool. It is possible to increase the number of PGs if needed. Recommended metadata pool sizes range from 64 PGs to 512 PGs. Size the data pool proportional to the number and sizes of files you expect in the file system.

   Important

   For the metadata pool, consider using

   • A higher replication level because any data loss to this pool can make the whole file system inaccessible
   • Storage with lower latency such as Solid-state Drive (SSD) disks because this directly affects the observed latency of file system operations on clients

2. Create the Ceph File System:

   ceph fs new <name> <metadata-pool> <data-pool>

   Specify the name of the Ceph File System, the metadata and data pool, for example:

   [root@monitor ~]# ceph fs new cephfs cephfs-metadata cephfs-data

3. Verify that one or more MDSs enter to the active state based on you configuration.

   ceph fs status <name>

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+----

1. On a Monitor host, create a client user.

   ceph auth get-or-create client.<id> <capabilities>

   Specify the client ID and desired capabilities.

   • To restrict the client to only write to and read from a particular pool in the cluster:

     ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=<pool>'

     For example, to restrict the client to only write to and read from the data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=data'

   • To prevent the client from modifying the pool that is used for files and directories:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=<pool>'

     For example, to prevent the client from modifying data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=data'

     Note

     Do not create capabilities for the metadata pool, as Ceph File System clients do not have access to it.

2. Verify the created key:

   ceph auth get client.<id>

   For example:

   [root@monitor ~]# ceph auth get client.1

3. If you plan to use the kernel client, create a secret file using the key retrieved from the previous step.

   On the client node, copy the string after key = into /etc/ceph/ceph.client.<id>.secret:

   For example, if the client ID is 1 add a single line to /etc/ceph/ceph.client.1.secret with the key:

   [root@client ~]# cat /etc/ceph/ceph.client.1.secret
   AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==

   Important

   Do not include the space in between key = and the string or else mounting will not work.

4. If you plan to use the File System in User Space (FUSE) client, copy the keyring to the client.

   1. On the Monitor node, export the keyring to a file:

      # ceph auth get client.<id> -o ceph.client.<id>.keyring

      For example, if the client ID is 1:

      [root@monitor ~]# ceph auth get client.1 -o ceph.client.1.keyring
      exported keyring for client.1

   2. Copy the client keyring from the Monitor node to the /etc/ceph/ directory on the client node:

      scp root@<monitor>:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

      Replace <monitor> with the Monitor host name or IP, for example:

      [root@client ~]# scp root@192.168.0.1:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

5. Set the appropriate permissions for the keyring file.

   chmod 644 <keyring>

   Specify the path to the keyring, for example:

   [root@client ~]# chmod 644 /etc/ceph/ceph.client.1.keyring

Procedure

1. Create an erasure-coded data pool for Ceph File System:

   ceph osd pool create $DATA_POOL $PG_NUM erasure

   For example, to create an erasure-coded pool named cephfs-data-ec with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-data-ec 64 erasure

2. Create a replicated metadata pool for Ceph File System:

   ceph osd pool create $METADATA_POOL $PG_NUM

   For example, to create a pool named cephfs-metadata with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

3. Enable overwrites on the erasure-coded pool:

   ceph osd pool set $DATA_POOL allow_ec_overwrites true

   For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec:

   [root@monitor ~]# ceph osd pool set cephfs-data-ec allow_ec_overwrites true

4. Create the Ceph File System:

   ceph fs new $FS_EC $METADATA_POOL $DATA_POOL

   Note

   Using an erasure-coded pool for the default data pool is discouraged, but you can use --force to override this default. Specify the name of the Ceph File System, and the metadata and data pools, for example:

   [root@monitor ~]# ceph fs new cephfs-ec cephfs-metadata cephfs-data-ec --force

5. Verify that one or more MDSs enter the active state based on your configuration:

   ceph fs status $FS_EC

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs-ec
   cephfs-ec - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs-metadata | metadata | 4638  | 26.7G |
   |  cephfs-data-ec |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+

6. If you want to add an additional erasure-coded pool, as data pool, to the existing file system,:

   1. Create an erasure-coded data pool for Ceph File System:

      ceph osd pool create $DATA_POOL $PG_NUM erasure

      For example, to create an erasure-coded pool named cephfs-data-ec1 with 64 placement groups:

      [root@monitor ~]# ceph osd pool create cephfs-data-ec1 64 erasure

   2. Enable overwrites on the erasure-coded pool:

      ceph osd pool set $DATA_POOL allow_ec_overwrites true

      For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph osd pool set cephfs-data-ec1 allow_ec_overwrites true

   3. Add the newly created pool to an existing Ceph File System:

      ceph fs add_data_pool $FS_EC $DATA_POOL

      For example, to add an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph fs add_data_pool cephfs-ec cephfs-data-ec1

   4. Verify that one or more MDSs enter the active state based on your configuration.

      ceph fs status $FS_EC

      Specify the name of the Ceph File System, for example:

      [root@monitor ~]# ceph fs status cephfs-ec
      cephfs-ec - 0 clients
      ======
      +------+--------+-------+---------------+-------+-------+
      | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
      +------+--------+-------+---------------+-------+-------+
      |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
      +------+--------+-------+---------------+-------+-------+
      +-----------------+----------+-------+-------+
      |       Pool      |   type   |  used | avail |
      +-----------------+----------+-------+-------+
      | cephfs-metadata | metadata | 4638  | 26.7G |
      |  cephfs-data-ec |   data   |    0  | 26.7G |
      |  cephfs-data-ec1|   data   |    0  | 26.7G |
      +-----------------+----------+-------+-------+
      
      +-------------+
      | Standby MDS |
      +-------------+
      |    node3    |
      |    node2    |
      +-------------+

1. Create a new data pool.

   ceph osd pool create <name> <pg_num>

   Replace:

   • <name> with the name of the pool
   • <pg_num> with the number of placement groups (PGs)

   For example:

   [root@monitor]# ceph osd pool create cephfs_data_ssd 64
   pool 'cephfs_data_ssd' created

2. Add the newly created pool under the control of the Metadata Servers.

   ceph mds add_data_pool <name>

   Replace:

   • <name> with the name of the pool

   For example:

   [root@monitor]# ceph mds add_data_pool cephfs_data_ssd
   added data pool 6 to fsmap

3. Verify that the pool was successfully added:

   [root@monitor]# ceph fs ls
   name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ssd]

4. If you use the cephx authentication, make sure that clients can access the new pool. See Section 3.3, “Creating Ceph File System Client Users” for details.

Procedure

1. Mark the cluster down.

   ceph fs set name cluster_down true

   Replace:

   • name with the name of the Ceph File System you want to remove

   For example:

   [root@monitor]# ceph fs set cephfs cluster_down true
   marked down

2. Display the status of the Ceph File System.

   ceph fs status

   For example:

   [root@monitor]# ceph fs status
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | ceph4 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 2246  |  975G |
   |   cephfs_data   |   data   |    0  |  975G |
   +-----------------+----------+-------+-------+

3. Fail all MDS ranks shown in the status.

   # ceph mds fail rank

   Replace:

   • rank with the rank of the MDS daemons to fail

   For example:

   [root@monitor]# ceph mds fail 0

4. Remove the Ceph File System.

   ceph fs rm name --yes-i-really-mean-it

   Replace:

   • name with the name of the Ceph File System you want to remove

   For example:

   [root@monitor]# ceph fs rm cephfs --yes-i-really-mean-it

5. Verify that the file system has been successfully removed.

   [root@monitor]# ceph fs ls

6. Optional. Remove data and metadata pools associated with the removed file system. See the Delete a pool section in the Red Hat Ceph Storage 3 Storage Strategies Guide.