Chapter 3. Deploying Ceph File Systems

1. Create two pools, one for storing data and one for storing metadata:

   ceph osd pool create <name> <pg_num>

   Specify the pool name and the number of placement groups (PGs), for example:

   [root@monitor ~]# ceph osd pool create cephfs-data 64
   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

   Typically, the metadata pool can start with a conservative number of PGs as it will generally have far fewer objects than the data pool. It is possible to increase the number of PGs if needed. Recommended metadata pool sizes range from 64 PGs to 512 PGs. Size the data pool proportional to the number and sizes of files you expect in the file system.

   Important

   For the metadata pool, consider using

   • A higher replication level because any data loss to this pool can make the whole file system inaccessible
   • Storage with lower latency such as Solid-state Drive (SSD) disks because this directly affects the observed latency of file system operations on clients

2. Create the Ceph File System:

   ceph fs new <name> <metadata-pool> <data-pool>

   Specify the name of the Ceph File System, the metadata and data pool, for example:

   [root@monitor ~]# ceph fs new cephfs cephfs-metadata cephfs-data

3. Verify that one or more MDSs enter to the active state based on you configuration.

   ceph fs status <name>

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+----

1. On a Monitor host, create a client user.

   ceph auth get-or-create client.<id> <capabilities>

   Specify the client ID and desired capabilities.

   • To restrict the client to only write to and read from a particular pool in the cluster:

     ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=<pool>'

     For example, to restrict the client to only write to and read from the data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=data'

   • To prevent the client from modifying the pool that is used for files and directories:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=<pool>'

     For example, to prevent the client from modifying data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=data'

     Note

     Do not create capabilities for the metadata pool, as Ceph File System clients do not have access to it.

2. Verify the created key:

   ceph auth get client.<id>

   For example:

   [root@monitor ~]# ceph auth get client.1

3. If you plan to use the kernel client, create a secret file using the key retrieved from the previous step.

   On the client node, copy the string after key = into /etc/ceph/ceph.client.<id>.secret:

   For example, if the client ID is 1 add a single line to /etc/ceph/ceph.client.1.secret with the key:

   [root@client ~]# cat /etc/ceph/ceph.client.1.secret
   AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==

   Important

   Do not include the space in between key = and the string or else mounting will not work.

4. If you plan to use the File System in User Space (FUSE) client, copy the keyring to the client.

   1. On the Monitor node, export the keyring to a file:

      # ceph auth get client.<id> -o ceph.client.<id>.keyring

      For example, if the client ID is 1:

      [root@monitor ~]# ceph auth get client.1 -o ceph.client.1.keyring
      exported keyring for client.1

   2. Copy the client keyring from the Monitor node to the /etc/ceph/ directory on the client node:

      scp root@<monitor>:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

      Replace <monitor> with the Monitor host name or IP, for example:

      [root@client ~]# scp root@192.168.0.1:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

5. Set the appropriate permissions for the keyring file.

   chmod 644 <keyring>

   Specify the path to the keyring, for example:

   [root@client ~]# chmod 644 /etc/ceph/ceph.client.1.keyring

Procedure

1. Create an erasure-coded data pool for Ceph File System:

   ceph osd pool create $DATA_POOL $PG_NUM erasure

   For example, to create an erasure-coded pool named cephfs-data-ec with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-data-ec 64 erasure

2. Create a replicated metadata pool for Ceph File System:

   ceph osd pool create $METADATA_POOL $PG_NUM

   For example, to create a pool named cephfs-metadata with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

3. Enable overwrites on the erasure-coded pool:

   ceph osd pool set $DATA_POOL allow_ec_overwrites true

   For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec:

   [root@monitor ~]# ceph osd pool set cephfs-data-ec allow_ec_overwrites true

4. Create the Ceph File System:

   ceph fs new $FS_EC $METADATA_POOL $DATA_POOL

   Note

   Using an erasure-coded pool for the default data pool is discouraged, but you can use --force to override this default. Specify the name of the Ceph File System, and the metadata and data pools, for example:

   [root@monitor ~]# ceph fs new cephfs-ec cephfs-metadata cephfs-data-ec --force

5. Verify that one or more MDSs enter the active state based on your configuration:

   ceph fs status $FS_EC

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs-ec
   cephfs-ec - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs-metadata | metadata | 4638  | 26.7G |
   |  cephfs-data-ec |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+

6. If you want to add an additional erasure-coded pool, as data pool, to the existing file system,:

   1. Create an erasure-coded data pool for Ceph File System:

      ceph osd pool create $DATA_POOL $PG_NUM erasure

      For example, to create an erasure-coded pool named cephfs-data-ec1 with 64 placement groups:

      [root@monitor ~]# ceph osd pool create cephfs-data-ec1 64 erasure

   2. Enable overwrites on the erasure-coded pool:

      ceph osd pool set $DATA_POOL allow_ec_overwrites true

      For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph osd pool set cephfs-data-ec1 allow_ec_overwrites true

   3. Add the newly created pool to an existing Ceph File System:

      ceph fs add_data_pool $FS_EC $DATA_POOL

      For example, to add an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph fs add_data_pool cephfs-ec cephfs-data-ec1

   4. Verify that one or more MDSs enter the active state based on your configuration.

      ceph fs status $FS_EC

      Specify the name of the Ceph File System, for example:

      [root@monitor ~]# ceph fs status cephfs-ec
      cephfs-ec - 0 clients
      ======
      +------+--------+-------+---------------+-------+-------+
      | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
      +------+--------+-------+---------------+-------+-------+
      |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
      +------+--------+-------+---------------+-------+-------+
      +-----------------+----------+-------+-------+
      |       Pool      |   type   |  used | avail |
      +-----------------+----------+-------+-------+
      | cephfs-metadata | metadata | 4638  | 26.7G |
      |  cephfs-data-ec |   data   |    0  | 26.7G |
      |  cephfs-data-ec1|   data   |    0  | 26.7G |
      +-----------------+----------+-------+-------+
      
      +-------------+
      | Standby MDS |
      +-------------+
      |    node3    |
      |    node2    |
      +-------------+