Chapter 3. Deploying Ceph File Systems

1. Create two pools, one for storing data and one for storing metadata:

   ceph osd pool create <name> <pg_num>

   Copy to Clipboard Toggle word wrap

   Specify the pool name and the number of placement groups (PGs), for example:

   [root@monitor ~]# ceph osd pool create cephfs-data 64
   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

   Copy to Clipboard Toggle word wrap

   Typically, the metadata pool can start with a conservative number of PGs as it will generally have far fewer objects than the data pool. It is possible to increase the number of PGs if needed. Recommended metadata pool sizes range from 64 PGs to 512 PGs. Size the data pool proportional to the number and sizes of files you expect in the file system.

   Important

   For the metadata pool, consider using

   • A higher replication level because any data loss to this pool can make the whole file system inaccessible
   • Storage with lower latency such as Solid-state Drive (SSD) disks because this directly affects the observed latency of file system operations on clients

2. Create the Ceph File System:

   ceph fs new <name> <metadata-pool> <data-pool>

   Copy to Clipboard Toggle word wrap

   Specify the name of the Ceph File System, the metadata and data pool, for example:

   [root@monitor ~]# ceph fs new cephfs cephfs-metadata cephfs-data

   Copy to Clipboard Toggle word wrap

3. Verify that one or more MDSs enter to the active state based on you configuration.

   ceph fs status <name>

   Copy to Clipboard Toggle word wrap

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+----

   Copy to Clipboard Toggle word wrap

1. On a Monitor host, create a client user.

   ceph auth get-or-create client.<id> <capabilities>

   Copy to Clipboard Toggle word wrap

   Specify the client ID and desired capabilities.

   • To restrict the client to only write to and read from a particular pool in the cluster:

     ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=<pool>'

     Copy to Clipboard Toggle word wrap

     For example, to restrict the client to only write to and read from the data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow rw' osd 'allow rw pool=data'

     Copy to Clipboard Toggle word wrap

   • To prevent the client from modifying the pool that is used for files and directories:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=<pool>'

     Copy to Clipboard Toggle word wrap

     For example, to prevent the client from modifying data pool:

     [root@monitor ~]# ceph auth get-or-create client.1 mon 'allow r' mds 'allow r' osd 'allow r pool=data'

     Copy to Clipboard Toggle word wrap
     Note

     Do not create capabilities for the metadata pool, as Ceph File System clients do not have access to it.

2. Verify the created key:

   ceph auth get client.<id>

   Copy to Clipboard Toggle word wrap

   For example:

   [root@monitor ~]# ceph auth get client.1

   Copy to Clipboard Toggle word wrap

3. If you plan to use the kernel client, create a secret file using the key retrieved from the previous step.

   On the client node, copy the string after key = into /etc/ceph/ceph.client.<id>.secret:

   For example, if the client ID is 1 add a single line to /etc/ceph/ceph.client.1.secret with the key:

   [root@client ~]# cat /etc/ceph/ceph.client.1.secret
   AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==

   Copy to Clipboard Toggle word wrap
   Important

   Do not include the space in between key = and the string or else mounting will not work.

4. If you plan to use the File System in User Space (FUSE) client, copy the keyring to the client.

   1. On the Monitor node, export the keyring to a file:

      # ceph auth get client.<id> -o ceph.client.<id>.keyring

      Copy to Clipboard Toggle word wrap

      For example, if the client ID is 1:

      [root@monitor ~]# ceph auth get client.1 -o ceph.client.1.keyring
      exported keyring for client.1

      Copy to Clipboard Toggle word wrap

   2. Copy the client keyring from the Monitor node to the /etc/ceph/ directory on the client node:

      scp root@<monitor>:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

      Copy to Clipboard Toggle word wrap

      Replace <monitor> with the Monitor host name or IP, for example:

      [root@client ~]# scp root@192.168.0.1:/root/ceph.client.1.keyring /etc/ceph/ceph.client.1.keyring

      Copy to Clipboard Toggle word wrap

5. Set the appropriate permissions for the keyring file.

   chmod 644 <keyring>

   Copy to Clipboard Toggle word wrap

   Specify the path to the keyring, for example:

   [root@client ~]# chmod 644 /etc/ceph/ceph.client.1.keyring

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an erasure-coded data pool for Ceph File System:

   ceph osd pool create $DATA_POOL $PG_NUM erasure

   Copy to Clipboard Toggle word wrap

   For example, to create an erasure-coded pool named cephfs-data-ec with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-data-ec 64 erasure

   Copy to Clipboard Toggle word wrap

2. Create a replicated metadata pool for Ceph File System:

   ceph osd pool create $METADATA_POOL $PG_NUM

   Copy to Clipboard Toggle word wrap

   For example, to create a pool named cephfs-metadata with 64 placement groups:

   [root@monitor ~]# ceph osd pool create cephfs-metadata 64

   Copy to Clipboard Toggle word wrap

3. Enable overwrites on the erasure-coded pool:

   ceph osd pool set $DATA_POOL allow_ec_overwrites true

   Copy to Clipboard Toggle word wrap

   For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec:

   [root@monitor ~]# ceph osd pool set cephfs-data-ec allow_ec_overwrites true

   Copy to Clipboard Toggle word wrap

4. Create the Ceph File System:

   ceph fs new $FS_EC $METADATA_POOL $DATA_POOL

   Copy to Clipboard Toggle word wrap
   Note

   Using an erasure-coded pool for the default data pool is discouraged, but you can use --force to override this default. Specify the name of the Ceph File System, and the metadata and data pools, for example:

   [root@monitor ~]# ceph fs new cephfs-ec cephfs-metadata cephfs-data-ec --force

   Copy to Clipboard Toggle word wrap

5. Verify that one or more MDSs enter the active state based on your configuration:

   ceph fs status $FS_EC

   Copy to Clipboard Toggle word wrap

   Specify the name of the Ceph File System, for example:

   [root@monitor ~]# ceph fs status cephfs-ec
   cephfs-ec - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+
   | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
   +------+--------+-------+---------------+-------+-------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
   +------+--------+-------+---------------+-------+-------+
   +-----------------+----------+-------+-------+
   |       Pool      |   type   |  used | avail |
   +-----------------+----------+-------+-------+
   | cephfs-metadata | metadata | 4638  | 26.7G |
   |  cephfs-data-ec |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+

   Copy to Clipboard Toggle word wrap

6. If you want to add an additional erasure-coded pool, as data pool, to the existing file system,:

   1. Create an erasure-coded data pool for Ceph File System:

      ceph osd pool create $DATA_POOL $PG_NUM erasure

      Copy to Clipboard Toggle word wrap

      For example, to create an erasure-coded pool named cephfs-data-ec1 with 64 placement groups:

      [root@monitor ~]# ceph osd pool create cephfs-data-ec1 64 erasure

      Copy to Clipboard Toggle word wrap

   2. Enable overwrites on the erasure-coded pool:

      ceph osd pool set $DATA_POOL allow_ec_overwrites true

      Copy to Clipboard Toggle word wrap

      For example, to enable overwrites on an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph osd pool set cephfs-data-ec1 allow_ec_overwrites true

      Copy to Clipboard Toggle word wrap

   3. Add the newly created pool to an existing Ceph File System:

      ceph fs add_data_pool $FS_EC $DATA_POOL

      Copy to Clipboard Toggle word wrap

      For example, to add an erasure-coded pool named cephfs-data-ec1:

      [root@monitor ~]# ceph fs add_data_pool cephfs-ec cephfs-data-ec1

      Copy to Clipboard Toggle word wrap

   4. Verify that one or more MDSs enter the active state based on your configuration.

      ceph fs status $FS_EC

      Copy to Clipboard Toggle word wrap

      Specify the name of the Ceph File System, for example:

      [root@monitor ~]# ceph fs status cephfs-ec
      cephfs-ec - 0 clients
      ======
      +------+--------+-------+---------------+-------+-------+
      | Rank | State  |  MDS  |    Activity   |  dns  |  inos |
      +------+--------+-------+---------------+-------+-------+
      |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |
      +------+--------+-------+---------------+-------+-------+
      +-----------------+----------+-------+-------+
      |       Pool      |   type   |  used | avail |
      +-----------------+----------+-------+-------+
      | cephfs-metadata | metadata | 4638  | 26.7G |
      |  cephfs-data-ec |   data   |    0  | 26.7G |
      |  cephfs-data-ec1|   data   |    0  | 26.7G |
      +-----------------+----------+-------+-------+
      
      +-------------+
      | Standby MDS |
      +-------------+
      |    node3    |
      |    node2    |
      +-------------+

      Copy to Clipboard Toggle word wrap