Chapter 11. Ceph File System mirrors

Procedure

1. On the source storage cluster, deploy the CephFS mirroring daemon:

   Syntax

   ceph orch apply cephfs-mirror ["NODE_NAME"]

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph orch apply cephfs-mirror "node1.example.com"
   Scheduled cephfs-mirror update...

   Copy to Clipboard Toggle word wrap

   This command creates a Ceph user called, cephfs-mirror, and deploys the cephfs-mirror daemon on the given node.

2. On the target storage cluster, create a user for each CephFS peer:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME CLIENT_NAME / rwps

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph fs authorize cephfs client.mirror_remote / rwps
   [client.mirror_remote]
           key = AQCjZ5Jg739AAxAAxduIKoTZbiFJ0lgose8luQ==

   Copy to Clipboard Toggle word wrap

3. On the source storage cluster, enable the CephFS mirroring module:

   Example

   [root@mon ~]# ceph mgr module enable mirroring

   Copy to Clipboard Toggle word wrap

4. On the source storage cluster, enable mirroring on a Ceph File System:

   Syntax

   ceph fs snapshot mirror enable FILE_SYSTEM_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph fs snapshot mirror enable cephfs

   Copy to Clipboard Toggle word wrap

   1. Optional. To disable snapshot mirroring, use the following command:

      Syntax

      ceph fs snapshot mirror disable FILE_SYSTEM_NAME

      Copy to Clipboard Toggle word wrap

      Example

      [root@mon ~]# ceph fs snapshot mirror disable cephfs

      Copy to Clipboard Toggle word wrap

      Warning

      Disabling snapshot mirroring on a file system removes the configured peers. You have to import the peers again by bootstrapping them.

5. Prepare the target peer storage cluster.

   1. On a target node, enable the mirroring Ceph Manager module:

      Example

      [root@mon ~]# ceph mgr module enable mirroring

      Copy to Clipboard Toggle word wrap

   2. On the same target node, create the peer bootstrap:

      Syntax

      ceph fs snapshot mirror peer_bootstrap create FILE_SYSTEM_NAME CLIENT_NAME SITE_NAME

      Copy to Clipboard Toggle word wrap

      The SITE_NAME is a user-defined string to identify the target storage cluster.

      Example

      [root@mon ~]# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote remote-site
      {"token": "eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ=="}

      Copy to Clipboard Toggle word wrap

      Copy the token string between the double quotes for use in the next step.

6. On the source storage cluster, import the bootstrap token from the target storage cluster:

   Syntax

   ceph fs snapshot mirror peer_bootstrap import FILE_SYSTEM_NAME TOKEN

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==

   Copy to Clipboard Toggle word wrap

7. On the source storage cluster, list the CephFS mirror peers:

   Syntax

   ceph fs snapshot mirror peer_list FILE_SYSTEM_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph fs snapshot mirror peer_list cephfs

   Copy to Clipboard Toggle word wrap

   1. Optional. To remove a snapshot peer, use the following command:

      Syntax

      ceph fs snapshot mirror peer_remove FILE_SYSTEM_NAME PEER_UUID

      Copy to Clipboard Toggle word wrap

      Example

      [root@mon ~]# ceph fs snapshot mirror peer_remove cephfs a2dc7784-e7a1-4723-b103-03ee8d8768f8

      Copy to Clipboard Toggle word wrap

      Note

      See the Viewing the mirror status for a Ceph File System link in the Additional Resources section of this procedure on how to find the peer UUID value.

8. On the source storage cluster, configure a directory for snapshot mirroring:

   Syntax

   ceph fs snapshot mirror add FILE_SYSTEM_NAME PATH

   Copy to Clipboard Toggle word wrap

   Example

   [root@mon ~]# ceph fs snapshot mirror add cephfs /volumes/_nogroup/subvol_1

   Copy to Clipboard Toggle word wrap

   Important

   Only absolute paths inside the Ceph File System are valid.

   Note

   The Ceph Manager mirroring module normalizes the path. For example, the /d1/d2/../dN directories are equivalent to /d1/d2. Once a directory has been added for mirroring, its ancestor directories and subdirectories are prevented from being added for mirroring.

   1. Optional. To stop snapshot mirroring for a directory, use the following command:

      Syntax

      ceph fs snapshot mirror remove FILE_SYSTEM_NAME PATH

      Copy to Clipboard Toggle word wrap

      Example

      [root@mon ~]# ceph fs snapshot mirror remove cephfs /home/user1

      Copy to Clipboard Toggle word wrap

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

   Copy to Clipboard Toggle word wrap

2. Find the Ceph File System ID on the node running the CephFS mirroring daemon:

   Syntax

   ceph --admin-daemon PATH_TO_THE_ASOK_FILE help

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph --admin-daemon /var/run/ceph/1011435c-9e30-4db6-b720-5bf482006e0e/ceph-client.cephfs-mirror.node1.bndvox.asok help
   {
       ...
       "fs mirror peer status cephfs@11 1011435c-9e30-4db6-b720-5bf482006e0e": "get peer mirror status",
       "fs mirror status cephfs@11": "get filesystem mirror status",
       ...
   }

   Copy to Clipboard Toggle word wrap

   The Ceph File System ID in this example is cephfs@11.

3. To view the mirror status:

   Syntax

   ceph --admin-daemon PATH_TO_THE_ASOK_FILE fs mirror status FILE_SYSTEM_NAME@_FILE_SYSTEM_ID

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph --admin-daemon /var/run/ceph/1011435c-9e30-4db6-b720-5bf482006e0e/ceph-client.cephfs-mirror.node1.bndvox.asok fs mirror status cephfs@11
   {
     "rados_inst": "192.168.0.5:0/1476644347",
     "peers": {
         "1011435c-9e30-4db6-b720-5bf482006e0e": { 

   1

   
             "remote": {
                 "client_name": "client.mirror_remote",
                 "cluster_name": "remote-site",
                 "fs_name": "cephfs"
             }
         }
     },
     "snap_dirs": {
         "dir_count": 1
     }
   }

   Copy to Clipboard Toggle word wrap

   1

   This is the unique peer UUID.

4. To view the peer status:

   Syntax

   ceph --admin-daemon PATH_TO_ADMIN_SOCKET fs mirror status FILE_SYSTEM_NAME@FILE_SYSTEM_ID PEER_UUID

   Copy to Clipboard Toggle word wrap

   Example

   [ceph: root@host01 /]# ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@11 1011435c-9e30-4db6-b720-5bf482006e0e
   {
     "/home/user1": {
         "state": "idle", 

   1

   
         "last_synced_snap": {
             "id": 120,
             "name": "snap1",
             "sync_duration": 0.079997898999999997,
             "sync_time_stamp": "274900.558797s"
         },
         "snaps_synced": 2, 

   2

   
         "snaps_deleted": 0, 

   3

   
         "snaps_renamed": 0
     }
   }

   Copy to Clipboard Toggle word wrap

   The state can be one of these three values:

   1

   idle means the directory is currently not being synchronized.

   2

   syncing means the directory is currently being synchronized.

   3

   failed means the directory has hit the upper limit of consecutive failures.