Chapter 5. Mirroring Ceph block devices

Procedure

1. On the cluster where the images originate, enable the exclusive-lock and journaling features on an image.

   1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE[,FEATURE]

      Example

      [root@rbd-client ~]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling

   2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE_NAME

      Example

      [root@rbd-client ~]# rbd feature enable data/image1 exclusive-lock,journaling

   3. To enable exclusive-lock and journaling on all new images by default, add the following setting to the Ceph configuration file:

      rbd_default_features = 125

2. In the site-a cluster, complete the following steps:

   1. On a monitor node, create the user that the rbd-mirror daemon will use to connect to the cluster. The example creates a site-a user and outputs the key to a file named site-a.client.site-a.keyring:

      Syntax

      ceph auth get-or-create client.CLUSTER_NAME mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/CLUSTER_NAME.client.USER_NAME.keyring

      Example

      [root@mon ~]# ceph auth get-or-create client.site-a mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/site-a.client.site-a.keyring

   2. Copy the Ceph configuration file and the newly created key file from the monitor node to the site-b monitor and client nodes.
   3. Rename the Ceph configuration file from ceph.conf to CLUSTER-NAME.conf. In these examples, the file is /etc/ceph/site-a.conf.

3. In the site-b cluster, complete the following steps:

   1. On the Ansible administration node, add an [rbdmirrors] group in the Ansible inventory file. The usual inventory file is /etc/ansible/hosts.

   2. Under the [rbdmirrors] group, add the name of the site-b client node on which the rbd-mirror daemon will run. The daemon will pull image changes from site-a to site-b.

      [rbdmirrors]
      ceph-client

   3. Navigate to the /usr/share/ceph-ansible/ directory:

      [root@admin ~]# cd /usr/share/ceph-ansible

   4. Create a new rbdmirrors.yml file by copying group_vars/rbdmirrors.yml.sample to group_vars/rbdmirrors.yml:

      [root@admin ceph-ansible]# cp group_vars/rbdmirrors.yml.sample group_vars/rbdmirrors.yml

   5. Open the group_vars/rbdmirrors.yml file for editing.

   6. Set ceph_rbd_mirror_configure to true. Set ceph_rbd_mirror_pool to the pool in which you want to mirror images. In these examples, data is the name of the pool.

      ceph_rbd_mirror_configure: true
      ceph_rbd_mirror_pool: "data"

   7. By default, ceph-ansible configures mirroring using pool mode, which mirrors all images in a pool. Enable image mode where only images that have mirroring explicitly enabled are mirrored. To enable image mode, set ceph_rbd_mirror_mode to image:

      ceph_rbd_mirror_mode: image

   8. Set a name for the cluster that rbd-mirror will pull from. In these examples, the other cluster is site-a.

      ceph_rbd_mirror_remote_cluster: "site-a"

   9. On the Ansible administration node, set the user name of the key using ceph_rbd_mirror_remote_user. Use the same name you used when you created the key. In these examples the user is named client.site-a.

      ceph_rbd_mirror_remote_user: "client.site-a"

   10. As the ceph-ansible user, run the Ansible playbook:

       • Bare-metal deployments:

         [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rbdmirrors -i hosts

       • Container deployments:

         [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit rbdmirrors -i hosts

4. Explicitly enable mirroring on the desired images in both site-a and site-b clusters:

   Syntax

   Journal-based mirroring:

   rbd mirror image enable POOL/IMAGE

   Snapshot-based mirroring:

   rbd mirror image enable POOL/IMAGE snapshot

   Example

   [root@mon ~]# rbd mirror image enable data/image1
   [root@mon ~]# rbd mirror image enable data/image1 snapshot

   Note

   Repeat this step whenever you want to mirror new image to peer cluster.

5. Verify the mirroring status. Run the following command from a Ceph Monitor node in the site-b cluster:

   Example

   Journal-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   7d486c3f-d5a1-4bee-ae53-6c4f1e0c8eac
     state:       up+replaying 1
     description: replaying, master_position=[object_number=3, tag_tid=1, entry_tid=3], mirror_position=[object_number=3, tag_tid=1, entry_tid=3], entries_behind_master=0
     last_update: 2019-04-22 13:19:27

   Snapshot-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   06acc9e6-a63d-4aa1-bd0d-4f3a79b0ae33
     state:       up+replaying 1
     description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642689843,"remote_snapshot_timestamp":1642689843,"replay_state":"idle"}
     service:     admin on ceph-rbd2-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:41:57

   1 1

   If images are in the state up+replaying, then mirroring is functioning properly.

   Note

   Based on the connection between the sites, mirroring can take a long time to sync the images.

Procedure

1. On the cluster where the images originate, enable the exclusive-lock and journaling features on an image.

   1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE[,FEATURE]

      Example

      [root@rbd-client ~]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling

   2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE_NAME

      Example

      [root@rbd-client ~]# rbd feature enable data/image1 exclusive-lock,journaling

   3. To enable exclusive-lock and journaling on all new images by default, add the following setting to the Ceph configuration file:

      rbd_default_features = 125

2. In the site-a cluster, complete the following steps:

   1. On a monitor node, create the user the rbd-mirror daemon will use to connect to the cluster. The example creates a site-a user and outputs the key to a file named site-a.client.site-a.keyring, and the Ceph configuration file is /etc/ceph/site-a.conf.

      Syntax

      ceph auth get-or-create client.PRIMARY_CLUSTER_NAME mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/PRIMARY_CLUSTER_NAME.client.USER_NAME.keyring -c /etc/ceph/PRIMARY_CLUSTER_NAME.conf

      Example

      [root@mon ~]# ceph auth get-or-create client.site-a mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/site-a.client.site-a.keyring -c /etc/ceph/site-a.conf

   2. Copy the keyring to the site-b cluster. Copy the file to the client node in the site-b cluster that the rbd-daemon will run on. Save the file to /etc/ceph/site-a.client.site-a.keyring:

      Syntax

      scp /etc/ceph/PRIMARY_CLUSTER_NAME.client.USER_NAME.keyring root@SECONDARY_CLIENT_NODE_NAME:/etc/ceph/PRIMARY_CLUSTER_NAME.client.USER_NAME.keyring

      Example

      [root@mon ~]# scp /etc/ceph/site-a.client.site-a.keyring root@client.site-b:/etc/ceph/site-a.client.site-a.keyring

   3. Copy the Ceph configuration file from the monitor node to the site-b monitor node and client nodes. The Ceph configuration file in this example is /etc/ceph/site-a.conf.

      Syntax

      scp /etc/ceph/PRIMARY_CLUSTER_NAME.conf root@SECONDARY_MONITOR_NODE_NAME:/etc/ceph/PRIMARY_CLUSTER_NAME.conf
      scp /etc/ceph/PRIMARY_CLUSTER_NAME.conf user@SECONDARY_CLIENT_NODE_NAME:/etc/ceph/PRIMARY_CLUSTER_NAME.conf

      Example

      [root@mon ~]# scp /etc/ceph/site-a.conf root@mon.site-b:/etc/ceph/site-a.conf
      [root@mon ~]# scp /etc/ceph/site-a.conf user@client.site-b:/etc/ceph/site-a.conf

3. In the site-b cluster, complete the following steps:

   1. Configure mirroring from site-a to site-b. On the Ansible administration node, add an [rbdmirrors] group in the Ansible inventory file, usually /usr/share/ceph-ansible/hosts.

   2. Under the [rbdmirrors] group, add the name of a site-b client node that the rbd-mirror daemon will run on. This daemon pulls image changes from site-a to site-b.

      Example

      [rbdmirrors]
      client.site-b

   3. Navigate to the /usr/share/ceph-ansible/ directory:

      [root@admin ~]$ cd /usr/share/ceph-ansible

   4. Create a new rbdmirrors.yml file by copying group_vars/rbdmirrors.yml.sample to group_vars/rbdmirrors.yml:

      [root@admin ceph-ansible]# cp group_vars/rbdmirrors.yml.sample group_vars/rbdmirrors.yml

   5. Open for editing the group_vars/rbdmirrors.yml file.

   6. Set ceph_rbd_mirror_configure to true, and set ceph_rbd_mirror_pool to the pool you want to mirror images in. In these examples, data is the name of the pool.

      ceph_rbd_mirror_configure: true
      ceph_rbd_mirror_pool: "data"

   7. By default, ceph-ansible configures mirroring using pool mode, which mirrors all images in a pool. Enable image mode where only images that have mirroring explicitly enabled are mirrored. To enable image mode, set ceph_rbd_mirror_mode to image:

      ceph_rbd_mirror_mode: image

   8. Set a name for the cluster that rbd-mirror in the group_vars/rbdmirrors.yml file. In these examples, the other cluster is site-a.

      ceph_rbd_mirror_remote_cluster: "site-a"

   9. On the Ansible administration node, set the user name of the key using ceph_rbd_mirror_remote_user in the group_vars/rbdmirrors.yml file. Use the same name you used when you created the key. In these examples the user is named client.site-a.

      ceph_rbd_mirror_remote_user: "client.site-a"

   10. As the ansible user, run the Ansible playbook:

       • Bare-metal deployments:

         [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rbdmirrors -i hosts

       • Container deployments:

         [user@admin ceph-ansible]$ ansible-playbook site-container.yml --limit rbdmirrors -i hosts

4. Verify the mirroring status. Run the following command from a Ceph Monitor node on the site-b cluster:

   Example

   Journal-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   7d486c3f-d5a1-4bee-ae53-6c4f1e0c8eac
     state:       up+replaying 1
     description: replaying, master_position=[object_number=3, tag_tid=1, entry_tid=3], mirror_position=[object_number=3, tag_tid=1, entry_tid=3], entries_behind_master=0
     last_update: 2021-04-22 13:19:27

   Snapshot-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   06acc9e6-a63d-4aa1-bd0d-4f3a79b0ae33
     state:       up+replaying 1
     description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642689843,"remote_snapshot_timestamp":1642689843,"replay_state":"idle"}
     service:     admin on ceph-rbd2-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:41:57

   1 1

   If images are in the state up+replaying, then mirroring is functioning properly.

   Note

   Based on the connection between the sites, mirroring can take a long time to sync the images.

5. In the site-b cluster, complete the following steps. The steps are largely the same as above:

   1. On a monitor node, create the user the rbd-mirror daemon will use to connect to the cluster. The example creates a site-b user and outputs the key to a file named site-b.client.site-b.keyring, and the Ceph configuration file is /etc/ceph/site-b.conf.

      Syntax

      ceph auth get-or-create client.SECONDARY_CLUSTER_NAME mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/SECONDARY_CLUSTER_NAME.client.USER_NAME.keyring -c /etc/ceph/SECONDARY_CLUSTER_NAME.conf

      Example

      [root@mon ~]# ceph auth get-or-create client.site-b mon 'profile rbd' osd 'profile rbd pool=data' -o /etc/ceph/site-b.client.site-b.keyring -c /etc/ceph/site-b.conf

   2. Copy the keyring to the site-a cluster. Copy the file to the client node in the site-a cluster that the rbd-daemon will run on. Save the file to /etc/ceph/site-b.client.site-b.keyring:

      Syntax

      scp /etc/ceph/SECONDARY_CLUSTER_NAME.client.USER_NAME.keyring root@PRIMARY_CLIENT_NODE_NAME:/etc/ceph/SECONDARY_CLUSTER_NAME.client.USER_NAME.keyring

      Example

      [root@mon ~]# scp /etc/ceph/site-b.client.site-b.keyring root@client.site-a:/etc/ceph/site-b.client.site-b.keyring

   3. Copy the Ceph configuration file from the monitor node to the site-a monitor node and client nodes. The Ceph configuration file in this example is /etc/ceph/site-b.conf.

      Syntax

      scp /etc/ceph/SECONDARY_CLUSTER_NAME.conf root@PRIMARY_MONITOR_NODE_NAME:/etc/ceph/SECONDARY_CLUSTER_NAME.conf
      scp /etc/ceph/SECONDARY_CLUSTER_NAME.conf user@PRIMARY_CLIENT_NODE_NAME:/etc/ceph/SECONDARY_CLUSTER_NAME.conf

      Example

      [root@mon ~]# scp /etc/ceph/site-b.conf root@mon.site-a:/etc/ceph/site-b.conf
      [root@mon ~]# scp /etc/ceph/site-b.conf user@client.site-a:/etc/ceph/site-b.conf

6. In the site-a cluster, complete the following steps:

   1. Configure mirroring from site-b to site-a. On the Ansible administration node, add an [rbdmirrors] group in the Ansible inventory file, usually /usr/share/ceph-ansible/hosts.

   2. Under the [rbdmirrors] group, add the name of a site-a client node that the rbd-mirror daemon will run on. This daemon pulls image changes from site-b to site-a.

      Example

      [rbdmirrors]
      client.site-a

   3. Navigate to the /usr/share/ceph-ansible/ directory:

      [root@admin ~]# cd /usr/share/ceph-ansible

   4. Create a new rbdmirrors.yml file by copying group_vars/rbdmirrors.yml.sample to group_vars/rbdmirrors.yml:

      [root@admin ceph-ansible]# cp group_vars/rbdmirrors.yml.sample group_vars/rbdmirrors.yml

   5. Open for editing the group_vars/rbdmirrors.yml file.

   6. Set ceph_rbd_mirror_configure to true, and set ceph_rbd_mirror_pool to the pool you want to mirror images in. In these examples, data is the name of the pool.

      ceph_rbd_mirror_configure: true
      ceph_rbd_mirror_pool: "data"

   7. By default, ceph-ansible configures mirroring using pool mode which mirrors all images in a pool. Enable image mode where only images that have mirroring explicitly enabled are mirrored. To enable image mode, set ceph_rbd_mirror_mode to image:

      ceph_rbd_mirror_mode: image

   8. On the Ansible administration node, set a name for the cluster that rbd-mirror in the group_vars/rbdmirrors.yml file. Following the examples, the other cluster is named site-b.

      ceph_rbd_mirror_remote_cluster: "site-b"

   9. On the Ansible administration node, set the user name of the key using ceph_rbd_mirror_remote_user in group_vars/rbdmirrors.yml file. In these examples the user is named client.site-b.

      ceph_rbd_mirror_remote_user: "client.site-b"

   10. As the Ansible user on the administration node, run the Ansible playbook:

       • Bare-metal deployments:

         [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rbdmirrors -i hosts

       • Container deployments:

         [user@admin ceph-ansible]$ ansible-playbook site-container.yml --limit rbdmirrors -i hosts

7. Explicitly enable mirroring on the desired images in both site-a and site-b clusters:

   Syntax

   Journal-based mirroring:

   rbd mirror image enable POOL/IMAGE

   Snapshot-based mirroring:

   rbd mirror image enable POOL/IMAGE snapshot

   Example

   [root@mon ~]# rbd mirror image enable data/image1
   [root@mon ~]# rbd mirror image enable data/image1 snapshot

   Note

   Repeat this step whenever you want to mirror new image to peer cluster.

8. Verify the mirroring status. Run the following command from the client node on the site-a cluster:

   Example

   Journal-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   08027096-d267-47f8-b52e-59de1353a034
     state:       up+stopped 1
     description: local image is primary
     last_update: 2021-04-16 15:45:31

   Snapshot-based mirroring:

   [root@mon ~]# rbd mirror image status data/image1
   image1:
     global_id:   47fd1aae-5f19-4193-a5df-562b5c644ea7
     state:       up+stopped 1
     description: local image is primary
     service:     admin on ceph-rbd1-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:42:54
     peer_sites:
       name: rbd-mirror.site-b
       state: up+replaying
       description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642693094,"remote_snapshot_timestamp":1642693094,"replay_state":"idle"}
       last_update: 2022-01-20 12:42:59
     snapshots:
       5 .mirror.primary.47fd1aae-5f19-4193-a5df-562b5c644ea7.dda146c6-5f21-4e75-ba93-660f6e57e301 (peer_uuids:[bfd09289-c9c9-40c8-b2d3-ead9b6a99a45])

   1 1

   The images should be in state up+stopped. Here, up means the rbd-mirror daemon is running and stopped means the image is not a target for replication from another cluster. This is because the images are primary on this cluster.

Procedure

1. Install the rbd-mirror package on the client node connected to the site-b storage cluster:

   Red Hat Enterprise Linux 7

   [root@rbd-client ~]# yum install rbd-mirror

   Red Hat Enterprise Linux 8

   [root@rbd-client ~]# dnf install rbd-mirror

   Note

   The package is provided by the Red Hat Ceph Storage Tools repository.

2. Enable the exclusive-lock, and journaling features on an image.

   1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE [,FEATURE]

      Example

      [root@rbd-client ~]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling

   2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE [,FEATURE]

      Example

      [root@rbd-client ~]# rbd feature enable data/image1 exclusive-lock,journaling

   3. To enable exclusive-lock and journaling on all new images by default, add the following setting to the Ceph configuration file:

      rbd_default_features = 125

3. Choose the mirroring mode, either pool or image mode.

   Important

   Use image mode for snapshot-based mirroring.

   1. Enabling pool mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE

      Example

      [root@rbd-client ~]# rbd mirror pool enable data pool

      This example enables mirroring of the whole pool named data.

   2. Enabling image mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE

      Example

      [root@rbd-client ~]# rbd mirror pool enable data image

      This example enables image mode mirroring on the pool named data.

   3. Verify that mirroring has been successfully enabled:

      Syntax

      rbd mirror pool info POOL_NAME

      Example

      [root@rbd-client ~]# rbd mirror pool info data
      Mode: image
      Site Name: 94cbd9ca-7f9a-441a-ad4b-52a33f9b7148
      
      Peer Sites: none

4. In the site-a cluster, complete the following steps:

   1. On the Ceph client node, create a user:

      Syntax

      ceph auth get-or-create client.PRIMARY_CLUSTER_NAME mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring

      Example

      [root@rbd-client-site-a ~]# ceph auth get-or-create client.rbd-mirror.site-a mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-a.keyring

   2. Copy keyring to site-b cluster:

      Syntax

      scp /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring root@SECONDARY_CLUSTER:_PATH_

      Example

      [root@rbd-client-site-a ~]# scp /etc/ceph/ceph.client.rbd-mirror.site-a.keyring root@rbd-client-site-b:/etc/ceph/

   3. On a Ceph client node, bootstrap the storage cluster peers.

      1. Register the storage cluster peer to the pool:

         Syntax

         rbd mirror pool peer bootstrap create --site-name LOCAL_SITE_NAME POOL_NAME > PATH_TO_BOOTSTRAP_TOKEN

         Example

         [root@rbd-client-site-a ~]# rbd mirror pool peer bootstrap create --site-name rbd-mirror.site-a data > /root/bootstrap_token_rbd-mirror.site-a

         Note

         This example bootstrap command creates the client.rbd-mirror-peer Ceph user.

      2. Copy the bootstrap token file to the site-b storage cluster.

         Syntax

         scp PATH_TO_BOOTSTRAP_TOKEN root@SECONDARY_CLUSTER:/root/

         Example

         [root@rbd-client-site-a ~]# scp /root/bootstrap_token_site-a root@ceph-rbd2:/root/

5. In the site-b cluster, complete the following steps:

   1. On the client node, create a user:

      Syntax

      ceph auth get-or-create client.SECONDARY_CLUSTER_NAME mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.SECONDARY_CLUSTER_NAME.keyring

      Example

      [root@rbd-client-site-b ~]# ceph auth get-or-create client.rbd-mirror.site-b mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-b.keyring

   2. Copy keyring to the site-a cluster, the Ceph client node:

      Syntax

      scp /etc/ceph/ceph.SECONDARY_CLUSTER_NAME.keyring root@PRIMARY_CLUSTER:_PATH_

      Example

      [root@rbd-client-site-b ~]# scp /etc/ceph/ceph.client.rbd-mirror.site-b.keyring root@rbd-client-site-a:/etc/ceph/

   3. Import the bootstrap token:

      Syntax

      rbd mirror pool peer bootstrap import --site-name LOCAL_SITE_NAME --direction rx-only POOL_NAME PATH_TO_BOOTSTRAP_TOKEN

      Example

      [root@rbd-client-site-b ~]# rbd mirror pool peer bootstrap import --site-name rbd-mirror.site-b --direction rx-only data /root/bootstrap_token_rbd-mirror.site-a

      Note

      For one-way RBD mirroring, you must use the --direction rx-only argument, as two-way mirroring is the default when bootstrapping peers.

   4. Enable and start the rbd-mirror daemon on client node:

      Syntax

      systemctl enable ceph-rbd-mirror.target
      systemctl enable ceph-rbd-mirror@rbd-mirror.CLIENT_ID
      systemctl start ceph-rbd-mirror@rbd-mirror.CLIENT_ID

      Replace CLIENT_ID with the Ceph user created earlier.

      Example

      [root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror.target
      [root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror@rbd-mirror.site-a
      [root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror@rbd-mirror.site-a

   Important

   Each rbd-mirror daemon must have a unique Client ID.

6. To verify the mirroring status, run the following command from a Ceph Monitor node in the site-a and site-b clusters:

   Syntax

   rbd mirror image status POOL_NAME/IMAGE_NAME

   Example

   Journal-based mirroring:

   [root@mon-site-a ~]# rbd mirror image status data/image1
   image1:
     global_id:   08027096-d267-47f8-b52e-59de1353a034
     state:       up+stopped 1
     description: local image is primary
     last_update: 2021-04-22 13:45:31

   Snapshot-based mirroring:

   [root@mon-site-a ~]# rbd mirror image status data/image1
   image1:
     global_id:   47fd1aae-5f19-4193-a5df-562b5c644ea7
     state:       up+stopped 1
     description: local image is primary
     service:     admin on ceph-rbd1-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:42:54
     peer_sites:
       name: rbd-mirror.site-b
       state: up+replaying
       description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642693094,"remote_snapshot_timestamp":1642693094,"replay_state":"idle"}
       last_update: 2022-01-20 12:42:59
     snapshots:
       5 .mirror.primary.47fd1aae-5f19-4193-a5df-562b5c644ea7.dda146c6-5f21-4e75-ba93-660f6e57e301 (peer_uuids:[bfd09289-c9c9-40c8-b2d3-ead9b6a99a45])

   1 1

   Here, up means the rbd-mirror daemon is running, and stopped means this image is not the target for replication from another storage cluster. This is because the image is primary on this storage cluster.

   Example

   Journal-based mirroring:

   [root@mon-site-b ~]# rbd mirror image status data/image1
   image1:
     global_id:   7d486c3f-d5a1-4bee-ae53-6c4f1e0c8eac
     state:       up+replaying 1
     description: replaying, master_position=[object_number=3, tag_tid=1, entry_tid=3], mirror_position=[object_number=3, tag_tid=1, entry_tid=3], entries_behind_master=0
     last_update: 2021-04-22 14:19:27

   Snapshot-based mirroring:

   [root@mon-site-b ~]# rbd mirror image status data/image1
   image1:
     global_id:   06acc9e6-a63d-4aa1-bd0d-4f3a79b0ae33
     state:       up+replaying 1
     description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642689843,"remote_snapshot_timestamp":1642689843,"replay_state":"idle"}
     service:     admin on ceph-rbd2-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:41:57

   1 1

   If images are in the state up+replaying, then mirroring is functioning properly. Here, up means the rbd-mirror daemon is running, and replaying means this image is the target for replication from another storage cluster.

   Note

   Depending on the connection between the sites, mirroring can take a long time to sync the images.

Procedure

1. Install the rbd-mirror package on the client node connected to the site-a storage cluster, and the client node connected to the site-b storage cluster:

   Red Hat Enterprise Linux 7

   [root@rbd-client ~]# yum install rbd-mirror

   Red Hat Enterprise Linux 8

   [root@rbd-client ~]# dnf install rbd-mirror

   Note

   The package is provided by the Red Hat Ceph Storage Tools repository.

2. Enable the exclusive-lock, and journaling features on an image.

   1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE [,FEATURE]

      Example

      [root@rbd-client ~]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling

   2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE [,FEATURE]

      Example

      [root@rbd-client ~]# rbd feature enable data/image1 exclusive-lock,journaling

   3. To enable exclusive-lock and journaling on all new images by default, add the following setting to the Ceph configuration file:

      rbd_default_features = 125

3. Choose the mirroring mode, either pool or image mode.

   Important

   Use image mode for snapshot-based mirroring.

   1. Enabling pool mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE

      Example

      [root@rbd-client ~]# rbd mirror pool enable data pool

      This example enables mirroring of the whole pool named data.

   2. Enabling image mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE

      Example

      [root@rbd-client ~]# rbd mirror pool enable data image

      This example enables image mode mirroring on the pool named data.

   3. Verify that mirroring has been successfully enabled:

      Syntax

      rbd mirror pool info POOL_NAME

      Example

      [root@rbd-client ~]# rbd mirror pool info data
      Mode: image
      Site Name: 94cbd9ca-7f9a-441a-ad4b-52a33f9b7148
      
      Peer Sites: none

4. In the site-a cluster, complete the following steps:

   1. On the Ceph client node, create a user:

      Syntax

      ceph auth get-or-create client.PRIMARY_CLUSTER_NAME mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring

      Example

      [root@rbd-client-site-a ~]# ceph auth get-or-create client.rbd-mirror.site-a mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-a.keyring

   2. Copy keyring to site-b cluster:

      Syntax

      scp /etc/ceph/ceph.PRIMARY_CLUSTER_NAME.keyring root@SECONDARY_CLUSTER:_PATH_

      Example

      [root@rbd-client-site-a ~]# scp /etc/ceph/ceph.client.rbd-mirror.site-a.keyring root@rbd-client-site-b:/etc/ceph/

   3. On a Ceph client node, bootstrap the storage cluster peers.

      1. Register the storage cluster peer to the pool:

         Syntax

         rbd mirror pool peer bootstrap create --site-name LOCAL_SITE_NAME POOL_NAME > PATH_TO_BOOTSTRAP_TOKEN

         Example

         [root@rbd-client-site-a ~]# rbd mirror pool peer bootstrap create --site-name rbd-mirror.site-a data > /root/bootstrap_token_rbd-mirror.site-a

         Note

         This example bootstrap command creates the client.rbd-mirror-peer Ceph user.

      2. Copy the bootstrap token file to the site-b storage cluster.

         Syntax

         scp PATH_TO_BOOTSTRAP_TOKEN root@SECONDARY_CLUSTER:/root/

         Example

         [root@rbd-client-site-a ~]# scp /root/bootstrap_token_site-a root@ceph-rbd2:/root/

5. In the site-b cluster, complete the following steps:

   1. On the client node, create a user:

      Syntax

      ceph auth get-or-create client.SECONDARY_CLUSTER_NAME mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.SECONDARY_CLUSTER_NAME.keyring

      Example

      [root@rbd-client-site-b ~]# ceph auth get-or-create client.rbd-mirror.site-b mon 'profile rbd-mirror' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.site-b.keyring

   2. Copy keyring to the site-a cluster, the Ceph client node:

      Syntax

      scp /etc/ceph/ceph.SECONDARY_CLUSTER_NAME.keyring root@PRIMARY_CLUSTER:_PATH_

      Example

      [root@rbd-client-site-b ~]# scp /etc/ceph/ceph.client.rbd-mirror.site-b.keyring root@rbd-client-site-a:/etc/ceph/

   3. Import the bootstrap token:

      Syntax

      rbd mirror pool peer bootstrap import --site-name LOCAL_SITE_NAME --direction rx-tx POOL_NAME PATH_TO_BOOTSTRAP_TOKEN

      Example

      [root@rbd-client-site-b ~]# rbd mirror pool peer bootstrap import --site-name rbd-mirror.site-b --direction rx-tx data /root/bootstrap_token_rbd-mirror.site-a

      Note

      The --direction argument is optional, as two-way mirroring is the default when bootstrapping peers.

6. Enable and start the rbd-mirror daemon on the primary and secondary client nodes:

   Syntax

   systemctl enable ceph-rbd-mirror.target
   systemctl enable ceph-rbd-mirror@rbd-mirror.CLIENT_ID
   systemctl start ceph-rbd-mirror@rbd-mirror.CLIENT_ID

   Replace CLIENT_ID with the Ceph user created earlier.

   Example

   [root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror.target
   [root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror@rbd-mirror.site-a
   [root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror@rbd-mirror.site-a
   [root@rbd-client-site-a ~]# systemctl enable ceph-rbd-mirror@rbd-mirror.site-b
   [root@rbd-client-site-a ~]# systemctl start ceph-rbd-mirror@rbd-mirror.site-b

   In the above example, users are enabled in the primary cluster site-a

   Example

   [root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror.target
   [root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror@rbd-mirror.site-a
   [root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror@rbd-mirror.site-a
   [root@rbd-client-site-b ~]# systemctl enable ceph-rbd-mirror@rbd-mirror.site-b
   [root@rbd-client-site-b ~]# systemctl start ceph-rbd-mirror@rbd-mirror.site-b

   In the above example, users are enabled in the secondary cluster site-b

   Important

   Each rbd-mirror daemon must have a unique Client ID.

7. To verify the mirroring status, run the following command from a Ceph Monitor node in the site-a and site-b clusters:

   Syntax

   rbd mirror image status POOL_NAME/IMAGE_NAME

   Example

   Journal-based mirroring:

   [root@mon-site-a ~]# rbd mirror image status data/image1
   image1:
     global_id:   08027096-d267-47f8-b52e-59de1353a034
     state:       up+stopped 1
     description: local image is primary
     last_update: 2021-04-22 13:45:31

   Snapshot-based mirroring:

   [root@mon-site-a ~]# rbd mirror image status data/image1
   image1:
     global_id:   47fd1aae-5f19-4193-a5df-562b5c644ea7
     state:       up+stopped 1
     description: local image is primary
     service:     admin on ceph-rbd1-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:42:54
     peer_sites:
       name: rbd-mirror.site-b
       state: up+replaying
       description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642693094,"remote_snapshot_timestamp":1642693094,"replay_state":"idle"}
       last_update: 2022-01-20 12:42:59
     snapshots:
       5 .mirror.primary.47fd1aae-5f19-4193-a5df-562b5c644ea7.dda146c6-5f21-4e75-ba93-660f6e57e301 (peer_uuids:[bfd09289-c9c9-40c8-b2d3-ead9b6a99a45])

   1 1

   Here, up means the rbd-mirror daemon is running, and stopped means this image is not the target for replication from another storage cluster. This is because the image is primary on this storage cluster.

   Example

   Journal-based mirroring:

   [root@mon-site-b ~]# rbd mirror image status data/image1
   image1:
     global_id:   7d486c3f-d5a1-4bee-ae53-6c4f1e0c8eac
     state:       up+replaying 1
     description: replaying, master_position=[object_number=3, tag_tid=1, entry_tid=3], mirror_position=[object_number=3, tag_tid=1, entry_tid=3], entries_behind_master=0
     last_update: 2021-04-22 14:19:27

   Snapshot-based mirroring:

   [root@mon-site-b ~]# rbd mirror image status data/image1
   image1:
     global_id:   06acc9e6-a63d-4aa1-bd0d-4f3a79b0ae33
     state:       up+replaying 1
     description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1642689843,"remote_snapshot_timestamp":1642689843,"replay_state":"idle"}
     service:     admin on ceph-rbd2-vasi-43-5hwia4-node2
     last_update: 2022-01-20 12:41:57

   1 1

   If images are in the state up+replaying, then mirroring is functioning properly. Here, up means the rbd-mirror daemon is running, and replaying means this image is the target for replication from another storage cluster.

   Note

   Depending on the connection between the sites, mirroring can take a long time to sync the images.