Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 6. Mirroring Ceph block devices

As a storage administrator, you can add another layer of redundancy to Ceph block devices by mirroring data images between Red Hat Ceph Storage clusters. Understanding and using Ceph block device mirroring can provide you protection against data loss, such as a site failure. There are two configurations for mirroring Ceph block devices, one-way mirroring or two-way mirroring, and you can configure mirroring on pools and individual images.

Prerequisites

  • A minimum of two healthy running Red Hat Ceph Storage clusters.
  • Network connectivity between the two storage clusters.
  • Access to a Ceph client node for each Red Hat Ceph Storage cluster.
  • A CephX user with administrator-level capabilities.

6.1. Ceph block device mirroring
Copy link

RADOS Block Device (RBD) mirroring is a process of asynchronous replication of Ceph Block Device images between two or more Ceph storage clusters. By locating a Ceph storage cluster in different geographic locations, RBD Mirroring helps help ensure high availability and disaster recovery by keeping a remote copy of RBD images.

The CRUSH hierarchies for the primary and secondary pools that are used for block device mirroring must have similar capacity and performance. You also need enough network bandwidth to avoid delays during mirroring. For example for journal-based mirroring, if the average write speed to images in the primary cluster is X MB/s, the network must support at least N*X throughput (where N is the number of mirrored images), plus some extra bandwidth (Y%) for safety.

The rbd-mirror daemon handles the synchronization by pulling changes from the primary image in one cluster and writing them to the mirrored image in another cluster.

6.1.1. Modes of RBD mirroring
Copy link

There are 2 modes of RBD mirroring:

Journal-based mirroring
This mode uses the RBD journaling image feature to provide point-in-time, crash-consistent replication between clusters by first recording all write operations in a journal before applying them to the image. The remote cluster replays these journal entries to maintain a consistent, up-to-date replica.

Because every write operation involves writing to both the journal and the image, write latency may almost double when journaling is enabled.

Snapshot-based mirroring
This mode uses RBD mirror snapshots to replicate images by identifying and copying only the changed data and metadata between two snapshots. The remote cluster applies these differences to maintain a consistent replica. The RBD fast-diff image feature accelerates this process by quickly identifying modified data blocks without scanning the entire image. During a failover, the full set of snapshot deltas must be synchronized. Any partial updates are automatically rolled back to ensure image consistency.

The rbd-mirror daemon can run either on a single Ceph storage cluster for one-way mirroring or on two Ceph storage clusters for two-way mirroring that participate in the mirroring relationship.

The rbd-mirror daemon is responsible for synchronizing images from one Ceph storage cluster to another Ceph storage cluster by pulling changes from the remote primary image and writes those changes to the local, non-primary image.

For RBD mirroring to work, either using one-way or two-way replication, a couple of assumptions are made:

  • A pool with the same name exists on both storage clusters.
  • For journal-based mirroring, a pool contains journal-enabled images that you want to mirror.
Important

In one-way or two-way replication, each instance of rbd-mirror must be able to connect to the other Ceph storage cluster simultaneously. Also, the network must have sufficient bandwidth between the two data center sites to handle mirroring.

Mirroring is configured on a per-pool basis with mirror peering storage clusters. Ceph supports two mirroring modes, depending on the type of images in the pool.

Pool Mode
All images in a pool with the journaling feature enabled are mirrored.
Image Mode
Only a specific subset of images within a pool are mirrored. You must enable mirroring for each image separately.

Whether or not an image can be modified depends on its state:

  • You can modify images in the primary state.
  • You can not modify images in the non-primary state.

Images are automatically promoted to primary when mirroring is enabled on an image. The promotion can happen:

  • Implicitly by enabling mirroring in pool mode and only for journal-based images.
  • Explicitly by enabling the mirroring of a specific image.

It is possible to demote primary images and promote non-primary images.

6.2.1. One-way Replication
Copy link

One-way mirroring implies that a primary image or pool of images in one storage cluster gets replicated to a secondary storage cluster. One-way mirroring also supports replicating to multiple secondary storage clusters.

On the secondary storage cluster, the image is the non-primary replicate; that is, Ceph clients cannot write to the image. When data is mirrored from a primary storage cluster to a secondary storage cluster, the rbd-mirror runs only on the secondary storage cluster.

For one-way mirroring to work, a couple of assumptions are made:

  • You have two Red Hat Storage Ceph storage clusters and you want to replicate images from a primary storage cluster to a secondary storage cluster.
  • The secondary storage cluster contains the rbd-mirror daemon that can run on one of the cluster nodes. The rbd-mirror daemon connects to the primary storage cluster to sync images to the secondary storage cluster.

Figure 6.1. One-way mirroring

One-way mirroring
View larger image
One-way mirroring

6.2.2. Two-way Replication
Copy link

Two-way replication adds an rbd-mirror daemon on the primary cluster so images can be demoted on it and promoted on the secondary cluster. Changes can then be made to the images on the secondary cluster and they will be replicated in the reverse direction, from secondary to primary. Both clusters must have rbd-mirror running to allow promoting and demoting images on either cluster. Currently, two-way replication is only supported between two sites.

For two-way mirroring to work, a couple of assumptions are made:

  • You have two storage clusters and you want to be able to replicate images between them in either direction.
  • Both storage clusters have rbd-mirror daemon running. The images that are primary on cluster1 are synced by the rbd-mirror daemon on the remote cluster and the images on the remote cluster are synced to the primary.

Figure 6.2. Two-way mirroring

Two-way mirroring
View larger image
Two-way mirroring

When using one-way replication, you can mirror to multiple secondary storage clusters.

Examples in this section distinguish between two storage clusters by referring to the primary storage cluster with the primary images as site-a, and the secondary storage cluster you are replicating the images to, as site-b. The pool name that is used in these examples is called data.

Prerequisites

  • A minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to a Ceph client node for each storage cluster.
  • A CephX user with administrator-level capabilities.

Procedure

  1. Log into the cephadm shell on both the sites:

    Example

    [root@site-a ~]# cephadm shell
[root@site-b ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. On site-b, schedule the deployment of mirror daemon on the secondary cluster:

    Syntax

    ceph orch apply rbd-mirror --placement=NODENAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@site-b /]# ceph orch apply rbd-mirror --placement=host04
    Copy to Clipboard Toggle word wrap

    Note

    The nodename is the host where you want to configure mirroring in the secondary cluster.

  3. Enable journaling features on an image on site-a.

    1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE FEATURE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling
      Copy to Clipboard Toggle word wrap

      Note

      If exclusive-lock is already enabled, use journaling as the only argument, else it returns the following error: 

      one or more requested features are already enabled
(22) Invalid argument
      Copy to Clipboard Toggle word wrap

    2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE, FEATURE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd feature enable data/image1 exclusive-lock, journaling
      Copy to Clipboard Toggle word wrap

    3. To enable journaling on all new images by default, set the configuration parameter using ceph config set command:

      Example

      [ceph: root@site-a /]# ceph config set global rbd_default_features 125
[ceph: root@site-a /]# ceph config show mon.host01 rbd_default_features
      Copy to Clipboard Toggle word wrap

  4. Choose the mirroring mode, either pool or image mode, on both the storage clusters.

    1. Enabling pool mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool enable data pool
[ceph: root@site-b /]# rbd mirror pool enable data pool
      Copy to Clipboard Toggle word wrap

      This example enables mirroring of the whole pool named data.

    2. Enabling image mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool enable data image
[ceph: root@site-b /]# rbd mirror pool enable data image
      Copy to Clipboard Toggle word wrap

      This example enables image mode mirroring on the pool named data.

      Note

      To enable mirroring on specific images in a pool, see the Enabling image mirroring section in the Red Hat Ceph Storage Block Device Guide for more details.

    3. Verify that mirroring has been successfully enabled at both the sites:

      Syntax

      rbd mirror pool info POOL_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool info data
Mode: pool
Site Name: c13d8065-b33d-4cb5-b35f-127a02768e7f

Peer Sites: none

[ceph: root@site-b /]# rbd mirror pool info data
Mode: pool
Site Name: a4c667e2-b635-47ad-b462-6faeeee78df7

Peer Sites: none
      Copy to Clipboard Toggle word wrap

  5. On a Ceph client node, bootstrap the storage cluster peers.

    1. Create Ceph user accounts, and register the storage cluster peer to the pool:

      Syntax

      rbd mirror pool peer bootstrap create --site-name PRIMARY_LOCAL_SITE_NAME POOL_NAME > PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-a /]# rbd mirror pool peer bootstrap create --site-name site-a data > /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      This example bootstrap command creates the client.rbd-mirror.site-a and the client.rbd-mirror-peer Ceph users.

    2. Copy the bootstrap token file to the site-b storage cluster.

    3. Import the bootstrap token on the site-b storage cluster:

      Syntax

      rbd mirror pool peer bootstrap import --site-name SECONDARY_LOCAL_SITE_NAME --direction rx-only POOL_NAME PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-b /]# rbd mirror pool peer bootstrap import --site-name site-b --direction rx-only data /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      For one-way RBD mirroring, you must use the --direction rx-only argument, as two-way mirroring is the default when bootstrapping peers.

Verification steps

  1. To verify the mirroring status, run the following command from a Ceph Monitor node on the primary and secondary sites:

    Syntax

    rbd mirror image status POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@mon-site-a /]# rbd mirror image status data/image1
image1:
  global_id:   c13d8065-b33d-4cb5-b35f-127a02768e7f
  state:       up+stopped
  description: remote image is non-primary
  service:     host03.yuoosv on host03
  last_update: 2021-10-06 09:13:58
    Copy to Clipboard Toggle word wrap

    Here, up means the rbd-mirror daemon is running, and stopped means this image is not the target for replication from another storage cluster. This is because the image is primary on this storage cluster.

    Example

    [ceph: root@mon-site-b /]# rbd mirror image status data/image1
image1:
  global_id:   c13d8065-b33d-4cb5-b35f-127a02768e7f
    Copy to Clipboard Toggle word wrap

When using two-way replication you can only mirror between two storage clusters.

Examples in this section will distinguish between two storage clusters by referring to the primary storage cluster with the primary images as site-a, and the secondary storage cluster you are replicating the images to, as site-b. The pool name used in these examples is called data.

Prerequisites

  • A minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to a Ceph client node for each storage cluster.
  • A CephX user with administrator-level capabilities.

Procedure

  1. Log into the cephadm shell on both the sites:

    Example

    [root@site-a ~]# cephadm shell
[root@site-b ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. On the site-a primary cluster, run the following command:

    Example

    [ceph: root@site-a /]# ceph orch apply rbd-mirror --placement=host01
    Copy to Clipboard Toggle word wrap

    Note

    The nodename is the host where you want to configure mirroring.

  3. On site-b, schedule the deployment of mirror daemon on the secondary cluster:

    Syntax

    ceph orch apply rbd-mirror --placement=NODENAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@site-b /]# ceph orch apply rbd-mirror --placement=host04
    Copy to Clipboard Toggle word wrap

    Note

    The nodename is the host where you want to configure mirroring in the secondary cluster.

  4. Enable journaling features on an image on site-a.

    1. For new images, use the --image-feature option:

      Syntax

      rbd create IMAGE_NAME --size MEGABYTES --pool POOL_NAME --image-feature FEATURE FEATURE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd create image1 --size 1024 --pool data --image-feature exclusive-lock,journaling
      Copy to Clipboard Toggle word wrap

      Note

      If exclusive-lock is already enabled, use journaling as the only argument, else it returns the following error: 

      one or more requested features are already enabled
(22) Invalid argument
      Copy to Clipboard Toggle word wrap

    2. For existing images, use the rbd feature enable command:

      Syntax

      rbd feature enable POOL_NAME/IMAGE_NAME FEATURE, FEATURE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd feature enable data/image1 exclusive-lock, journaling
      Copy to Clipboard Toggle word wrap

    3. To enable journaling on all new images by default, set the configuration parameter using ceph config set command:

      Example

      [ceph: root@site-a /]# ceph config set global rbd_default_features 125
[ceph: root@site-a /]# ceph config show mon.host01 rbd_default_features
      Copy to Clipboard Toggle word wrap

  5. Choose the mirroring mode, either pool or image mode, on both the storage clusters.

    1. Enabling pool mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool enable data pool
[ceph: root@site-b /]# rbd mirror pool enable data pool
      Copy to Clipboard Toggle word wrap

      This example enables mirroring of the whole pool named data.

    2. Enabling image mode:

      Syntax

      rbd mirror pool enable POOL_NAME MODE
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool enable data image
[ceph: root@site-b /]# rbd mirror pool enable data image
      Copy to Clipboard Toggle word wrap

      This example enables image mode mirroring on the pool named data.

      Note

      To enable mirroring on specific images in a pool, see the Enabling image mirroring section in the Red Hat Ceph Storage Block Device Guide for more details.

    3. Verify that mirroring has been successfully enabled at both the sites:

      Syntax

      rbd mirror pool info POOL_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@site-a /]# rbd mirror pool info data
Mode: pool
Site Name: c13d8065-b33d-4cb5-b35f-127a02768e7f

Peer Sites: none

[ceph: root@site-b /]# rbd mirror pool info data
Mode: pool
Site Name: a4c667e2-b635-47ad-b462-6faeeee78df7

Peer Sites: none
      Copy to Clipboard Toggle word wrap

  6. On a Ceph client node, bootstrap the storage cluster peers.

    1. Create Ceph user accounts, and register the storage cluster peer to the pool:

      Syntax

      rbd mirror pool peer bootstrap create --site-name PRIMARY_LOCAL_SITE_NAME POOL_NAME > PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-a /]# rbd mirror pool peer bootstrap create --site-name site-a data > /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      This example bootstrap command creates the client.rbd-mirror.site-a and the client.rbd-mirror-peer Ceph users.

    2. Copy the bootstrap token file to the site-b storage cluster.

    3. Import the bootstrap token on the site-b storage cluster:

      Syntax

      rbd mirror pool peer bootstrap import --site-name SECONDARY_LOCAL_SITE_NAME --direction rx-tx POOL_NAME PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-b /]# rbd mirror pool peer bootstrap import --site-name site-b --direction rx-tx data /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      The --direction argument is optional, as two-way mirroring is the default when bootstrapping peers.

Verification steps

  1. To verify the mirroring status, run the following command from a Ceph Monitor node on the primary and secondary sites:

    Syntax

    rbd mirror image status POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@mon-site-a /]# rbd mirror image status data/image1
image1:
  global_id:   a4c667e2-b635-47ad-b462-6faeeee78df7
  state:       up+stopped
  description: local image is primary
  service:     host03.glsdbv on host03.ceph.redhat.com
  last_update: 2021-09-16 10:55:58
  peer_sites:
    name: a
    state: up+stopped
    description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
    last_update: 2021-09-16 10:55:50
    Copy to Clipboard Toggle word wrap

    Here, up means the rbd-mirror daemon is running, and stopped means this image is not the target for replication from another storage cluster. This is because the image is primary on this storage cluster.

    Example

    [ceph: root@mon-site-b /]# rbd mirror image status data/image1
image1:
  global_id:   a4c667e2-b635-47ad-b462-6faeeee78df7
  state:       up+replaying
  description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}
  service:     host05.dtisty on host05
  last_update: 2021-09-16 10:57:20
  peer_sites:
    name: b
    state: up+stopped
    description: local image is primary
    last_update: 2021-09-16 10:57:28
    Copy to Clipboard Toggle word wrap

    If images are in the state up+replaying, then mirroring is functioning properly. Here, up means the rbd-mirror daemon is running, and replaying means this image is the target for replication from another storage cluster.

    Note

    Depending on the connection between the sites, mirroring can take a long time to sync the images.

6.3. Administering RBD mirroring
Copy link

As a storage administrator, you can perform a range of tasks to manage and maintain the Ceph Block Device (RBD) mirroring environment.

These tasks include:

  • Viewing information about storage cluster peers.
  • Add or remove a storage cluster peer.
  • Getting mirroring status for a pool or image.
  • Enabling mirroring on a pool or image.
  • Disabling mirroring on a pool or image.
  • Delaying block device replication.
  • Promoting and demoting an image.

These tasks help ensure smooth replication operations and support failover and recovery scenarios in multi-site deployments.

6.3.1. Viewing peer information
Copy link

View information about storage cluster peers.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. To view information about the peers:

    Syntax

    rbd mirror pool info POOL_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror pool info data
Mode: pool
Site Name: a

Peer Sites:

UUID: 950ddadf-f995-47b7-9416-b9bb233f66e3
Name: b
Mirror UUID: 4696cd9d-1466-4f98-a97a-3748b6b722b3
Direction: rx-tx
Client: client.rbd-mirror-peer
    Copy to Clipboard Toggle word wrap

6.3.2. Managing mirroring on a pool
Copy link

You can enable, disable and get the status of mirroring on a Ceph Block Device pool by using the command-line interface.

Prerequisites

  • Before you begin, make sure that you have root-level access to the node.

6.3.2.1. Enabling mirroring on a pool
Copy link

Run the following commands on both peer clusters to enable mirroring on a pool.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. To enable mirroring on a pool:

    Syntax

    rbd mirror pool enable POOL_NAME MODE
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror pool enable data pool
    Copy to Clipboard Toggle word wrap

    This example enables mirroring of the whole pool named data.

    Example

    [root@rbd-client ~]# rbd mirror pool enable data image
    Copy to Clipboard Toggle word wrap

    This example enables image mode mirroring on the pool named data.

Additional Resources

6.3.2.2. Disabling mirroring on a pool
Copy link

Before disabling mirroring, remove the peer clusters.

Note

When you disable mirroring on a pool, you also disable it on any images within the pool for which mirroring was enabled separately in image mode.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. To disable mirroring on a pool:

    Syntax

    rbd mirror pool disable POOL_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror pool disable data
    Copy to Clipboard Toggle word wrap

    This example disables mirroring of a pool named data.

6.3.2.3. Getting mirroring status for a pool
Copy link

You can get the mirror status for a pool on the storage clusters.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To get the mirroring pool summary:

    Syntax

    rbd mirror pool status POOL_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@site-a ~]# rbd mirror pool status data
health: OK
daemon health: OK
image health: OK
images: 1 total
    1 replaying
    Copy to Clipboard Toggle word wrap

    Tip

    To output status details for every mirroring image in a pool, use the --verbose option.

6.3.3. Managing mirroring on images
Copy link

You can enable and disable mirroring on Ceph Block Device images by using the command-line interface. You can also get the mirroring status for an image. Also, learn to promote, demote, and resynchronize an image.

6.3.3.1. Enabling image mirroring
Copy link

Enable mirroring on the whole pool in image mode on both peer storage clusters.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Enable mirroring for a specific image within the pool:

    Syntax

    rbd mirror image enable POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image enable data/image2
    Copy to Clipboard Toggle word wrap

    This example enables mirroring for the image2 image in the data pool.

Additional Resources

6.3.3.2. Disabling image mirroring
Copy link

You can disable Ceph Block Device mirroring on images.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To disable mirroring for a specific image:

    Syntax

    rbd mirror image disable POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image disable data/image2
    Copy to Clipboard Toggle word wrap

    This example disables mirroring of the image2 image in the data pool.

You can get the mirror status for an image by running the mirror image status command.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To get the status of a mirrored image:

    Syntax

    rbd mirror image status POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@site-a ~]# rbd mirror image status data/image2
image2:
  global_id:   1e3422a2-433e-4316-9e43-1827f8dbe0ef
  state:       up+unknown
  description: remote image is non-primary
  service:     pluto008.yuoosv on pluto008
  last_update: 2021-10-06 09:37:58
    Copy to Clipboard Toggle word wrap

    This example gets the status of the image2 image in the data pool.

6.3.3.4. Promoting an image
Copy link

You can promote or demote an image in a pool.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To promote an image to primary:

    Syntax

    rbd mirror image promote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image promote data/image2
    Copy to Clipboard Toggle word wrap

    This example promotes image2 in the data pool.

    Depending on which type of mirroring you are using, see either Recover from a disaster with one-way mirroring or Recover from a disaster with two-way mirroring for details.

    Syntax

    rbd mirror image promote --force POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image promote --force data/image2
    Copy to Clipboard Toggle word wrap

    Use forced promotion when the demotion cannot be propagated to the peer Ceph storage cluster. For example, because of cluster failure or communication outage.

6.3.3.5. Demoting an image
Copy link

You can demote an image in a pool.

Warning

Do not force promote non-primary images that are still syncing, because the images will not be valid after the promotion.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To demote an image to non-primary:

    Syntax

    rbd mirror image demote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image demote data/image2
    Copy to Clipboard Toggle word wrap

    This example demotes the image2 image in the data pool.

6.3.3.6. Resynchronizing images
Copy link

You can re-synchronize an image if an inconsistency occurs between the two peer clusters. When an image enters an inconsistent state, the rbd-mirror daemon skips mirroring for that image until the issue is resolved.

Prerequisites

  • A running Red Hat Ceph Storage cluster with snapshot-based mirroring configured.
  • Root-level access to the node.

Procedure

  1. To request a re-synchronization to the primary image:

    Syntax

    rbd mirror image resync POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image resync data/image2
    Copy to Clipboard Toggle word wrap

    This example requests resynchronization of image2 in the data pool.

6.3.4. Managing mirroring on a consistency group
Copy link

Learn to enable and disable mirroring on Ceph Block Device consistency group by using the command-line interface. You can also get the mirroring status for the group. Also, learn to promote, demote, and resynchronize the group. The commands in the following sections support the optional namespace parameter and you can use it in the command as POOL_NAME/NAMESPACE/GROUP_NAME.

Note

You can mirror a maximum of 100 images in total per cluster and a maximum of 50 images per group.

6.3.4.1. Enabling mirroring on a group
Copy link

Enable mirroring for a group within the pool, by using the mirror group enable command.

Note
  • Group mirroring works with image mode only.
  • Consistency group mirroring only supports snapshot mirroring mode and journal mode is not supported.

Prerequisistes

Procedure

  • Run the following command to enable mirroring on a group.

    Syntax

    rbd mirror group enable _POOL_NAME_/_GROUP_NAME_
    Copy to Clipboard Toggle word wrap

    Example

    The following example enables mirroring for the test_group group in the test_pool pool. 

    [root@rbd-client ~]# rbd mirror group enable test_pool/test_group
    Copy to Clipboard Toggle word wrap
Note
  • After you enable mirroring for a group, you cannot add new images to that group. To add images, you must disable group mirroring and add the wanted images. You can then re-enable mirroring.
  • Images in the group cannot be enabled for mirroring.

6.3.4.2. Disabling mirroring on a group
Copy link

Disable mirroring for a group within the pool, by using the mirror group disable command.

Prerequisistes

  • Make sure that you have root-level access to the node.

Procedure

  • Run the following command to disable mirroring on a group.

    Syntax

    rbd mirror group disable _POOL_NAME_/_GROUP_NAME_
    Copy to Clipboard Toggle word wrap

    Example

    The following example disables mirroring for the test_group group in the test_pool pool. 

    [root@rbd-client ~]# rbd mirror group disable test_pool/test_group
    Copy to Clipboard Toggle word wrap

6.3.4.3. Getting mirroring status for a group
Copy link

You can get the mirror status for the group, by running the mirror group status command.

Prerequisite

  • Make sure that you have root-level access to the node.

Procedure

  • Run the following command to get the group mirroring status.

    Syntax

    rbd mirror group status _POOL_NAME_/_GROUP_NAME_
    Copy to Clipboard Toggle word wrap

    Example

    The following example gets the status of the test_group group in the test_pool pool. 

    [root@site-a ~]# rbd mirror group status test_pool/test_group
test_group:
  global_id:   8206677f-00e5-4b28-a6cc-121be028cb84
  state:       up+stopped
  description: local group is primary
  service:     admin on dhcp53-181.lab.eng.abc.pqr.com
  last_update: 2025-06-13 17:59:16
  images:
    image:       3/9c6c0e9b-abbb-4219-891a-b451396487a4
    state:       up+stopped
    description: local image is primary
  peer_sites:
    name: site-b
    state: up+replaying
    description: replaying, {"last_snapshot_bytes":0,"last_snapshot_complete_seconds":10,"local_snapshot_timestamp":1749817730,"remote_snapshot_timestamp":1749817730}
    last_update: 2025-06-13 17:59:16
    images:
      image:       3/9c6c0e9b-abbb-4219-891a-b451396487a4
      state:       up+replaying
      description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1749817730,"remote_snapshot_timestamp":1749817730,"replay_state":"idle"}
  snapshots:
    .mirror.primary.8206677f-00e5-4b28-a6cc-121be028cb84.104dda662be0
    Copy to Clipboard Toggle word wrap

6.3.4.4. Promoting a group
Copy link

You can promote a group by using the command in the procedure.

Prerequisite

  • A running Red Hat Ceph storage cluster with snapshot-based group mirroring is configured.
  • Root-level access to the node.

Procedure

  • Promote a group to primary, by using the mirror group promote command.

Syntax

rbd mirror group promote _POOL_NAME_/_GROUP_NAME_
Copy to Clipboard Toggle word wrap

Example

The following example promotes the group test_group in the test_pool pool to primary. 

[root@rbd-client ~]# rbd mirror group promote test_pool/test_group
Copy to Clipboard Toggle word wrap

Use forced promotion when the demotion cannot be propagated to the peer Ceph storage cluster because of cluster failure or communication outage.

Important

Before executing the mirror group promote command with --force flag, ensure that the secondary mirror daemon is stopped. This step is a temporary workaround for known issues and may not be necessary once those issues are resolved.

  1. Run the command to stop and kill the secondary mirror daemon. 

    ceph orch stop rbd-mirror

kill -SIGKILL PID_OF_SECONDARY_RBD_MIRROR
    Copy to Clipboard Toggle word wrap

  2. Execute the mirror group promote command with the --force flag and restart the mirror daemon after the mirror group promote command completes execution. 

    rbd mirror group promote --force POOL_NAME/GROUP_NAME

ceph orch start rbd-mirror
    Copy to Clipboard Toggle word wrap

Example

[root@rbd-client ~]# rbd mirror group promote --force test_pool/test_group
Copy to Clipboard Toggle word wrap

6.3.4.5. Demoting a group
Copy link

Demote a group by using the command in the procedure.

Prerequisite

  • A running Red Hat Ceph storage cluster with snapshot-based group mirroring is configured.
  • Root-level access to the node.

Procedure

  • Demote a group to non-primary, by using the mirror group demote command.

Syntax

rbd mirror group demote POOL_NAME/GROUP_NAME
Copy to Clipboard Toggle word wrap

Example

The following example demotes the group test_group image in the test_pool pool to non-primary. 

[root@rbd-client ~]# rbd mirror group demote test_pool/test_group
Copy to Clipboard Toggle word wrap

6.3.4.6. Resynchronizing a group
Copy link

You can resynchronize a group if an inconsistency occurs between the two peer clusters. When a group enters an inconsistent state, the rbd-mirror daemon skips mirroring for that group until the issue is resolved.

Resynchronize a group by using the command in the procedure.

Prerequisite

  • Root-level access to the node.

Procedure

  • Resynchronize to the primary group, by using the mirror group resync command.

Syntax

rbd mirror group resync _POOL_NAME_/_GROUP_NAME_
Copy to Clipboard Toggle word wrap

Example

The following example requests resynchronzation of test_group in the test_pool pool. 

[root@rbd-client ~]# rbd mirror group resync test_pool/test_group
Copy to Clipboard Toggle word wrap

6.3.5. Managing mirroring on a namespace
Copy link

You can enable and disable Ceph Block Device mirroring on namespaces by using the command-line interface.

Before configuring a namespace to mirror with a different remote namespace, mirroring must be disabled for both the local and remote namespaces on both the clusters.

6.3.5.1. Enabling namespace mirroring
Copy link

You can configure mirroring on a namespace in a pool. The pool must already be enabled for mirroring. Mirror the namespace to a namespace with the same or a different name in the remote pool from a remote cluster (secondary cluster).

You can mirror a namespace to a namespace with a different name in the remote pool using the --remote-namespace option. The default behaviour of the namespace is to mirror to a namespace with the same name on the remote pool.

Prerequisites

  • Root-level access to the node.
  • 2 running Red Hat Ceph Storage cluster.
  • rbd-mirror daemon service is enabled on both clusters.
  • Enable mirroring for the pool in which the namespace is added.

Procedure

  • Run the following command on both clusters where you want to enable mirroring on a namespace.

    Syntax

    rbd mirror pool enable POOL_NAME/LOCAL_NAMESPACE_NAME _MODE --remote-namespace REMOTE_NAMESPACE_NAME
    Copy to Clipboard Toggle word wrap

    Note

    The --remote-namespace parameter is optional.

    The mirroring mode can either be image or pool:

  • Image mode: When configured in image mode, mirroring must be explicitly enabled on each image.

  • Pool mode (default): When configured in pool mode, all images in the namespace with the journaling feature enabled are mirrored.

    Example

    [root@rbd-client ~]# rbd mirror pool enable image-pool/namespace-a image --remote-namespace namespace-b

Remote cluster:
[root@rbd-client ~]#  rbd mirror pool enable image-pool/namespace-b image --remote-namespace namespace-a
    Copy to Clipboard Toggle word wrap

    This example enables image mode mirroring between image-pool/namespace-a on the first cluster and image-pool/namespace-b on the second cluster.

The namespace and remote namespace on the first cluster must match the remote namespace and namespace respectively on the remote cluster.

Note

If the --remote-namespace option is not provided, the namespace is mirrored to a namespace with the same name in the remote pool.

6.3.5.2. Disabling namespace mirroring
Copy link

You can disable Ceph Block Device mirroring on namespaces.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.
  • Any mirror enabled images in the namespace must be explicitly disabled before disabling mirroring on the namespace if configured in image mode.

Procedure

  • Run the following command on both clusters where you want to disable mirroring on a namespace.

    Syntax

    rbd mirror pool disable POOL_NAME/NAMESPACE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror pool disable image-pool/namespace-a
[root@rbd-client ~]# rbd mirror pool disable image-pool/namespace-b
    Copy to Clipboard Toggle word wrap

To enable a namespace with a different remote namespace, the namespace and the corresponding remote namespace on both clusters must be disabled for mirroring before they can be re-enabled.

6.3.6. Adding a storage cluster peer
Copy link

Add a storage cluster peer to enable the rbd-mirror daemon to discover the remote cluster.

For example, to add the site-a storage cluster as a peer to site-b, perform the following steps from a client node in the site-b cluster.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Register the peer to the pool:

    Syntax

    rbd --cluster CLUSTER_NAME mirror pool peer add POOL_NAME PEER_CLIENT_NAME@PEER_CLUSTER_NAME -n CLIENT_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd --cluster site-b mirror pool peer add data client.site-a@site-a -n client.site-b
    Copy to Clipboard Toggle word wrap

6.3.7. Removing a storage cluster peer
Copy link

Specify the peer UUID to remove a storage cluster peer.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Specify the pool name and the peer Universally Unique Identifier (UUID).

    Syntax

    rbd mirror pool peer remove POOL_NAME PEER_UUID
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror pool peer remove data 7e90b4ce-e36d-4f07-8cbc-42050896825d
    Copy to Clipboard Toggle word wrap

    Tip

    To view the peer UUID, use the rbd mirror pool info command.

6.3.8. Delaying block device replication
Copy link

Whether using one-way or two-way replication, you can configure delayed replication for RADOS Block Device (RBD) mirrored images. Delayed replication provides a buffer period, allowing you to revert unintended changes on the primary image before they are propagated to the secondary image.

Note

Delaying block device replication is only applicable with journal-based mirroring.

To implement delayed replication, the rbd-mirror daemon within the destination storage cluster should set the rbd_mirroring_replay_delay = MINIMUM_DELAY_IN_SECONDS configuration option. This setting can either be applied globally within the ceph.conf file utilized by the rbd-mirror daemons, or on an individual image basis.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. To utilize delayed replication for a specific image, on the primary image, run the following rbd CLI command:

    Syntax

    rbd image-meta set POOL_NAME/IMAGE_NAME conf_rbd_mirroring_replay_delay MINIMUM_DELAY_IN_SECONDS
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd image-meta set vms/vm-1 conf_rbd_mirroring_replay_delay 600
    Copy to Clipboard Toggle word wrap

    This example sets a 10 minute minimum replication delay on image vm-1 in the vms pool.

You can convert journal-based mirroring to snapshot-based mirroring by disabling mirroring and enabling snapshot.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@rbd-client ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. Disable mirroring for a specific image within the pool:

    Syntax

    rbd mirror image disable POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@rbd-client /]# rbd mirror image disable mirror_pool/mirror_image
Mirroring disabled
    Copy to Clipboard Toggle word wrap

  3. Enable snapshot-based mirroring for the image:

    Syntax

    rbd mirror image enable POOL_NAME/IMAGE_NAME snapshot
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@rbd-client /]# rbd mirror image enable mirror_pool/mirror_image snapshot
Mirroring enabled
    Copy to Clipboard Toggle word wrap

    This example enables snapshot-based mirroring for the mirror_image image in the mirror_pool pool.

6.3.10. Creating an image mirror-snapshot
Copy link

Create a mirror snapshot to replicate changes in a Ceph Block Device image when using snapshot-based mirroring.

Prerequisites

  • A minimum of two healthy running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the Red Hat Ceph Storage clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where a snapshot mirror will be created.
Important

By default, a maximum of 5 image mirror-snapshots are retained. The most recent image mirror-snapshot is automatically removed if the limit is reached. If required, the limit can be overridden through the rbd_mirroring_max_mirroring_snapshots configuration. Image mirror-snapshots are automatically deleted when the image is removed or when mirroring is disabled.

Procedure

  • To create an image-mirror snapshot:

    Syntax

    rbd --cluster CLUSTER_NAME mirror image snapshot POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@site-a ~]# rbd mirror image snapshot data/image1
    Copy to Clipboard Toggle word wrap

Additional Resources

6.3.11. Scheduling mirror-snapshots
Copy link

Mirror-snapshots can be automatically created when mirror-snapshot schedules are defined. The mirror-snapshot can be scheduled globally, per-pool or per-image levels. Multiple mirror-snapshot schedules can be defined at any level but only the most specific snapshot schedules that match an individual mirrored image will run.

6.3.11.1. Creating a mirror-snapshot schedule
Copy link

You can create a mirror-snapshot schedule using the snapshot schedule command.

Prerequisites

  • A minimum of two healthy running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the Red Hat Ceph Storage clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where the mirror image needs to be scheduled.

Procedure

  1. To create a mirror-snapshot schedule:

    Syntax

    rbd --cluster CLUSTER_NAME mirror snapshot schedule add --pool POOL_NAME --image IMAGE_NAME INTERVAL [START_TIME]
    Copy to Clipboard Toggle word wrap

    The CLUSTER_NAME should be used only when the cluster name is different from the default name ceph. The interval can be specified in days, hours, or minutes using d, h, or m suffix respectively. The optional START_TIME can be specified using the ISO 8601 time format.

    Example

    [root@site-a ~]# rbd mirror snapshot schedule add --pool data --image image1 6h
    Copy to Clipboard Toggle word wrap

    Example

    [root@site-a ~]# rbd mirror snapshot schedule add --pool data --image image1 24h 14:00:00-05:00
    Copy to Clipboard Toggle word wrap

6.3.11.2. Listing snapshot schedules by level
Copy link

You can list all snapshot schedules at a specific level.

Prerequisites

  • A minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the Red Hat Ceph Storage clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where the mirror image needs to be scheduled.

Procedure

  1. To list all snapshot schedules for a specific global, pool or image level, with an optional pool or image name:

    Syntax

    rbd --cluster site-a mirror snapshot schedule ls --pool POOL_NAME --recursive
    Copy to Clipboard Toggle word wrap

    Additionally, the --recursive option can be specified to list all schedules at the specified level as shown below:

    Example

    [root@rbd-client ~]# rbd mirror snapshot schedule ls --pool data --recursive
POOL        NAMESPACE IMAGE  SCHEDULE
data         -         -       every 1d starting at 14:00:00-05:00
data         -        image1   every 6h
    Copy to Clipboard Toggle word wrap

6.3.11.3. Removing a mirror-snapshot schedule
Copy link

You can remove a mirror-snapshot schedule using the snapshot schedule remove command.

Prerequisites

  • A minimum of two healthy running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the Red Hat Ceph Storage clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where the mirror image needs to be scheduled.

Procedure

  1. To remove a mirror-snapshot schedule:

    Syntax

    rbd --cluster CLUSTER_NAME mirror snapshot schedule remove --pool POOL_NAME --image IMAGE_NAME INTERVAL START_TIME
    Copy to Clipboard Toggle word wrap

    The interval can be specified in days, hours, or minutes using d, h, m suffix respectively. The optional START_TIME can be specified using the ISO 8601 time format.

    Example

    [root@site-a ~]# rbd mirror snapshot schedule remove --pool data --image image1 6h
    Copy to Clipboard Toggle word wrap

    Example

    [root@site-a ~]# rbd mirror snapshot schedule remove --pool data --image image1 24h 14:00:00-05:00
    Copy to Clipboard Toggle word wrap

6.3.11.4. Viewing upcoming snapshot schedule
Copy link

You can view the status for the next snapshots to be created for snapshot-based mirroring RBD images.

Prerequisites

  • A minimum of two healthy running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the Red Hat Ceph Storage clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where the mirror image needs to be scheduled.

Procedure

  1. To view the status for the next snapshots to be created:

    Syntax

    rbd --cluster site-a mirror snapshot schedule status [--pool POOL_NAME] [--image IMAGE_NAME]
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror snapshot schedule status
SCHEDULE    TIME       IMAGE
2021-09-21 18:00:00 data/image1
    Copy to Clipboard Toggle word wrap

Additional Resources

You can create a mirror group snapshot to replicate changes in a Ceph Block Device when using snapshot-based mirroring.

Prerequisites

  • A minimum of two running IBM Storage Ceph clusters.
  • Root-level access to the Ceph client nodes for the IBM Storage Ceph clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the IBM Storage Ceph cluster where a snapshot mirror will be created.

Procedure

  • Create a mirror group mirroring snapshot.

Syntax

rbd --cluster _CLUSTER_NAME_ mirror group snapshot _POOL_NAME_/_GROUP_NAME_
Copy to Clipboard Toggle word wrap

Example

[root@site-a ~]# rbd --cluster site-a mirror group snapshot test_pool/test_group
Copy to Clipboard Toggle word wrap

You can create, remove, list, and check the status of group mirror snapshot schedule by using the command-line interface.

Prerequisites

  • A minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to the Ceph client nodes for the IBM Storage Ceph clusters.
  • A CephX user with administrator-level capabilities.
  • Access to the Red Hat Ceph Storage cluster where creation of group mirror-snapshots need to be scheduled.

6.3.13.1. Creating a schedule
Copy link

You can create a group snapshot schedule, by using the mirror group snapshot schedule add command.

Procedure

  • Run the following command to create mirror snapshot schedule for group.

Syntax

rbd --cluster _CLUSTER_NAME_ mirror group snapshot schedule add --pool _POOL_NAME_ --group _GROUP_NAME_ _INTERVAL_ [START_TIME]
Copy to Clipboard Toggle word wrap

Only use --cluster when the cluster name is different from the default name, ceph.

Specify the interval in days, hours, or minutes using d, h, or m suffix, respectively. The optional START_TIME must be specified using the ISO 8601 (hh:mm:ss+|–hh:mm) time format.

Example

The following example creates a mirror group snapshot schedule with the default ceph cluster, every 6 hours, and does not specify a start time. 

[root@site-a ~]# rbd mirror group snapshot schedule add --pool test_pool --group test_group 6h
Copy to Clipboard Toggle word wrap

The following example creates a mirror group snapshot schedule with the default ceph cluster, every 24 hours, with a start time of 2:00 PM at -5 GMT. 

[root@site-a ~]# rbd mirror group snapshot schedule add --pool test_pool --group test_group 24h 14:00:00-05:00
Copy to Clipboard Toggle word wrap

6.3.13.2. Listing snapshot schedules by level
Copy link

You can list all snapshot schedules at the global, pool, namespace or group level by using the by using the mirror group snapshot schedule ls command.

Procedure

  • List all snapshot schedules at the global, pool, namespace or group level by running the following command.

Syntax

rbd --cluster _CLUSTER_NAME_ mirror group snapshot schedule ls --pool _POOL_NAME_ --namespace _NAMESPACE_ --group _GROUP_NAME _--recursive
Copy to Clipboard Toggle word wrap

Use the --recursive option to list all schedules at the specified level.

Example

[root@rbd-client ~]# # rbd --cluster site-a mirror group snapshot schedule ls --pool test_pool --recursive

POOL NAMESPACE GROUP SCHEDULE
test_pool test_group every 30m
Copy to Clipboard Toggle word wrap

  • To list the mirror group snapshot schedule, run the following command.

Syntax

rbd --cluster _CLUSTER_NAME_ mirror group snapshot schedule ls --pool _POOL_NAME_ --group _GROUP_NAME_ --recursive
Copy to Clipboard Toggle word wrap

Example

[root@rbd-client ~]# rbd --cluster site-a mirror group snapshot schedule ls --pool test_pool --group test_group --recursive

POOL NAMESPACE GROUP SCHEDULE
test_pool test_group every 30m
Copy to Clipboard Toggle word wrap

6.3.13.3. Removing a schedule
Copy link

You can remove a group snapshot schedule, by using the mirror group snapshot schedule remove command.

Procedure

  • Run the following command to remove the snapshot schedule for a group.

Syntax

rbd --cluster _CLUSTER_NAME_ mirror group snapshot schedule remove --pool _POOL_NAME_ --group _GROUP_NAME_ _INTERVAL_ _START_TIME_
Copy to Clipboard Toggle word wrap

Only use --cluster when the cluster name is different from the default name, ceph.

Specify the interval in days, hours, or minutes using d, h, or m suffix, respectively. The optional START_TIME must be specified using the ISO 8601 (hh:mm:ss+|–hh:mm) time format.

Example

The following example removes a snapshot schedule with the default ceph cluster, every 6 hours, and does not specify a start time. 

[root@site-a ~]# rbd mirror group snapshot schedule remove --pool test_pool --group test_group 6h
Copy to Clipboard Toggle word wrap

The following example removes a snapshot schedule with the default ceph cluster, every 24 hours, with a start time of 2:00 PM at -5 GMT. 

[root@site-a ~]# rbd mirror group snapshot schedule remove --pool test_pool --group test_group 24h 14:00:00-05:00
Copy to Clipboard Toggle word wrap

6.3.13.4. Viewing upcoming snapshot schedule
Copy link

You can view the status for the next snapshots to be created for snapshot-based mirroring Ceph Block Device consistency group.

Procedure

  • View the status for the next snapshots to be created.

Syntax

rbd --cluster CLUSTER_NAME mirror group snapshot schedule status --pool _POOL_NAME_ --namespace _NAMESPACE_NAME_ --group _GROUP_NAME_
Copy to Clipboard Toggle word wrap

Example

[root@rbd-client ~]# rbd mirror --cluster site-a mirror group snapshot schedule status --pool test_pool --group test_group
SCHEDULE    TIME       GROUP
2025-05-01 18:00:00 test_pool/test_group
Copy to Clipboard Toggle word wrap

6.4. Recover from a disaster
Copy link

As a storage administrator, you can be prepared for eventual hardware failure by knowing how to recover the data from another storage cluster where mirroring was configured.

In the examples, the primary storage cluster is known as the site-a, and the secondary storage cluster is known as the site-b. Additionally, the storage clusters both have a data pool with two images, image1 and image2.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • One-way or two-way mirroring was configured.

6.4.1. Disaster recovery
Copy link

Asynchronous replication of block data between two or more Red Hat Ceph Storage clusters reduces downtime and prevents data loss in the event of a significant data center failure. These failures have a widespread impact, also referred as a large blast radius, and can be caused by impacts to the power grid and natural disasters.

Customer data needs to be protected during these scenarios. Volumes must be replicated with consistency and efficiency and also within Recovery Point Objective (RPO) and Recovery Time Objective (RTO) targets. This solution is called a Wide Area Network- Disaster Recovery (WAN-DR).

In such scenarios it is hard to restore the primary system and the data center. The quickest way to recover is to failover the applications to an alternate Red Hat Ceph Storage cluster (disaster recovery site) and make the cluster operational with the latest copy of the data available. The solutions that are used to recover from these failure scenarios are guided by the application:

  • Recovery Point Objective (RPO): The amount of data loss, an application tolerate in the worst case.
  • Recovery Time Objective (RTO): The time taken to get the application back on line with the latest copy of the data available.

Additional Resources

  • See the Mirroring Ceph block devices Chapter in the Red Hat Ceph Storage Block Device Guide for details.
  • See the Encryption in transit section in the Red Hat Ceph Storage Data Security and Hardening Guide to know more about data transmission over the wire in an encrypted state.

To recover from a disaster when using one-way mirroring use the following procedures. They show how to fail over to the secondary cluster after the primary cluster terminates, and how to fail back. The shutdown can be orderly or non-orderly.

Important

One-way mirroring supports multiple secondary sites. If you are using additional secondary clusters, choose one of the secondary clusters to fail over to. Synchronize from the same cluster during fail back.

To recover from a disaster when using two-way mirroring use the following procedures. They show how to fail over to the mirrored data on the secondary cluster after the primary cluster terminates, and how to failback. The shutdown can be orderly or non-orderly.

6.4.4. Failover after an orderly shutdown
Copy link

Failover to the secondary storage cluster after an orderly shutdown.

Prerequisites

  • Minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to the node.
  • Pool mirroring or image mirroring configured with one-way mirroring.

Procedure

  1. Stop all clients that use the primary image. This step depends on which clients use the image. For example, detach volumes from any OpenStack instances that use the image.

  2. Demote the primary images located on the site-a cluster by running the following commands on a monitor node in the site-a cluster:

    Syntax

    rbd mirror image demote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image demote data/image1
[root@rbd-client ~]# rbd mirror image demote data/image2
    Copy to Clipboard Toggle word wrap

  3. Promote the non-primary images located on the site-b cluster by running the following commands on a monitor node in the site-b cluster:

    Syntax

    rbd mirror image promote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image promote data/image1
[root@rbd-client ~]# rbd mirror image promote data/image2
    Copy to Clipboard Toggle word wrap

  4. After some time, check the status of the images from a monitor node in the site-b cluster. They should show a state of up+stopped and be listed as primary: 

    [root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-17 16:04:37
[root@rbd-client ~]# rbd mirror image status data/image2
image2:
  global_id:   596f41bc-874b-4cd4-aefe-4929578cc834
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-17 16:04:37
    Copy to Clipboard Toggle word wrap
  5. Resume the access to the images. This step depends on which clients use the image.

Additional Resources

6.4.5. Failover after a non-orderly shutdown
Copy link

Failover to secondary storage cluster after a non-orderly shutdown.

Prerequisites

  • Minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to the node.
  • Pool mirroring or image mirroring configured with one-way mirroring.

Procedure

  1. Verify that the primary storage cluster is down.
  2. Stop all clients that use the primary image. This step depends on which clients use the image. For example, detach volumes from any OpenStack instances that use the image.

  3. Promote the non-primary images from a Ceph Monitor node in the site-b storage cluster. Use the --force option, because the demotion cannot be propagated to the site-a storage cluster:

    Syntax

    rbd mirror image promote --force POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image promote --force data/image1
[root@rbd-client ~]# rbd mirror image promote --force data/image2
    Copy to Clipboard Toggle word wrap

  4. Check the status of the images from a Ceph Monitor node in the site-b storage cluster. They should show a state of up+stopping_replay. The description should say force promoted, meaning it is in the intermittent state. Wait until the state comes to up+stopped to validate the site is successfully promoted.

    Example

    [root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopping_replay
  description: force promoted
  last_update: 2023-04-17 13:25:06

[root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: force promoted
  last_update: 2023-04-17 13:25:06
    Copy to Clipboard Toggle word wrap

Additional Resources

6.4.6. Prepare for fail back
Copy link

If two storage clusters were originally configured only for one-way mirroring, in order to fail back, configure the primary storage cluster for mirroring in order to replicate the images in the opposite direction.

During failback scenario, the existing peer that is inaccessible must be removed before adding a new peer to an existing cluster.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the client node.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@rbd-client ~]# cephadm shell
    Copy to Clipboard Toggle word wrap

  2. On the site-a storage cluster , run the following command:

    Example

    [ceph: root@rbd-client /]# ceph orch apply rbd-mirror --placement=host01
    Copy to Clipboard Toggle word wrap

  3. Remove any inaccessible peers.

    Important

    This step must be run on the peer site which is up and running.

    Note

    Multiple peers are supported only for one way mirroring.

    1. Get the pool UUID:

      Syntax

      rbd mirror pool info POOL_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# rbd mirror pool info pool_failback
      Copy to Clipboard Toggle word wrap

    2. Remove the inaccessible peer:

      Syntax

      rbd mirror pool peer remove POOL_NAME PEER_UUID
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# rbd mirror pool peer remove pool_failback f055bb88-6253-4041-923d-08c4ecbe799a
      Copy to Clipboard Toggle word wrap

  4. Create a block device pool with a name same as its peer mirror pool.

    1. To create an rbd pool, execute the following:

      Syntax

      ceph osd pool create POOL_NAME PG_NUM
ceph osd pool application enable POOL_NAME rbd
rbd pool init -p POOL_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [root@rbd-client ~]# ceph osd pool create pool1
[root@rbd-client ~]# ceph osd pool application enable pool1 rbd
[root@rbd-client ~]# rbd pool init -p pool1
      Copy to Clipboard Toggle word wrap

  5. On a Ceph client node, bootstrap the storage cluster peers.

    1. Create Ceph user accounts, and register the storage cluster peer to the pool:

      Syntax

      rbd mirror pool peer bootstrap create --site-name LOCAL_SITE_NAME POOL_NAME > PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-a /]# rbd mirror pool peer bootstrap create --site-name site-a data > /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      This example bootstrap command creates the client.rbd-mirror.site-a and the client.rbd-mirror-peer Ceph users.

    2. Copy the bootstrap token file to the site-b storage cluster.

    3. Import the bootstrap token on the site-b storage cluster:

      Syntax

      rbd mirror pool peer bootstrap import --site-name LOCAL_SITE_NAME --direction rx-only POOL_NAME PATH_TO_BOOTSTRAP_TOKEN
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@rbd-client-site-b /]# rbd mirror pool peer bootstrap import --site-name site-b --direction rx-only data /root/bootstrap_token_site-a
      Copy to Clipboard Toggle word wrap

      Note

      For one-way RBD mirroring, you must use the --direction rx-only argument, as two-way mirroring is the default when bootstrapping peers.

  6. From a monitor node in the site-a storage cluster, verify the site-b storage cluster was successfully added as a peer:

    Example

    [ceph: root@rbd-client /]# rbd mirror pool info -p data
Mode: image
Peers:
  UUID                                 NAME   CLIENT
  d2ae0594-a43b-4c67-a167-a36c646e8643 site-b client.site-b
    Copy to Clipboard Toggle word wrap

Additional Resources

  • For detailed information, see the User Management chapter in the Red Hat Ceph Storage Administration Guide.

6.4.6.1. Fail back to the primary storage cluster
Copy link

When the formerly primary storage cluster recovers, fail back to the primary storage cluster.

Note

If you have scheduled snapshots at the image level, then you need to re-add the schedule as image resync operations changes the RBD Image ID and the previous schedule becomes obsolete.

Prerequisites

  • Minimum of two running Red Hat Ceph Storage clusters.
  • Root-level access to the node.
  • Pool mirroring or image mirroring configured with one-way mirroring.

Procedure

  1. Check the status of the images from a monitor node in the site-b cluster again. They should show a state of up-stopped and the description should say local image is primary:

    Example

    [root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 17:37:48
[root@rbd-client ~]# rbd mirror image status data/image2
image2:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 17:38:18
    Copy to Clipboard Toggle word wrap

  2. From a Ceph Monitor node on the site-a storage cluster determine if the images are still primary:

    Syntax

    rbd mirror pool info POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd info data/image1
[root@rbd-client ~]# rbd info data/image2
    Copy to Clipboard Toggle word wrap

    In the output from the commands, look for mirroring primary: true or mirroring primary: false, to determine the state.

  3. Demote any images that are listed as primary by running a command like the following from a Ceph Monitor node in the site-a storage cluster:

    Syntax

    rbd mirror image demote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image demote data/image1
    Copy to Clipboard Toggle word wrap

  4. Resynchronize the images ONLY if there was a non-orderly shutdown. Run the following commands on a monitor node in the site-a storage cluster to resynchronize the images from site-b to site-a:

    Syntax

    rbd mirror image resync POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image resync data/image1
Flagged image for resync from primary
[root@rbd-client ~]# rbd mirror image resync data/image2
Flagged image for resync from primary
    Copy to Clipboard Toggle word wrap

  5. After some time, ensure resynchronization of the images is complete by verifying they are in the up+replaying state. Check their state by running the following commands on a monitor node in the site-a storage cluster:

    Syntax

    rbd mirror image status POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image status data/image1
[root@rbd-client ~]# rbd mirror image status data/image2
    Copy to Clipboard Toggle word wrap

  6. Demote the images on the site-b storage cluster by running the following commands on a Ceph Monitor node in the site-b storage cluster:

    Syntax

    rbd mirror image demote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image demote data/image1
[root@rbd-client ~]# rbd mirror image demote data/image2
    Copy to Clipboard Toggle word wrap

    Note

    If there are multiple secondary storage clusters, this only needs to be done from the secondary storage cluster where it was promoted.

  7. Promote the formerly primary images located on the site-a storage cluster by running the following commands on a Ceph Monitor node in the site-a storage cluster:

    Syntax

    rbd mirror image promote POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image promote data/image1
[root@rbd-client ~]# rbd mirror image promote data/image2
    Copy to Clipboard Toggle word wrap

  8. Check the status of the images from a Ceph Monitor node in the site-a storage cluster. They should show a status of up+stopped and the description should say local image is primary:

    Syntax

    rbd mirror image status POOL_NAME/IMAGE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 11:14:51
[root@rbd-client ~]# rbd mirror image status data/image2
image2:
  global_id:   596f41bc-874b-4cd4-aefe-4929578cc834
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 11:14:51
    Copy to Clipboard Toggle word wrap

6.4.7. Remove two-way mirroring
Copy link

After fail back is complete, you can remove two-way mirroring and disable the Ceph block device mirroring service.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Remove the site-b storage cluster as a peer from the site-a storage cluster:

    Example

    [root@rbd-client ~]# rbd mirror pool peer remove data client.remote@remote --cluster local
[root@rbd-client ~]# rbd --cluster site-a mirror pool peer remove data client.site-b@site-b -n client.site-a
    Copy to Clipboard Toggle word wrap

  2. Stop and disable the rbd-mirror daemon on the site-a client:

    Syntax

    systemctl stop ceph-rbd-mirror@CLIENT_ID
systemctl disable ceph-rbd-mirror@CLIENT_ID
systemctl disable ceph-rbd-mirror.target
    Copy to Clipboard Toggle word wrap

    Example

    [root@rbd-client ~]# systemctl stop ceph-rbd-mirror@site-a
[root@rbd-client ~]# systemctl disable ceph-rbd-mirror@site-a
[root@rbd-client ~]# systemctl disable ceph-rbd-mirror.target
    Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat