Chapter 4. Regional-DR solution for OpenShift Data Foundation

1. Ensure requirements across the three: Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Regional-DR.
2. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Creating OpenShift Data Foundation cluster on managed clusters.
3. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing ODF Multicluster Orchestrator on Hub cluster.
4. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

5. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   There can be more than a single policy.

6. Testing your disaster recovery solution with:

   1. Subscription-based application:

      • Create Subscription-based applications. See Creating sample application.
      • Test failover and relocate operations using the sample subscription-based application between managed clusters. See Subscription-based application failover and relocating subscription-based application.

   2. ApplicationSet-based application:

      • Create sample applications. See Creating ApplicationSet-based applications.
      • Test failover and relocate operations using the sample application between managed clusters. See ApplicationSet-based application failover and relocating ApplicationSet-based application.

   3. Discovered applications

      • Ensure all requirements mentioned in Prerequisites is addressed. See Prerequisites for disaster recovery protection of discovered applications
      • Create a sample discovered application. See Creating a sample discovered application
      • Enroll the discovered application. See Enrolling a sample discovered application for disaster recovery protection
      • Test failover and relocate. See Discovered application failover and relocate

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

   For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).

   Note

   While creating the storage cluster, in the Data Protection step, you must select the Prepare cluster for disaster recovery (Regional-DR only) checkbox.

2. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue with the next step.

3. In the OpenShift Web Console, navigate to Installed Operators OpenShift Data Foundation Storage System ocs-storagecluster-storagesystem Resources and verify that the Status of StorageCluster is Ready and has a green tick mark next to it.

4. [Optional] If Globalnet was enabled when Submariner was installed, then edit the StorageCluster after the OpenShift Data Foundation install finishes.

   For Globalnet networks, manually edit the StorageCluster yaml to add the clusterID and set enabled to true. Replace <clustername> with your RHACM imported or newly created managed cluster name. Edit the StorageCluster on both the Primary managed cluster and the Secondary managed cluster.

   Warning

   Do not make this change in the StorageCluster unless you enabled Globalnet when Submariner was installed.

   $ oc edit storagecluster -o yaml -n openshift-storage

   spec:
     network:
       multiClusterService:
         clusterID: <clustername>
         enabled: true

5. After the above changes are made,

   1. Wait for the OSD pods to restart and OSD services to be created.
   2. Wait for all MONS to failover.

   3. Ensure that the MONS and OSD services are exported.

      $ oc get serviceexport -n openshift-storage

      NAME              AGE
      rook-ceph-mon-d   4d14h
      rook-ceph-mon-e   4d14h
      rook-ceph-mon-f   4d14h
      rook-ceph-osd-0   4d14h
      rook-ceph-osd-1   4d14h
      rook-ceph-osd-2   4d14h

   4. Ensure that cluster is in a Ready state and cluster health has a green tick indicating Health ok. Verify using step 3.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters Data Services Disaster recovery.
2. On the Overview tab, click Create a disaster recovery policy or you can navigate to Policies tab and click Create DRPolicy.
3. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4bos1-ocp4bos2-5m).

4. Select two clusters from the list of managed clusters to which this new policy will be associated with.

   Note

   If you get an error message "OSDs not migrated" after selecting the clusters, then follow the instructions from knowledgebase article on Migration of existing OSD to the optimized OSD in OpenShift Data Foundation for Regional-DR cluster before proceeding with the next step.

5. Replication policy is automatically set to Asynchronous(async) based on the OpenShift clusters selected and a Sync schedule option will become available.

6. Set Sync schedule.

   Important

   For every desired replication interval a new DRPolicy must be created with a unique name (such as: ocp4bos1-ocp4bos2-10m). The same clusters can be selected but the Sync schedule can be configured with a different replication interval in minutes/hours/days. The minimum is one minute.

7. Click Create.

8. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

   Note

   Editing of SchedulingInterval, ReplicationClassSelector, VolumeSnapshotClassSelector and DRClusters field values are not supported in the DRPolicy.

9. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

   1. Get the names of the DRClusters on the Hub cluster.

      $ oc get drclusters

      Example output:

      NAME        AGE
      ocp4bos1   4m42s
      ocp4bos2   4m42s

   2. Check S3 access to each bucket created on each managed cluster. Use the DRCluster validation command, where <drcluster_name> is replaced with your unique name.

      Note

      Editing of Region and S3ProfileName field values are non supported in DRClusters.

      $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

      Example output:

      Succeeded

      Note

      Make sure to run commands for both DRClusters on the Hub cluster.

10. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

    $ oc get csv,pod -n openshift-dr-system

    Example output:

    NAME                                                                            DISPLAY                         VERSION        REPLACES   PHASE
    clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.15.0         Openshift DR Cluster Operator   4.15.0                    Succeeded
    clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0               VolSync                         0.8.0                     Succeeded
    
    NAME                                             READY   STATUS    RESTARTS   AGE
    pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h

    You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed cluster.

    Note

    On the initial run, VolSync operator is installed automatically. VolSync is used to set up volume replication between two clusters to protect CephFs-based PVCs. The replication feature is enabled by default.

11. Verify that the status of the OpenShift Data Foundation mirroring daemon health on the Primary managed cluster and the Secondary managed cluster.

    $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

    Example output:

    {"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}

    Caution

    It could take up to 10 minutes for the daemon_health and health to go from Warning to OK. If the status does not become OK eventually, then use the RHACM console to verify that the Submariner connection between managed clusters is still in a healthy state. Do not proceed until all values are OK.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. After the Failover application modal is shown, select policy and target cluster to which the associated application will failover in case of a disaster.

5. Click the Select subscription group dropdown to verify the default selection or modify this setting.

   By default, the subscription group that replicates for the application resources is selected.

6. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox application is now failing over to the Secondary-managed cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. When the Failover application modal is shown, verify the details presented are correct and check the status of the Failover readiness. If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start.
5. Click Initiate. The busybox resources are now created on the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the ongoing activities associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

6. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox resources are now created on the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. Click Initiate. The busybox resources are now created on the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the relocation status associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications and failover all protected applications on the failed replacement cluster to the surviving cluster.

2. Validate that all protected applications are running on the surviving cluster before moving to the next step.

   Note

   The PROGRESSION state for each application DRPlacementControl shows as Cleaning Up. This is to be expected if the replacement cluster is offline or down.

3. From the Hub cluster, delete the DRCluster for the replacement cluster.

   $ oc delete drcluster <drcluster_name> --wait=false

   Note

   Use --wait=false since the DRCluster will not be deleted until a later step.

4. Disable disaster recovery for each protected application on the surviving cluster. Perform all the sub-steps on the hub cluster.

   1. For each application, edit the Placement and ensure that the surviving cluster is selected.

      $ oc edit placement <placement_name> -n <namespace>

      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: Placement
      metadata:
      annotations:
        cluster.open-cluster-management.io/experimental-scheduling-disable: "true"
      [...]
      spec:
      clusterSets:
      - submariner
      predicates:
      - requiredClusterSelector:
          claimSelector: {}
          labelSelector:
            matchExpressions:
            - key: name
              operator: In
              values:
              - cluster1  <-- Modify to be surviving cluster name
      [...]

      Note

      For Subscription-based applications the associated Placement can be found in the same namespace on the hub cluster similar to the managed clusters. For ApplicationSets-based applications the associated Placement can be found in the openshift-gitops namespace on the hub cluster.

   2. Verify that the s3Profile is removed for the replacement cluster by running the following command on the surviving cluster for each protected application’s VolumeReplicationGroup.

      $ oc get vrg -n <application_namespace> -o jsonpath='{.items[0].spec.s3Profiles}' | jq

   3. Delete all DRPlacementControl (DRPC) resources from the Hub cluster after the protected application Placement resources are all configured to use the surviving cluster and replacement cluster s3Profile(s) removed from protected applications.

      1. Before deleting the DRPC, edit the DRPC of each application and add the annotation drplacementcontrol.ramendr.openshift.io/do-not-delete-pvc: "true".

         $ oc edit drpc {drpc_name} -n {namespace}

         apiVersion: ramendr.openshift.io/v1alpha1
            kind: DRPlacementControl
            metadata:
              annotations:
              ## Add this annotation
              drplacementcontrol.ramendr.openshift.io/do-not-delete-pvc: "true"

      2. Verify that the annotation has been copied to the associated VolumeReplicationGroup (VRG) on the surviving cluster for each protected application.

         $ oc get vrg -n {namespace} -o jsonpath='{.items[*].metadata.annotations}' | jq

      3. Delete DRPC.

         $ oc delete drpc {drpc_name} -n {namespace}

         Note

         For Subscription-based applications the associated DRPlacementControl can be found in the same namespace as the managed clusters on the hub cluster. For ApplicationSet-based applications the associated DRPlacementControl can be found in the openshift-gitops namespace on the hub cluster.

      4. Verify that all DRPlacementControl resources are deleted before proceeding to the next step. This command is a query across all namespaces. There should be no resources found.

         $ oc get drpc -A

   4. Edit each applications Placement and remove the annotation cluster.open-cluster-management.io/experimental-scheduling-disable: "true".

      $ oc edit placement {placement_name} -n {namespace}

      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: Placement
      metadata:
      annotations:
        ## Remove this annotation
        cluster.open-cluster-management.io/experimental-scheduling-disable: "true"
      [...]

5. Repeat the process detailed in the last step and the sub-steps for every protected application on the surviving cluster. Disabling DR for protected applications is now completed.

6. on the hub cluster, run the following script to remove all disaster recovery configurations from the surviving cluster and the hub cluster.

   #!/bin/bash
   
   secrets=$(oc get secrets -n openshift-operators | grep Opaque | cut -d" " -f1)
   echo $secrets
   for secret in $secrets
   do
       oc patch -n openshift-operators secret/$secret -p '{"metadata":{"finalizers":null}}' --type=merge
   done
   
   mirrorpeers=$(oc get mirrorpeer -o name)
   echo $mirrorpeers
   for mp in $mirrorpeers
   do
       oc patch $mp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $mp
   done
   
   drpolicies=$(oc get drpolicy -o name)
   echo $drpolicies
   for drp in $drpolicies
   do
       oc patch $drp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $drp
   done
   
   drclusters=$(oc get drcluster -o name)
   echo $drclusters
   for drp in $drclusters
   do
       oc patch $drp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $drp
   done
   
   oc delete project openshift-operators
   
   managedclusters=$(oc get managedclusters -o name | cut -d"/" -f2)
   echo $managedclusters
   for mc in $managedclusters
   do
       secrets=$(oc get secrets -n $mc | grep multicluster.odf.openshift.io/secret-type | cut -d" " -f1)
       echo $secrets
       for secret in $secrets
       do
           set -x
           oc patch -n $mc secret/$secret -p '{"metadata":{"finalizers":null}}' --type=merge
           oc delete -n $mc secret/$secret
       done
   done
   
   oc delete clusterrolebinding spoke-clusterrole-bindings

   Note

   This script uses the command oc delete project openshift-operators to remove the Disaster Recovery (DR) operators in this namespace on the hub cluster. If there are other non-DR operators in this namespace, you must install them again from OperatorHub.

7. After the namespace openshift-operators is automatically created again, add the monitoring label back for collecting the disaster recovery metrics.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

8. On the surviving cluster, ensure that the object bucket created during the DR installation is deleted. Delete the object bucket if it was not removed by script. The name of the object bucket used for DR starts with odrbucket.

   $ oc get obc -n openshift-storage

9. Uninstall Submariner for only the replacement cluster (failed cluster) using the RHACM console.

   1. Navigate to Infrastructure Clusters Clustersets Submariner add-ons view and uninstall Submariner for only the replacement cluster.

      Note

      The uninstall process of Submariner for the replacement cluster (failed cluster) will stay GREEN and not complete until the replacement cluster has been detached from the RHACM console.

   2. Navigate back to Clusters view and detach replacement cluster.

   3. After the Submariner add-ons for the replacement cluster (failed cluster) is removed from the RHACM console view, delete all stale Submariner resources left on the hub cluster.

      The following example is of removing the Submariner stale resources from the Hub cluster. In this example the perf5 cluster is the failed cluster and perf8 is the surviving cluster. Remember to change the resource names as per your environment.

      Note

      Do not remove resources for the surviving cluster.

      $ oc get endpoints.submariner.io,clusters.submariner.io -n submariner-broker
      NAME                                                               AGE
      endpoint.submariner.io/perf5-submariner-cable-perf5-10-70-56-163   4h2m
      endpoint.submariner.io/perf8-submariner-cable-perf8-10-70-56-90    4h7m
      
      NAME                          AGE
      cluster.submariner.io/perf5   4h2m
      cluster.submariner.io/perf8   4h7m

      Delete the replacement cluster stale resources (for example, from perf5)

      $ oc delete endpoint.submariner.io/perf5-submariner-cable-perf5-10-70-56-163
      endpoint.submariner.io "perf5-submariner-cable-perf5-10-70-56-163" deleted

      $ oc delete cluster.submariner.io/perf5
      cluster.submariner.io "perf5" deleted

   4. Create new OpenShift cluster (recovery cluster) and import into Infrastructure Clusters view.
   5. Add the new recovery cluster to the Clusterset used by Submariner.

   6. Install Submariner add-ons only for the new recovery cluster.

      Note

      If GlobalNet is used for the surviving cluster make sure to enable GlobalNet for the recovery cluster as well.

10. Install OpenShift Data Foundation on the recovery cluster. The OpenShift Data Foundation version should be OpenShift Data Foundation 4.16 (or greater) and the same version of ODF as on the surviving cluster. While creating the storage cluster, in the Data Protection step, you must select the Prepare cluster for disaster recovery (Regional-DR only) checkbox.

    Note

    Make sure to follow the optional instructions in the documentation to modify the OpenShift Data Foundation storage cluster on the recovery cluster if GlobalNet has been enabled when installing Submariner.

11. On the Hub cluster, install the ODF Multicluster Orchestrator operator from OperatorHub. For instructions, see chapter on Installing OpenShift Data Foundation Multicluster Orchestrator operator.

12. Using the RHACM console, navigate to Data Services Disaster recovery Policies tab.

    1. Select Create DRPolicy and name your policy.
    2. Select the recovery cluster and the surviving cluster.
    3. Create the policy. For instructions see chapter on Creating Disaster Recovery Policy on Hub cluster.

    Proceed to the next step only after the status of DRPolicy changes to Validated.

13. Verify that cephblockpool IDs remain unchanged.

    1. Run the following command on the recovery cluster.

       $ oc get cm -n openshift-storage rook-ceph-csi-mapping-config -o yaml

       The result is the sample output.

       apiVersion: v1
       data:
         csi-mapping-config-json: '[{"ClusterIDMapping":{"openshift-storage":"openshift-storage"},"RBDPoolIDMapping":[{"1":"1"}]}]'
       kind: ConfigMap
       [...]

    2. Run the following command on the surviving cluster.

       $ oc get cm -n openshift-storage rook-ceph-csi-mapping-config -o yaml

       The result is the sample output.

       apiVersion: v1
       data:
         csi-mapping-config-json: '[{"ClusterIDMapping":{"openshift-storage":"openshift-storage"},"RBDPoolIDMapping":[{"3":"1"}]}]'
       kind: ConfigMap
       [...]

    3. Check the RBDPoolIDMapping in the yaml for both the clusters. If RBDPoolIDMapping does not match, then edit the rook-ceph-csi-mapping-config config map of recovery cluster to add the additional or missing RBDPoolIDMapping directly as shown in the following examples.

       csi-mapping-config-json: '[{"ClusterIDMapping":{"openshift-storage":"openshift-storage"},"RBDPoolIDMapping":[{"1":"1"}]},{"ClusterIDMapping":{"openshift-storage":"openshift-storage"},"RBDPoolIDMapping":[{"1":"3"}]}]’

       Note

       After editing the configmap, restart rook-ceph-operator pod in the namespace openshift-storage on the surviving cluster by deleting the pod.

14. Apply the DRPolicy to the applications on the surviving cluster that were originally protected before the replacement cluster failed.
15. Relocate the newly protected applications on the surviving cluster back to the new recovery cluster. Using the RHACM console, navigate to the Applications menu to perform the relocation.

Procedure

1. On the Hub cluster, navigate to Applications Overview tab.

2. In the Data policy column, click the policy link for the application you applied the policy to.

   A Data Policies modal page appears with the number of disaster recovery policies applied to each application along with failover and relocation status.

3. On the Data Policies modal page, click the View more details link.

   A detailed Data Policies modal page is displayed that shows the policy names and the ongoing activities (Last sync, Activity status) associated with the policy that is applied to the application.

   The Last sync time reported in the modal page, represents the most recent sync time of all volumes that are DR protected for the application.