Configuring OpenShift Data Foundation Disaster Recovery for OpenShift Workloads

1. Ensure requirements across the Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Metro-DR.
2. Ensure you meet the requirements for deploying Red Hat Ceph Storage stretch cluster with arbiter. See Requirements for deploying Red Hat Ceph Storage.
3. Deploy and configure Red Hat Ceph Storage stretch mode. For instructions on enabling Ceph cluster on two different data centers using stretched mode functionality, see Deploying Red Hat Ceph Storage.
4. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Installing OpenShift Data Foundation on managed clusters.
5. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing OpenShift Data Foundation Multicluster Orchestrator on Hub cluster.
6. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

7. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   The Metro-DR solution can only have one DRpolicy.

8. Testing your disaster recovery solution with:

   1. Subscription-based application:

      • Create sample applications. See Creating a sample Subscription-based application.
      • Test failover and relocate operations using the sample application between managed clusters. See Subscription-based application failover and relocating subscription-based application.

   2. ApplicationSet-based application:

      • Create sample applications. See Creating ApplicationSet-based applications.
      • Test failover and relocate operations using the sample application between managed clusters. See ApplicationSet-based application failover and relocating ApplicationSet-based application.

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

2. After installing the operator, create a StorageSystem using the option Full deployment type and Connect with external storage platform where your Backing storage type is Red Hat Ceph Storage.

   For detailed instructions, refer to Deploying OpenShift Data Foundation in external mode.

   Use the following flags with the ceph-external-cluster-details-exporter.py script.

   1. At a minimum, you must use the following three flags with the ceph-external-cluster-details-exporter.py script:

      --rbd-data-pool-name

      With the name of the RBD pool that was created during RHCS deployment for OpenShift Container Platform. For example, the pool can be called rbdpool.

      --rgw-endpoint

      Provide the endpoint in the format <ip_address>:<port>. It is the RGW IP of the RGW daemon running on the same site as the OpenShift Container Platform cluster that you are configuring.

      --run-as-user

      With a different client name for each site.

   2. The following flags are optional if default values were used during the RHCS deployment:

      --cephfs-filesystem-name

      With the name of the CephFS filesystem we created during RHCS deployment for OpenShift Container Platform, the default filesystem name is cephfs.

      --cephfs-data-pool-name

      With the name of the CephFS data pool we created during RHCS deployment for OpenShift Container Platform, the default pool is called cephfs.data.

      --cephfs-metadata-pool-name

      With the name of the CephFS metadata pool we created during RHCS deployment for OpenShift Container Platform, the default pool is called cephfs.meta.

   3. Run the following command on the bootstrap node ceph1, to get the IP for the RGW endpoints in datacenter1 and datacenter2:

      ceph orch ps | grep rgw.objectgw

      Example output:

      rgw.objectgw.ceph3.mecpzm  ceph3  *:8080       running (5d)     31s ago   7w     204M        -  16.2.7-112.el8cp
      rgw.objectgw.ceph6.mecpzm  ceph6  *:8080       running (5d)     31s ago   7w     204M        -  16.2.7-112.el8cp

      host ceph3.example.com
      host ceph6.example.com

      Example output:

      ceph3.example.com has address 10.0.40.24
      ceph6.example.com has address 10.0.40.66

   4. Run the ceph-external-cluster-details-exporter.py with the parameters that are configured for the first OpenShift Container Platform managed cluster cluster1 on bootstrapped node ceph1.

      python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name cephfs --cephfs-data-pool-name cephfs.cephfs.data  --cephfs-metadata-pool-name cephfs.cephfs.meta --<rgw-endpoint> XXX.XXX.XXX.XXX:8080 --run-as-user client.odf.cluster1 > ocp-cluster1.json

      Note

      Modify the <rgw-endpoint> XXX.XXX.XXX.XXX according to your environment.

   5. Run the ceph-external-cluster-details-exporter.py with the parameters that are configured for the first OpenShift Container Platform managed cluster cluster2 on bootstrapped node ceph1.

      python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name cephfs --cephfs-data-pool-name cephfs.cephfs.data  --cephfs-metadata-pool-name cephfs.cephfs.meta --rgw-endpoint XXX.XXX.XXX.XXX:8080 --run-as-user client.odf.cluster2 > ocp-cluster2.json

      Note

      Modify the <rgw-endpoint> XXX.XXX.XXX.XXX according to your environment.

      • Save the two files generated in the bootstrap cluster (ceph1) ocp-cluster1.json and ocp-cluster2.json to your local machine.
      • Use the contents of file ocp-cluster1.json on the OpenShift Container Platform console on cluster1 where external OpenShift Data Foundation is being deployed.
      • Use the contents of file ocp-cluster2.json on the OpenShift Container Platform console on cluster2 where external OpenShift Data Foundation is being deployed.
3. Review the settings and then select Create StorageSystem.

4. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-external-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   Wait for the status result to be Ready for both queries on the Primary managed cluster and the Secondary managed cluster.

5. On the OpenShift Web Console, navigate to Installed Operators → OpenShift Data Foundation → Storage System → ocs-storagecluster-storagesystem → Resources. Verify that the Status of StorageCluster is Ready and has a green tick mark next to it.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters → Data Services → Data policies.
2. Click Create DRPolicy.
3. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4perf1-ocp4perf2).
4. Select two clusters from the list of managed clusters to which this new policy will be associated with.
5. Replication policy is automatically set to sync based on the OpenShift clusters selected.
6. Click Create.

7. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

   Note

   Editing of SchedulingInterval, ReplicationClassSelector, VolumeSnapshotClassSelector and DRClusters field values are not supported in the DRPolicy.

8. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

   1. Get the names of the DRClusters on the Hub cluster.

      $ oc get drclusters

      Example output:

      NAME        AGE
      ocp4perf1   4m42s
      ocp4perf2   4m42s

   2. Check S3 access to each bucket created on each managed cluster. Use the DRCluster validation command, where <drcluster_name> is replaced with your unique name.

      Note

      Editing of Region and S3ProfileName field values are non supported in DRClusters.

      $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

      Example output:

      Succeeded

      Note

      Make sure to run commands for both DRClusters on the Hub cluster.

9. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

   $ oc get csv,pod -n openshift-dr-system

   Example output:

   NAME                                                                            DISPLAY                         VERSION        REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.14.0         Openshift DR Cluster Operator   4.14.0                    Succeeded
   clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0               VolSync                         0.8.0                     Succeeded
   
   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h

   You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed cluster.

10. Verify that the secret is propagated correctly on the Primary managed cluster and the Secondary managed cluster.

    oc get secrets -n openshift-dr-system | grep Opaque

    Match the output with the s3SecretRef from the Hub cluster:

    oc get cm -n openshift-operators ramen-hub-operator-config -oyaml

Procedure

1. Enable fencing on the Hub cluster.

   1. Open CLI terminal and edit the DRCluster resource, where <drcluster_name> is a unique name.

      Caution

      Once the managed cluster is fenced, all communication from applications to the OpenShift Data Foundation external storage cluster will fail and some Pods will be in an unhealthy state (for example: CreateContainerError, CrashLoopBackOff) on the cluster that is now fenced.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        ## Add this line
        clusterFence: Fenced
        cidrs:
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Verify the fencing status on the Hub cluster for the Primary managed cluster, replacing <drcluster_name> is your unique identifier.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Fenced

   3. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are now in the blocklist.

      $ ceph osd blocklist ls

      Example output

      cidr:10.1.161.1:0/32 2028-10-30T22:30:03.585634+0000
      cidr:10.1.161.14:0/32 2028-10-30T22:30:02.483561+0000
      cidr:10.1.161.51:0/32 2028-10-30T22:30:01.272267+0000
      cidr:10.1.161.63:0/32 2028-10-30T22:30:05.099655+0000
      cidr:10.1.161.129:0/32 2028-10-30T22:29:58.335390+0000
      cidr:10.1.161.130:0/32 2028-10-30T22:29:59.861518+0000

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Failover application.
5. After the Failover application modal is shown, select policy and target cluster to which the associated application will failover in case of a disaster.

6. Click the Select subscription group dropdown to verify the default selection or modify this setting.

   By default, the subscription group that replicates for the application resources is selected.

7. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 8.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
8. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
9. Close the modal window and track the status using the Data policy column on the Applications page.

10. Verify that the activity status shows as FailedOver for the application.

    1. Navigate to the Applications → Overview tab.
    2. In the Data policy column, click the policy link for the application you applied the policy to.
    3. On the Data policy popover, click the View more details link.

Procedure

1. Enable fencing on the Hub cluster.

   1. Open CLI terminal and edit the DRCluster resource, where <drcluster_name> is a unique name.

      Caution

      Once the managed cluster is fenced, all communication from applications to the OpenShift Data Foundation external storage cluster will fail and some Pods will be in an unhealthy state (for example: CreateContainerError, CrashLoopBackOff) on the cluster that is now fenced.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        ## Add this line
        clusterFence: Fenced
        cidrs:
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Verify the fencing status on the Hub cluster for the Primary managed cluster, replacing <drcluster_name> is your unique identifier.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Fenced

   3. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are now in the blocklist.

      $ ceph osd blocklist ls

      Example output

      cidr:10.1.161.1:0/32 2028-10-30T22:30:03.585634+0000
      cidr:10.1.161.14:0/32 2028-10-30T22:30:02.483561+0000
      cidr:10.1.161.51:0/32 2028-10-30T22:30:01.272267+0000
      cidr:10.1.161.63:0/32 2028-10-30T22:30:05.099655+0000
      cidr:10.1.161.129:0/32 2028-10-30T22:29:58.335390+0000
      cidr:10.1.161.130:0/32 2028-10-30T22:29:59.861518+0000

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Failover application.
5. When the Failover application modal is shown, verify the details presented are correct and check the status of the Failover readiness. If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start.
6. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
7. Close the modal window and track the status using the Data policy column on the Applications page.

8. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the ongoing activities associated with the policy in use with the application.

Procedure

1. Disable fencing on the Hub cluster.

   1. Edit the DRCluster resource for this cluster, replacing <drcluster_name> with a unique name.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        cidrs:
        [...]
        ## Modify this line
        clusterFence: Unfenced
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Gracefully reboot OpenShift Container Platform nodes that were Fenced. A reboot is required to resume the I/O operations after unfencing to avoid any further recovery orchestration failures. Reboot all nodes of the cluster by following the steps in the procedure, Rebooting a node gracefully.

      Note

      Make sure that all the nodes are initially cordoned and drained before you reboot and perform uncordon operations on the nodes.

   3. After all OpenShift nodes are rebooted and are in a Ready status, verify that all Pods are in a healthy state by running this command on the Primary managed cluster (or whatever cluster has been Unfenced).

      oc get pods -A | egrep -v 'Running|Completed'

      Example output:

      NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS       AGE

      The output for this query should be zero Pods before proceeding to the next step.

      Important

      If there are Pods still in an unhealthy status because of severed storage communication, troubleshoot and resolve before continuing. Because the storage cluster is external to OpenShift, it also has to be properly recovered after a site outage for OpenShift applications to be healthy.

      Alternatively, you can use the OpenShift Web Console dashboards and Overview tab to assess the health of applications and the external ODF storage cluster. The detailed OpenShift Data Foundation dashboard is found by navigating to Storage → Data Foundation.

   4. Verify that the Unfenced cluster is in a healthy state. Validate the fencing status in the Hub cluster for the Primary-managed cluster, replacing <drcluster_name> with a unique name.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Unfenced

   5. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are NOT in the blocklist.

      $ ceph osd blocklist ls

      Ensure that you do not see the IPs added during fencing.

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Relocate application.
5. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
6. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

7. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 8.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
8. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
9. Close the modal window and track the status using the Data policy column on the Applications page.

10. Verify that the activity status shows as Relocated for the application.

    1. Navigate to the Applications → Overview tab.
    2. In the Data policy column, click the policy link for the application you applied the policy to.
    3. On the Data policy popover, click the View more details link.

Procedure

1. Disable fencing on the Hub cluster.

   1. Edit the DRCluster resource for this cluster, replacing <drcluster_name> with a unique name.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        cidrs:
        [...]
        ## Modify this line
        clusterFence: Unfenced
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Gracefully reboot OpenShift Container Platform nodes that were Fenced. A reboot is required to resume the I/O operations after unfencing to avoid any further recovery orchestration failures. Reboot all nodes of the cluster by following the steps in the procedure, Rebooting a node gracefully.

      Note

      Make sure that all the nodes are initially cordoned and drained before you reboot and perform uncordon operations on the nodes.

   3. After all OpenShift nodes are rebooted and are in a Ready status, verify that all Pods are in a healthy state by running this command on the Primary managed cluster (or whatever cluster has been Unfenced).

      oc get pods -A | egrep -v 'Running|Completed'

      Example output:

      NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS       AGE

      The output for this query should be zero Pods before proceeding to the next step.

      Important

      If there are Pods still in an unhealthy status because of severed storage communication, troubleshoot and resolve before continuing. Because the storage cluster is external to OpenShift, it also has to be properly recovered after a site outage for OpenShift applications to be healthy.

      Alternatively, you can use the OpenShift Web Console dashboards and Overview tab to assess the health of applications and the external ODF storage cluster. The detailed OpenShift Data Foundation dashboard is found by navigating to Storage → Data Foundation.

   4. Verify that the Unfenced cluster is in a healthy state. Validate the fencing status in the Hub cluster for the Primary-managed cluster, replacing <drcluster_name> with a unique name.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Unfenced

   5. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are NOT in the blocklist.

      $ ceph osd blocklist ls

      Ensure that you do not see the IPs added during fencing.

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Relocate application.
5. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
6. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
7. Close the modal window and track the status using the Data policy column on the Applications page.

8. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the relocation status associated with the policy in use with the application.

1. Ensure requirements across the three: Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Regional-DR.
2. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Creating OpenShift Data Foundation cluster on managed clusters.
3. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing OpenShift Data Foundation Multicluster Orchestrator on Hub cluster.
4. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

5. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   There can be more than a single policy.

6. Testing your disaster recovery solution with:

   1. Subscription-based application:

      • Create Subscription-based applications. See Creating a sample Subscription-based application.
      • Test failover and relocate operations using the sample subscription-based application between managed clusters. See Subscription-based application failover between managed clusters and Relocating Subscription-based application between managed clusters.

   2. ApplicationSet-based application:

      • Create sample applications. See Creating ApplicationSet-based applications.
      • Test failover and relocate operations using the sample application between managed clusters. See ApplicationSet-based application failover between managed clusters and Relocating an ApplicationSet-based application between managed clusters.

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

   For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).

   Note

   While creating the storage cluster, in the Data Protection step, you must select the Prepare cluster for disaster recovery (Regional-DR only) checkbox.

2. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue with the next step.

3. In the OpenShift Web Console, navigate to Installed Operators → OpenShift Data Foundation → Storage System → ocs-storagecluster-storagesystem → Resources and verify that the Status of StorageCluster is Ready and has a green tick mark next to it.

4. [Optional] If Globalnet was enabled when Submariner was installed, then edit the StorageCluster after the OpenShift Data Foundation install finishes.

   For Globalnet networks, manually edit the StorageCluster yaml to add the clusterID and set enabled to true. Replace <clustername> with your RHACM imported or newly created managed cluster name. Edit the StorageCluster on both the Primary managed cluster and the Secondary managed cluster.

   Warning

   Do not make this change in the StorageCluster unless you enabled Globalnet when Submariner was installed.

   $ oc edit storagecluster -o yaml -n openshift-storage

   spec:
     network:
       multiClusterService:
         clusterID: <clustername>
         enabled: true

5. After the above changes are made,

   1. Wait for the OSD pods to restart and OSD services to be created.
   2. Wait for all MONS to failover.

   3. Ensure that the MONS and OSD services are exported.

      $ oc get serviceexport -n openshift-storage

      NAME              AGE
      rook-ceph-mon-d   4d14h
      rook-ceph-mon-e   4d14h
      rook-ceph-mon-f   4d14h
      rook-ceph-osd-0   4d14h
      rook-ceph-osd-1   4d14h
      rook-ceph-osd-2   4d14h

   4. Ensure that cluster is in a Ready state and cluster health has a green tick indicating Health ok. Verify using step 3.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters → Data Services → Data policies.
2. Click Create DRPolicy.
3. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4bos1-ocp4bos2-5m).
4. Select two clusters from the list of managed clusters to which this new policy will be associated with.
5. Replication policy is automatically set to Asynchronous(async) based on the OpenShift clusters selected and a Sync schedule option will become available.

6. Set Sync schedule.

   Important

   For every desired replication interval a new DRPolicy must be created with a unique name (such as: ocp4bos1-ocp4bos2-10m). The same clusters can be selected but the Sync schedule can be configured with a different replication interval in minutes/hours/days. The minimum is one minute.

7. Click Create.

8. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

   Note

   Editing of SchedulingInterval, ReplicationClassSelector, VolumeSnapshotClassSelector and DRClusters field values are not supported in the DRPolicy.

9. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

   1. Get the names of the DRClusters on the Hub cluster.

      $ oc get drclusters

      Example output:

      NAME        AGE
      ocp4bos1   4m42s
      ocp4bos2   4m42s

   2. Check S3 access to each bucket created on each managed cluster. Use the DRCluster validation command, where <drcluster_name> is replaced with your unique name.

      Note

      Editing of Region and S3ProfileName field values are non supported in DRClusters.

      $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

      Example output:

      Succeeded

      Note

      Make sure to run commands for both DRClusters on the Hub cluster.

10. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

    $ oc get csv,pod -n openshift-dr-system

    Example output:

    NAME                                                                            DISPLAY                         VERSION        REPLACES   PHASE
    clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.14.0         Openshift DR Cluster Operator   4.14.0                    Succeeded
    clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0               VolSync                         0.8.0                     Succeeded
    
    NAME                                             READY   STATUS    RESTARTS   AGE
    pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h

    You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed cluster.

    Note

    On the initial run, VolSync operator is installed automatically. VolSync is used to set up volume replication between two clusters to protect CephFs-based PVCs. The replication feature is enabled by default.

11. Verify that the status of the OpenShift Data Foundation mirroring daemon health on the Primary managed cluster and the Secondary managed cluster.

    $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

    Example output:

    {"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}

    Caution

    It could take up to 10 minutes for the daemon_health and health to go from Warning to OK. If the status does not become OK eventually, then use the RHACM console to verify that the Submariner connection between managed clusters is still in a healthy state. Do not proceed until all values are OK.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. After the Failover application modal is shown, select policy and target cluster to which the associated application will failover in case of a disaster.

5. Click the Select subscription group dropdown to verify the default selection or modify this setting.

   By default, the subscription group that replicates for the application resources is selected.

6. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 8.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. When the Failover application modal is shown, verify the details presented are correct and check the status of the Failover readiness. If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start.
5. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the ongoing activities associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

6. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 8.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. Click Initiate. All the system workloads and their available resources are now transferred to the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications → Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the relocation status associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications → Overview tab.

2. In the Data policy column, click the policy link for the application you applied the policy to.

   A Data Policies modal page appears with the number of disaster recovery policies applied to each application along with failover and relocation status.

3. On the Data Policies modal page, click the View more details link.

   A detailed Data Policies modal page is displayed that shows the policy names and the ongoing activities (Last sync, Activity status) associated with the policy that is applied to the application.

   The Last sync time reported in the modal page, represents the most recent sync time of all volumes that are DR protected for the application.

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Type local storage in the Filter by keyword​ box to find the Local Storage Operator from the list of operators, and click on it.

4. Set the following options on the Install Operator page:

   1. Update channel as stable.
   2. Installation mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-local-storage.
   4. Update approval as Automatic.
5. Click Install.

Procedure

1. Log in to the OpenShift Web Console.
2. Click Operators → OperatorHub.
3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to search for the OpenShift Data Foundation Operator.
4. Click Install.

5. Set the following options on the Install Operator page:

   1. Update Channel as stable-4.14.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

   4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you selected Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

6. Ensure that the Enable option is selected for the Console plugin.
7. Click Install.

Procedure

1. In the OpenShift Web Console, click Operators → Installed Operators to view all the installed operators.

   Ensure that the Project selected is openshift-storage.

2. Click on the OpenShift Data Foundation operator and then click Create StorageSystem.
3. In the Backing storage page, select the Create a new StorageClass using the local storage devices option.

4. Click Next.

   Important

   You are prompted to install the Local Storage Operator if it is not already installed. Click Install, and follow the procedure as described in Installing Local Storage Operator.

5. In the Create local volume set page, provide the following information:

   1. Enter a name for the LocalVolumeSet and the StorageClass.

      By default, the local volume set name appears for the storage class name. You can change the name.

   2. Choose one of the following:

      • Disks on all nodes

        Uses the available disks that match the selected filters on all the nodes.

      • Disks on selected nodes

        Uses the available disks that match the selected filters only on selected nodes.

        Important

        If the nodes selected do not match the OpenShift Data Foundation cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster is deployed.

        For minimum starting node requirements, see the Resource requirements section in the Planning guide.

   3. Select SSD or NVMe to build a supported configuration. You can select HDDs for unsupported test installations.

   4. Expand the Advanced section and set the following options:

      Volume Mode	Block is selected by default.
      Device Type	Select one or more device types from the dropdown list.
      Disk Size	Set a minimum size of 100GB for the device and maximum available size of the device that needs to be included.
      Maximum Disks Limit	This indicates the maximum number of PVs that can be created on a node. If this field is left empty, then PVs are created for all the available disks on the matching nodes.

   5. Click Next.

      A pop-up to confirm the creation of LocalVolumeSet is displayed.

   6. Click Yes to continue.

6. In the Capacity and nodes page, configure the following:

   1. Select Enable arbiter checkbox if you want to use the stretch clusters. This option is available only when all the prerequisites for arbiter are fulfilled and the selected nodes are populated. For more information, see Arbiter stretch cluster requirements in Requirements for enabling stretch cluster.

      Select the arbiter zone from the dropdown list.

   2. Available raw capacity is populated with the capacity value based on all the attached disks associated with the storage class. This takes some time to show up.

      The Selected nodes list shows the nodes based on the storage class.

   3. Click Next.

7. Optional: In the Security and network page, configure the following based on your requirement:

   1. Select the Enable encryption checkbox to encrypt block and file storage.

   2. Choose one or both of the following Encryption level:

      • Cluster-wide encryption

        Encrypts the entire cluster (block and file).

      • StorageClass encryption

        Creates encrypted persistent volume (block only) using encryption enabled storage class.

   3. Select Connect to an external key management service checkbox. This is optional for cluster-wide encryption.

      1. Key Management Service Provider is set to Vault by default.
      2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>''), Port number and Token.

   4. Expand Advanced Settings to enter the additional settings and certificate details based on your Vault configuration:

      1. Enter the Key Value secret path in the Backend Path that is dedicated and unique to OpenShift Data Foundation.
      2. Optional: Enter the TLS Server Name and Vault Enterprise Namespace.
      3. Upload the respective PEM encoded certificate file to provide the CA Certificate, Client Certificate and Client Private Key.
   5. Click Save.

   6. Choose one of the following:

      • Default (SDN)

        If you are using a single network.

      • Custom (Multus)

        If you are using multiple network interfaces.

        1. Select a Public Network Interface from the dropdown.

        2. Select a Cluster Network Interface from the dropdown.

           Note

           If you are using only one additional network interface, select the single NetworkAttachementDefinition, that is,ocs-public-cluster for the Public Network Interface, and leave the Cluster Network Interface blank.

   7. Click Next.

8. In the Review and create page, review the configuration details.

   To modify any configuration settings, click Back to go back to the previous configuration page.

9. Click Create StorageSystem.

10. For cluster-wide encryption with Key Management System (KMS), if you have used the Vault Key/Value (KV) secret engine API, version 2, then you need to edit the configmap.

    1. In the OpenShift Web Console, navigate to Workloads → ConfigMaps.
    2. To view the KMS connection details, click ocs-kms-connection-details.

    3. Edit the configmap.

       1. Click Action menu (⋮) → Edit ConfigMap.

       2. Set the VAULT_BACKEND parameter to v2.

          kind: ConfigMap
          apiVersion: v1
          metadata:
            name: ocs-kms-connection-details
          [...]
          data:
            KMS_PROVIDER: vault
            KMS_SERVICE_NAME: vault
          [...]
            VAULT_BACKEND: v2
          [...]

       3. Click Save.

1. Create a new project.

   $ oc new-project my-shared-storage

2. Deploy the example PHP application called file-uploader.

   $ oc new-app openshift/php:7.3-ubi8~https://github.com/christianh814/openshift-php-upload-demo --name=file-uploader

   Example Output:

   Found image 4f2dcc0 (9 days old) in image stream "openshift/php" under tag "7.2-ubi8" for "openshift/php:7.2-
   ubi8"
   
   Apache 2.4 with PHP 7.2
   -----------------------
   PHP 7.2 available as container is a base platform for building and running various PHP 7.2 applications and frameworks. PHP is an HTML-embedded scripting language. PHP attempts to make it easy for developers to write dynamically generated web pages. PHP also offers built-in database integration for several commercial and non-commercial database management systems, so writing a database-enabled webpage with PHP is fairly simple. The most common
   use of PHP coding is probably as a replacement for CGI scripts.
   
   Tags: builder, php, php72, php-72
   
   * A source build using source code from https://github.com/christianh814/openshift-php-upload-demo will be cr
   eated
   * The resulting image will be pushed to image stream tag "file-uploader:latest"
   * Use 'oc start-build' to trigger a new build
   
   --> Creating resources ...
       imagestream.image.openshift.io "file-uploader" created
       buildconfig.build.openshift.io "file-uploader" created
       deployment.apps "file-uploader" created
       service "file-uploader" created
   --> Success
       Build scheduled, use 'oc logs -f buildconfig/file-uploader' to track its progress.
   
       Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
        'oc expose service/file-uploader'
   
       Run 'oc status' to view your app.

3. View the build log and wait until the application is deployed.

   $ oc logs -f bc/file-uploader -n my-shared-storage

   Example Output:

   Cloning "https://github.com/christianh814/openshift-php-upload-demo" ...
   
       [...]
       Generating dockerfile with builder image image-registry.openshift-image-regis
   try.svc:5000/openshift/php@sha256:d97466f33999951739a76bce922ab17088885db610c
   0e05b593844b41d5494ea
   STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift/php@s
   ha256:d97466f33999951739a76bce922ab17088885db610c0e05b593844b41d5494ea
   STEP 2: LABEL "io.openshift.build.commit.author"="Christian Hernandez <christ
   ian.hernandez@yahoo.com>"       "io.openshift.build.commit.date"="Sun Oct 1 1
   7:15:09 2017 -0700"       "io.openshift.build.commit.id"="288eda3dff43b02f7f7
   b6b6b6f93396ffdf34cb2"       "io.openshift.build.commit.ref"="master"       "
   io.openshift.build.commit.message"="trying to modularize"       "io.openshift
   .build.source-location"="https://github.com/christianh814/openshift-php-uploa
   d-demo"       "io.openshift.build.image"="image-registry.openshift-image-regi
   stry.svc:5000/openshift/php@sha256:d97466f33999951739a76bce922ab17088885db610
   c0e05b593844b41d5494ea"
   STEP 3: ENV OPENSHIFT_BUILD_NAME="file-uploader-1"     OPENSHIFT_BUILD_NAMESP
   ACE="my-shared-storage"     OPENSHIFT_BUILD_SOURCE="https://github.com/christ
   ianh814/openshift-php-upload-demo"     OPENSHIFT_BUILD_COMMIT="288eda3dff43b0
   2f7f7b6b6b6f93396ffdf34cb2"
   STEP 4: USER root
   STEP 5: COPY upload/src /tmp/src
   STEP 6: RUN chown -R 1001:0 /tmp/src
   STEP 7: USER 1001
   STEP 8: RUN /usr/libexec/s2i/assemble
   ---> Installing application source...
   => sourcing 20-copy-config.sh ...
   ---> 17:24:39     Processing additional arbitrary httpd configuration provide
   d by s2i ...
   => sourcing 00-documentroot.conf ...
   => sourcing 50-mpm-tuning.conf ...
   => sourcing 40-ssl-certs.sh ...
   STEP 9: CMD /usr/libexec/s2i/run
   STEP 10: COMMIT temp.builder.openshift.io/my-shared-storage/file-uploader-1:3
   b83e447
   Getting image source signatures
   
   [...]

   The command prompt returns out of the tail mode after you see Push successful.

   Note

   The new-app command deploys the application directly from the git repository and does not use the OpenShift template, hence the OpenShift route resource is not created by default. You need to create the route manually.

Procedure

1. On the Hub cluster, open a terminal window

2. Add the following label to openshift-operator namespace.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

Procedure

1. On the Hub cluster, open a terminal window and perform the next steps.

2. Create the configmap file named observability-metrics-custom-allowlist.yaml.

   You can use the following YAML to list the disaster recovery metrics on Hub cluster. For details, see Adding custom metrics. To know more about ramen metrics, see Disaster recovery metrics.

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: observability-metrics-custom-allowlist
     namespace: open-cluster-management-observability
   data:
     metrics_list.yaml: |
       names:
         - ceph_rbd_mirror_snapshot_sync_bytes
         - ceph_rbd_mirror_snapshot_snapshots
       matches:
         - __name__="csv_succeeded",exported_namespace="openshift-dr-system",name=~"odr-cluster-operator.*"
         - __name__="csv_succeeded",exported_namespace="openshift-operators",name=~"volsync.*"

3. In the open-cluster-management-observability namespace, run the following command:

   $ oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml

4. After observability-metrics-custom-allowlist yaml is created, RHACM starts collecting the listed OpenShift Data Foundation metrics from all the managed clusters.

   To exclude a specific managed cluster from collecting the observability data, add the following cluster label to the clusters: observability: disabled.

Procedure

1. On the Hub cluster, ensure All Clusters option is selected.
2. Refresh the console to make the DR monitoring dashboard tab accessible.
3. Navigate to Data Services and click Data policies.
4. On the Overview tab, you can view the health status of the operators, clusters and applications. Green tick indicates that the operators are running and available..
5. Click the Disaster recovery tab to view a list of DR policy details and connected applications.