Chapter 3. Metro-DR solution for OpenShift Data Foundation

1. Ensure requirements across the Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Metro-DR.
2. Ensure you meet the requirements for deploying Red Hat Ceph Storage stretch cluster with arbiter. See Requirements for deploying Red Hat Ceph Storage.
3. Deploy and configure Red Hat Ceph Storage stretch mode. For instructions on enabling Ceph cluster on two different data centers using stretched mode functionality, see Deploying Red Hat Ceph Storage.
4. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Installing OpenShift Data Foundation on managed clusters.
5. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing ODF Multicluster Orchestrator on Hub cluster.
6. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

7. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   The Metro-DR solution can only have one DRpolicy.

8. Testing your disaster recovery solution with:

   1. Subscription-based application:

      • Create sample applications. See Creating sample application.
      • Test failover and relocate operations using the sample application between managed clusters. See Subscription-based application failover and relocating subscription-based application.

   2. ApplicationSet-based application:

      • Create sample applications. See Creating ApplicationSet-based applications.
      • Test failover and relocate operations using the sample application between managed clusters. See ApplicationSet-based application failover and relocating ApplicationSet-based application.

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

2. After installing the operator, create a StorageSystem using the option Full deployment type and Connect with external storage platform where your Backing storage type is Red Hat Ceph Storage.

   For detailed instructions, refer to Deploying OpenShift Data Foundation in external mode.

   Use the following flags with the ceph-external-cluster-details-exporter.py script.

   1. At a minimum, you must use the following three flags with the ceph-external-cluster-details-exporter.py script:

      --rbd-data-pool-name

      With the name of the RBD pool that was created during RHCS deployment for OpenShift Container Platform. For example, the pool can be called rbdpool.

      --rgw-endpoint

      Provide the endpoint in the format <ip_address>:<port>. It is the RGW IP of the RGW daemon running on the same site as the OpenShift Container Platform cluster that you are configuring.

      --run-as-user

      With a different client name for each site.

   2. The following flags are optional if default values were used during the RHCS deployment:

      --cephfs-filesystem-name

      With the name of the CephFS filesystem we created during RHCS deployment for OpenShift Container Platform, the default filesystem name is cephfs.

      --cephfs-data-pool-name

      With the name of the CephFS data pool we created during RHCS deployment for OpenShift Container Platform, the default pool is called cephfs.data.

      --cephfs-metadata-pool-name

      With the name of the CephFS metadata pool we created during RHCS deployment for OpenShift Container Platform, the default pool is called cephfs.meta.

   3. Run the following command on the bootstrap node ceph1, to get the IP for the RGW endpoints in datacenter1 and datacenter2:

      ceph orch ps | grep rgw.objectgw

      Example output:

      rgw.objectgw.ceph3.mecpzm  ceph3  *:8080       running (5d)     31s ago   7w     204M        -  16.2.7-112.el8cp
      rgw.objectgw.ceph6.mecpzm  ceph6  *:8080       running (5d)     31s ago   7w     204M        -  16.2.7-112.el8cp

      host ceph3.example.com
      host ceph6.example.com

      Example output:

      ceph3.example.com has address 10.0.40.24
      ceph6.example.com has address 10.0.40.66

   4. Run the ceph-external-cluster-details-exporter.py with the parameters that are configured for the first OpenShift Container Platform managed cluster cluster1 on bootstrapped node ceph1.

      python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name cephfs --cephfs-data-pool-name cephfs.cephfs.data  --cephfs-metadata-pool-name cephfs.cephfs.meta --<rgw-endpoint> XXX.XXX.XXX.XXX:8080 --run-as-user client.odf.cluster1 > ocp-cluster1.json

      Note

      Modify the <rgw-endpoint> XXX.XXX.XXX.XXX according to your environment.

   5. Run the ceph-external-cluster-details-exporter.py with the parameters that are configured for the first OpenShift Container Platform managed cluster cluster2 on bootstrapped node ceph1.

      python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbdpool --cephfs-filesystem-name cephfs --cephfs-data-pool-name cephfs.cephfs.data  --cephfs-metadata-pool-name cephfs.cephfs.meta --rgw-endpoint XXX.XXX.XXX.XXX:8080 --run-as-user client.odf.cluster2 > ocp-cluster2.json

      Note

      Modify the <rgw-endpoint> XXX.XXX.XXX.XXX according to your environment.

      • Save the two files generated in the bootstrap cluster (ceph1) ocp-cluster1.json and ocp-cluster2.json to your local machine.
      • Use the contents of file ocp-cluster1.json on the OpenShift Container Platform console on cluster1 where external OpenShift Data Foundation is being deployed.
      • Use the contents of file ocp-cluster2.json on the OpenShift Container Platform console on cluster2 where external OpenShift Data Foundation is being deployed.
3. Review the settings and then select Create StorageSystem.

4. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-external-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   Wait for the status result to be Ready for both queries on the Primary managed cluster and the Secondary managed cluster.

5. On the OpenShift Web Console, navigate to Installed Operators OpenShift Data Foundation Storage System ocs-external-storagecluster-storagesystem Resources. Verify that the Status of StorageCluster is Ready and has a green tick mark next to it.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters Data Services Data policies.
2. Click Create DRPolicy.
3. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4perf1-ocp4perf2).
4. Select two clusters from the list of managed clusters to which this new policy will be associated with.
5. Replication policy is automatically set to sync based on the OpenShift clusters selected.
6. Click Create.

7. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

   Note

   Editing of SchedulingInterval, ReplicationClassSelector, VolumeSnapshotClassSelector and DRClusters field values are not supported in the DRPolicy.

8. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

   1. Get the names of the DRClusters on the Hub cluster.

      $ oc get drclusters

      Example output:

   2. Check S3 access to each bucket created on each managed cluster. Use the DRCluster validation command, where <drcluster_name> is replaced with your unique name.

      Note

      Editing of Region and S3ProfileName field values are non supported in DRClusters.

      $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

      Example output:

      Succeeded

      Note

      Make sure to run commands for both DRClusters on the Hub cluster.

9. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

   $ oc get csv,pod -n openshift-dr-system

   Example output:

   NAME                                                                            DISPLAY                         VERSION        REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.15.0         Openshift DR Cluster Operator   4.15.0                    Succeeded
   clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0               VolSync                         0.8.0                     Succeeded
   
   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h

   You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed cluster.

10. Verify that the secret is propagated correctly on the Primary managed cluster and the Secondary managed cluster.

    oc get secrets -n openshift-dr-system | grep Opaque

    Match the output with the s3SecretRef from the Hub cluster:

    oc get cm -n openshift-operators ramen-hub-operator-config -oyaml

Procedure

1. Enable fencing on the Hub cluster.

   1. Open CLI terminal and edit the DRCluster resource, where <drcluster_name> is your unique name.

      Caution

      Once the managed cluster is fenced, all communication from applications to the OpenShift Data Foundation external storage cluster will fail and some Pods will be in an unhealthy state (for example: CreateContainerError, CrashLoopBackOff) on the cluster that is now fenced.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        ## Add this line
        clusterFence: Fenced
        cidrs:
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Verify the fencing status on the Hub cluster for the Primary managed cluster, replacing <drcluster_name> is your unique identifier.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Fenced

   3. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are now in the blocklist.

      $ ceph osd blocklist ls

      Example output

      cidr:10.1.161.1:0/32 2028-10-30T22:30:03.585634+0000
      cidr:10.1.161.14:0/32 2028-10-30T22:30:02.483561+0000
      cidr:10.1.161.51:0/32 2028-10-30T22:30:01.272267+0000
      cidr:10.1.161.63:0/32 2028-10-30T22:30:05.099655+0000
      cidr:10.1.161.129:0/32 2028-10-30T22:29:58.335390+0000
      cidr:10.1.161.130:0/32 2028-10-30T22:29:59.861518+0000

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Failover application.
5. After the Failover application modal is shown, select policy and target cluster to which the associated application will failover in case of a disaster.

6. Click the Select subscription group dropdown to verify the default selection or modify this setting.

   By default, the subscription group that replicates for the application resources is selected.

7. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
8. Click Initiate. The busybox application is now failing over to the Secondary-managed cluster.
9. Close the modal window and track the status using the Data policy column on the Applications page.

10. Verify that the activity status shows as FailedOver for the application.

    1. Navigate to the Applications Overview tab.
    2. In the Data policy column, click the policy link for the application you applied the policy to.
    3. On the Data policy popover, click the View more details link.

Procedure

1. Enable fencing on the Hub cluster.

   1. Open CLI terminal and edit the DRCluster resource, where <drcluster_name> is your unique name.

      Caution

      Once the managed cluster is fenced, all communication from applications to the OpenShift Data Foundation external storage cluster will fail and some Pods will be in an unhealthy state (for example: CreateContainerError, CrashLoopBackOff) on the cluster that is now fenced.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        ## Add this line
        clusterFence: Fenced
        cidrs:
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Verify the fencing status on the Hub cluster for the Primary managed cluster, replacing <drcluster_name> is your unique identifier.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Fenced

   3. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are now in the blocklist.

      $ ceph osd blocklist ls

      Example output

      cidr:10.1.161.1:0/32 2028-10-30T22:30:03.585634+0000
      cidr:10.1.161.14:0/32 2028-10-30T22:30:02.483561+0000
      cidr:10.1.161.51:0/32 2028-10-30T22:30:01.272267+0000
      cidr:10.1.161.63:0/32 2028-10-30T22:30:05.099655+0000
      cidr:10.1.161.129:0/32 2028-10-30T22:29:58.335390+0000
      cidr:10.1.161.130:0/32 2028-10-30T22:29:59.861518+0000

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Failover application.
5. When the Failover application modal is shown, verify the details presented are correct and check the status of the Failover readiness. If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start.
6. Click Initiate. The busybox resources are now created on the target cluster.
7. Close the modal window and track the status using the Data policy column on the Applications page.

8. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the ongoing activities associated with the policy in use with the application.

Procedure

1. Disable fencing on the Hub cluster.

   1. Edit the DRCluster resource for this cluster, replacing <drcluster_name> with a unique name.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        cidrs:
        [...]
        ## Modify this line
        clusterFence: Unfenced
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Gracefully reboot OpenShift Container Platform nodes that were Fenced. A reboot is required to resume the I/O operations after unfencing to avoid any further recovery orchestration failures. Reboot all nodes of the cluster by following the steps in the procedure, Rebooting a node gracefully.

      Note

      Make sure that all the nodes are initially cordoned and drained before you reboot and perform uncordon operations on the nodes.

   3. After all OpenShift nodes are rebooted and are in a Ready status, verify that all Pods are in a healthy state by running this command on the Primary managed cluster (or whatever cluster has been Unfenced).

      oc get pods -A | egrep -v 'Running|Completed'

      Example output:

      NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS       AGE

      The output for this query should be zero Pods before proceeding to the next step.

      Important

      If there are Pods still in an unhealthy status because of severed storage communication, troubleshoot and resolve before continuing. Because the storage cluster is external to OpenShift, it also has to be properly recovered after a site outage for OpenShift applications to be healthy.

      Alternatively, you can use the OpenShift Web Console dashboards and Overview tab to assess the health of applications and the external ODF storage cluster. The detailed OpenShift Data Foundation dashboard is found by navigating to Storage Data Foundation.

   4. Verify that the Unfenced cluster is in a healthy state. Validate the fencing status in the Hub cluster for the Primary-managed cluster, replacing <drcluster_name> with a unique name.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Unfenced

   5. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are NOT in the blocklist.

      $ ceph osd blocklist ls

      Ensure that you do not see the IPs added during fencing.

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Relocate application.
5. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
6. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

7. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
8. Click Initiate. The busybox resources are now created on the target cluster.
9. Close the modal window and track the status using the Data policy column on the Applications page.

10. Verify that the activity status shows as Relocated for the application.

    1. Navigate to the Applications Overview tab.
    2. In the Data policy column, click the policy link for the application you applied the policy to.
    3. On the Data policy popover, click the View more details link.

Procedure

1. Disable fencing on the Hub cluster.

   1. Edit the DRCluster resource for this cluster, replacing <drcluster_name> with a unique name.

      $ oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        cidrs:
        [...]
        ## Modify this line
        clusterFence: Unfenced
        [...]
      [...]

      Example output:

      drcluster.ramendr.openshift.io/ocp4perf1 edited

   2. Gracefully reboot OpenShift Container Platform nodes that were Fenced. A reboot is required to resume the I/O operations after unfencing to avoid any further recovery orchestration failures. Reboot all nodes of the cluster by following the steps in the procedure, Rebooting a node gracefully.

      Note

      Make sure that all the nodes are initially cordoned and drained before you reboot and perform uncordon operations on the nodes.

   3. After all OpenShift nodes are rebooted and are in a Ready status, verify that all Pods are in a healthy state by running this command on the Primary managed cluster (or whatever cluster has been Unfenced).

      oc get pods -A | egrep -v 'Running|Completed'

      Example output:

      NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS       AGE

      The output for this query should be zero Pods before proceeding to the next step.

      Important

      If there are Pods still in an unhealthy status because of severed storage communication, troubleshoot and resolve before continuing. Because the storage cluster is external to OpenShift, it also has to be properly recovered after a site outage for OpenShift applications to be healthy.

      Alternatively, you can use the OpenShift Web Console dashboards and Overview tab to assess the health of applications and the external ODF storage cluster. The detailed OpenShift Data Foundation dashboard is found by navigating to Storage Data Foundation.

   4. Verify that the Unfenced cluster is in a healthy state. Validate the fencing status in the Hub cluster for the Primary-managed cluster, replacing <drcluster_name> with a unique name.

      $ oc get drcluster.ramendr.openshift.io <drcluster_name> -o jsonpath='{.status.phase}{"\n"}'

      Example output:

      Unfenced

   5. Verify that the IPs that belong to the OpenShift Container Platform cluster nodes are NOT in the blocklist.

      $ ceph osd blocklist ls

      Ensure that you do not see the IPs added during fencing.

2. On the Hub cluster, navigate to Applications.
3. Click the Actions menu at the end of application row to view the list of available actions.
4. Click Relocate application.
5. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
6. Click Initiate. The busybox resources are now created on the target cluster.
7. Close the modal window and track the status using the Data policy column on the Applications page.

8. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the relocation status associated with the policy in use with the application.

Procedure

1. Perform the following steps on the Hub cluster:

   1. Fence the replacement cluster by using the CLI terminal to edit the DRCluster resource, where <drcluster_name> is the replacement cluster name.

      oc edit drcluster <drcluster_name>

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRCluster
      metadata:
      [...]
      spec:
        ## Add or modify this line
        clusterFence: Fenced
        cidrs:
        [...]
      [...]

   2. Using the RHACM console, navigate to Applications and failover all protected applications from the failed cluster to the surviving cluster.

   3. Verify and ensure that all protected applications are now running on the surviving cluster.

      Note

      The PROGRESSION state for each application DRPlacementControl will show as Cleaning Up. This is expected if the replacement cluster is offline or down.

2. Unfence the replacement cluster.

   Using the CLI terminal, edit the DRCluster resource, where <drcluster_name> is the replacement cluster name.

   $ oc edit drcluster <drcluster_name>

   apiVersion: ramendr.openshift.io/v1alpha1
   kind: DRCluster
   metadata:
   [...]
   spec:
     ## Modify this line
     clusterFence: Unfenced
     cidrs:
     [...]
   [...]

3. Delete the DRCluster for the replacement cluster.

   $ oc delete drcluster <drcluster_name> --wait=false

   Note

   Use --wait=false since the DRCluster will not be deleted until a later step.

4. Disable disaster recovery on the Hub cluster for each protected application on the surviving cluster.

   1. For each application, edit the Placement and ensure that the surviving cluster is selected.

      Note

      For Subscription-based applications the associated Placement can be found in the same namespace on the hub cluster similar to the managed clusters. For ApplicationSets-based applications the associated Placement can be found in the openshift-gitops namespace on the hub cluster.

      $ oc edit placement <placement_name> -n <namespace>

      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: Placement
      metadata:
      annotations:
        cluster.open-cluster-management.io/experimental-scheduling-disable: "true"
      [...]
      spec:
      clusterSets:
      - submariner
      predicates:
      - requiredClusterSelector:
          claimSelector: {}
          labelSelector:
            matchExpressions:
            - key: name
              operator: In
              values:
              - cluster1  <-- Modify to be surviving cluster name
      [...]

   2. Verify that the s3Profile is removed for the replacement cluster by running the following command on the surviving cluster for each protected application’s VolumeReplicationGroup.

      $ oc get vrg -n <application_namespace> -o jsonpath='{.items[0].spec.s3Profiles}' | jq

   3. After the protected application Placement resources are all configured to use the surviving cluster and replacement cluster s3Profile(s) removed from protected applications, all DRPlacementControl resources must be deleted from the Hub cluster.

      $ oc delete drpc <drpc_name> -n <namespace>

      Note

      For Subscription-based applications the associated DRPlacementControl can be found in the same namespace as the managed clusters on the hub cluster. For ApplicationSets-based applications the associated DRPlacementControl can be found in the openshift-gitops namespace on the hub cluster.

   4. Verify that all DRPlacementControl resources are deleted before proceeding to the next step. This command is a query across all namespaces. There should be no resources found.

      $ oc get drpc -A

   5. The last step is to edit each applications Placement and remove the annotation cluster.open-cluster-management.io/experimental-scheduling-disable: "true".

      $ oc edit placement <placement_name> -n <namespace>

      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: Placement
      metadata:
      annotations:
        ## Remove this annotation
        cluster.open-cluster-management.io/experimental-scheduling-disable: "true"
      [...]

5. Repeat the process detailed in the last step and the sub-steps for every protected application on the surviving cluster. Disabling DR for protected applications is now completed.

6. On the Hub cluster, run the following script to remove all disaster recovery configurations from the surviving cluster and the hub cluster.

   #!/bin/bash
   secrets=$(oc get secrets -n openshift-operators | grep Opaque | cut -d" " -f1)
   echo $secrets
   for secret in $secrets
   do
       oc patch -n openshift-operators secret/$secret -p '{"metadata":{"finalizers":null}}' --type=merge
   done
   mirrorpeers=$(oc get mirrorpeer -o name)
   echo $mirrorpeers
   for mp in $mirrorpeers
   do
       oc patch $mp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $mp
   done
   drpolicies=$(oc get drpolicy -o name)
   echo $drpolicies
   for drp in $drpolicies
   do
       oc patch $drp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $drp
   done
   drclusters=$(oc get drcluster -o name)
   echo $drclusters
   for drp in $drclusters
   do
       oc patch $drp -p '{"metadata":{"finalizers":null}}' --type=merge
       oc delete $drp
   done
   oc delete project openshift-operators
   managedclusters=$(oc get managedclusters -o name | cut -d"/" -f2)
   echo $managedclusters
   for mc in $managedclusters
   do
       secrets=$(oc get secrets -n $mc | grep multicluster.odf.openshift.io/secret-type | cut -d" " -f1)
       echo $secrets
       for secret in $secrets
       do
           set -x
           oc patch -n $mc secret/$secret -p '{"metadata":{"finalizers":null}}' --type=merge
           oc delete -n $mc secret/$secret
       done
   done
   
   oc delete clusterrolebinding spoke-clusterrole-bindings

   Note

   This script used the command oc delete project openshift-operators to remove the Disaster Recovery (DR) operators in this namespace on the hub cluster. If there are other non-DR operators in this namespace, you must install them again from OperatorHub.

7. After the namespace openshift-operators is automatically created again, add the monitoring label back for collecting the disaster recovery metrics.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

8. On the surviving cluster, ensure that the object bucket created during the DR installation is deleted. Delete the object bucket if it was not removed by script. The name of the object bucket used for DR starts with odrbucket.

   $ oc get obc -n openshift-storage

9. On the RHACM console, navigate to Infrastructure Clusters view.

   1. Detach the replacement cluster.
   2. Create a new OpenShift cluster (recovery cluster) and import the new cluster into the RHACM console. For instructions, see Creating a cluster and Importing a target managed cluster to the hub cluster.

10. Install OpenShift Data Foundation operator on the recovery cluster and connect it to the same external Ceph storage system as the surviving cluster. For detailed instructions, refer to Deploying OpenShift Data Foundation in external mode.

    Note

    Ensure that the OpenShift Data Foundation version is 4.15 (or greater) and the same version of OpenShift Data Foundation is on the surviving cluster.

11. On the hub cluster, install the ODF Multicluster Orchestrator operator from OperatorHub. For instructions, see chapter on Installing OpenShift Data Foundation Multicluster Orchestrator operator.

12. Using the RHACM console, navigate to Data Services Data policies.

    1. Select Create DRPolicy and name your policy.
    2. Select the recovery cluster and the surviving cluster.
    3. Create the policy. For instructions see chapter on Creating Disaster Recovery Policy on Hub cluster.

    Proceed to the next step only after the status of DRPolicy changes to Validated.

13. Apply the DRPolicy to the applications on the surviving cluster that were originally protected before the replacement cluster failed.
14. Relocate the newly protected applications on the surviving cluster back to the new recovery (primary) cluster. Using the RHACM console, navigate to the Applications menu to perform the relocation.