Chapter 4. Regional-DR solution for OpenShift Data Foundation [Technology Preview]

1. Ensure requirements across the three: Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Regional-DR.
2. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Creating OpenShift Data Foundation cluster on managed clusters.
3. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing ODF Multicluster Orchestrator on Hub cluster.
4. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

5. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   There can be more than a single policy.

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

   For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).

2. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue with the next step.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters.

   
2. Navigate to Data Services and click Data policies.
3. Click Create DRPolicy.
4. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4bos1-ocp4bos2-5m).
5. Select two clusters from the list of managed clusters to which this new policy will be associated with.
6. Replication policy is automaticaly set to Asynchronous(async) based on the OpenShift clusters selected and a Sync schedule option will become available.

7. Set Sync schedule.

   Important

   For every desired replication interval a new DRPolicy must be created with a unique name (such as: ocp4bos1-ocp4bos2-10m). The same clusters can be selected but the Sync schedule can be configured with a different replication interval in minutes/hours/days. The minimum is one minute.

8. Click Create.

9. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created.

   Note

   Replace <drpolicy_name> with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   Note

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

10. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

    1. Get the names of the DRClusters on the Hub cluster.

       $ oc get drclusters

       Example output:

       NAME        AGE
       ocp4bos1   4m42s
       ocp4bos2   4m42s

    2. Check S3 access to each bucket created on each managed cluster using this DRCluster validation command.

       Note

       Replace <drcluster_name> with your unique name.

       $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

       Example output:

       Succeeded

       Note

       Make sure to run command for both DRClusters on the Hub cluster.

11. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

    $ oc get csv,pod -n openshift-dr-system

    Example output:

    NAME                                                                      DISPLAY                         VERSION   REPLACES   PHASE
    clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.11.0   Openshift DR Cluster Operator   4.11.0               Succeeded
    
    NAME                                             READY   STATUS    RESTARTS   AGE
    pod/ramen-dr-cluster-operator-5564f9d669-f6lbc   2/2     Running   0          5m32s

    You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed clusters.

12. Verify that the status of the ODF mirroring daemon health on the Primary managed cluster and the Secondary managed cluster.

    $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

    Example output:

    {"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}

    Caution

    It could take up to 10 minutes for the daemon_health and health to go from Warning to OK. If the status does not become OK eventually then use the RHACM console to verify that the Submariner connection between managed clusters is still in a healthy state. Do not proceed until all values are OK.

13. When using VolSync to protect CephFs-based PVCs, then configure the VolSync copy method. The default copy method is to use snapshot. A snapshot is taken at the source and synced to the temporary destination PVC. Once the syncronization is complete, another snapshot is taken from this temporary PVC and saved on the destination cluster. On failover, the application PVC is restored from the latest snapshot found on the cluster.

    Using a snapshot as a copy method may not be desirable when using PVCs that contain thousands of files as CephFS will take a long time to create a writable PVC from snapshot. Furthermore, when using the copy method as snapshot, after a failover or replication, the entire PVC must be syncronized to the other side. This is a very expensive operation on high latency network and big PVC size.

    To avoid these issues, a “direct” copy method can be used instead. This method is preferred as synchronization is done directly to the application PVC, and a snapshot is also saved in case manual restoration is required.

    1. You can configure the copy method “direct” as follows:

       $ oc edit cm -n openshift-operators ramen-hub-operator-config

    2. Add the following to spec.data.ramen_manager_config.yaml section:

           volsync:
             destinationCopyMethod: Direct

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. When the Failover application popup is shown, select policy and target cluster to which the associated application will failover in case of a disaster.
5. By default, the subscription group that will replicate the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

6. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox resources are now created on the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data Policies modal page, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application popup is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

6. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox resources are now created on the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data Policies modal page, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications Overview tab.

2. In the Data policy column, click the policy link for the application you applied the policy to.

   A Data Policies modal page appears with the number of disaster recovery policies applied to each application along with failover and relocation status.

3. On the Data Policies modal page, click the View more details link.

   A detailed Data Policies modal page is displayed that shows the policy names and the ongoing activities (Last sync, Activity status) associated with the policy that is applied to the application.

   The Last sync time reported in the modal page, represents the most recent sync time of all volumes that are DR protected for the application.