Chapter 4. Regional-DR solution for OpenShift Data Foundation

1. Ensure requirements across the three: Hub, Primary and Secondary Openshift Container Platform clusters that are part of the DR solution are met. See Requirements for enabling Regional-DR.
2. Install OpenShift Data Foundation operator and create a storage system on Primary and Secondary managed clusters. See Creating OpenShift Data Foundation cluster on managed clusters.
3. Install the ODF Multicluster Orchestrator on the Hub cluster. See Installing ODF Multicluster Orchestrator on Hub cluster.
4. Configure SSL access between the Hub, Primary and Secondary clusters. See Configuring SSL access across clusters.

5. Create a DRPolicy resource for use with applications requiring DR protection across the Primary and Secondary clusters. See Creating Disaster Recovery Policy on Hub cluster.

   Note

   There can be more than a single policy.

6. Testing your disaster recovery solution with:

   1. Subscription-based application:

      • Create Subscription-based applications. See Creating sample application.
      • Test failover and relocate operations using the sample subscription-based application between managed clusters. See Subscription-based application failover and relocating subscription-based application.

   2. ApplicationSet-based application:

      • Create sample applications. See Creating ApplicationSet-based applications.
      • Test failover and relocate operations using the sample application between managed clusters. See ApplicationSet-based application failover and relocating ApplicationSet-based application.

Procedure

1. Install and configure the latest OpenShift Data Foundation cluster on each of the managed clusters.

   For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).

   Note

   While creating the storage cluster, in the Data Protection step, you must select the Prepare cluster for disaster recovery (Regional-DR only) checkbox.

2. Validate the successful deployment of OpenShift Data Foundation on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   For the Multicloud Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue with the next step.

3. In the OpenShift Web Console, navigate to Installed Operators OpenShift Data Foundation Storage System ocs-storagecluster-storagesystem Resources and verify that the Status of StorageCluster is Ready and has a green tick mark next to it.

4. [Optional] If Globalnet was enabled when Submariner was installed, then edit the StorageCluster after the OpenShift Data Foundation install finishes.

   For Globalnet networks, manually edit the StorageCluster yaml to add the clusterID and set enabled to true. Replace <clustername> with your RHACM imported or newly created managed cluster name. Edit the StorageCluster on both the Primary managed cluster and the Secondary managed cluster.

   Warning

   Do not make this change in the StorageCluster unless you enabled Globalnet when Submariner was installed.

   $ oc edit storagecluster -o yaml -n openshift-storage

   spec:
     network:
       multiClusterService:
         clusterID: <clustername>
         enabled: true

5. After the above changes are made,

   1. Wait for the OSD pods to restart and OSD services to be created.
   2. Wait for all MONS to failover.

   3. Ensure that the MONS and OSD services are exported.

      $ oc get serviceexport -n openshift-storage

      NAME              AGE
      rook-ceph-mon-d   4d14h
      rook-ceph-mon-e   4d14h
      rook-ceph-mon-f   4d14h
      rook-ceph-osd-0   4d14h
      rook-ceph-osd-1   4d14h
      rook-ceph-osd-2   4d14h

   4. Ensure that cluster is in a Ready state and cluster health has a green tick indicating Health ok. Verify using step 3.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the keyword filter to search for ODF Multicluster Orchestrator.
2. Click ODF Multicluster Orchestrator tile.

3. Keep all default settings and click Install.

   Ensure that the operator resources are installed in openshift-operators project and available to all namespaces.

   Note

   The ODF Multicluster Orchestrator also installs the Openshift DR Hub Operator on the RHACM hub cluster as a dependency.

4. Verify that the operator Pods are in a Running state. The OpenShift DR Hub operator is also installed at the same time in openshift-operators namespace.

   $ oc get pods -n openshift-operators

   Example output:

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap file to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the OpenShift console, navigate to All Clusters Data Services Data policies.
2. Click Create DRPolicy.
3. Enter Policy name. Ensure that each DRPolicy has a unique name (for example: ocp4bos1-ocp4bos2-5m).

4. Select two clusters from the list of managed clusters to which this new policy will be associated with.

   Note

   If you get an error message "OSDs not migrated" after selecting the clusters, then follow the instructions from knowledgebase article on Migration of existing OSD to the optimized OSD in OpenShift Data Foundation for Regional-DR cluster before proceeding with the next step.

5. Replication policy is automatically set to Asynchronous(async) based on the OpenShift clusters selected and a Sync schedule option will become available.

6. Set Sync schedule.

   Important

   For every desired replication interval a new DRPolicy must be created with a unique name (such as: ocp4bos1-ocp4bos2-10m). The same clusters can be selected but the Sync schedule can be configured with a different replication interval in minutes/hours/days. The minimum is one minute.

7. Click Create.

8. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with your unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

   When a DRPolicy is created, along with it, two DRCluster resources are also created. It could take up to 10 minutes for all three resources to be validated and for the status to show as Succeeded.

   Note

   Editing of SchedulingInterval, ReplicationClassSelector, VolumeSnapshotClassSelector and DRClusters field values are not supported in the DRPolicy.

9. Verify the object bucket access from the Hub cluster to both the Primary managed cluster and the Secondary managed cluster.

   1. Get the names of the DRClusters on the Hub cluster.

      $ oc get drclusters

      Example output:

      NAME        AGE
      ocp4bos1   4m42s
      ocp4bos2   4m42s

   2. Check S3 access to each bucket created on each managed cluster. Use the DRCluster validation command, where <drcluster_name> is replaced with your unique name.

      Note

      Editing of Region and S3ProfileName field values are non supported in DRClusters.

      $ oc get drcluster <drcluster_name> -o jsonpath='{.status.conditions[2].reason}{"\n"}'

      Example output:

      Succeeded

      Note

      Make sure to run commands for both DRClusters on the Hub cluster.

10. Verify that the OpenShift DR Cluster operator installation was successful on the Primary managed cluster and the Secondary managed cluster.

    $ oc get csv,pod -n openshift-dr-system

    Example output:

    NAME                                                                            DISPLAY                         VERSION        REPLACES   PHASE
    clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.15.0         Openshift DR Cluster Operator   4.15.0                    Succeeded
    clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0               VolSync                         0.8.0                     Succeeded
    
    NAME                                             READY   STATUS    RESTARTS   AGE
    pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h

    You can also verify that OpenShift DR Cluster Operator is installed successfully on the OperatorHub of each managed cluster.

    Note

    On the initial run, VolSync operator is installed automatically. VolSync is used to set up volume replication between two clusters to protect CephFs-based PVCs. The replication feature is enabled by default.

11. Verify that the status of the OpenShift Data Foundation mirroring daemon health on the Primary managed cluster and the Secondary managed cluster.

    $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

    Example output:

    {"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}

    Caution

    It could take up to 10 minutes for the daemon_health and health to go from Warning to OK. If the status does not become OK eventually, then use the RHACM console to verify that the Submariner connection between managed clusters is still in a healthy state. Do not proceed until all values are OK.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. After the Failover application modal is shown, select policy and target cluster to which the associated application will failover in case of a disaster.

5. Click the Select subscription group dropdown to verify the default selection or modify this setting.

   By default, the subscription group that replicates for the application resources is selected.

6. Check the status of the Failover readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox application is now failing over to the Secondary-managed cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Failover application.
4. When the Failover application modal is shown, verify the details presented are correct and check the status of the Failover readiness. If the status is Ready with a green tick, it indicates that the target cluster is ready for failover to start.
5. Click Initiate. The busybox resources are now created on the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as FailedOver for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the ongoing activities associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. By default, the subscription group that will deploy the application resources is selected. Click the Select subscription group dropdown to verify the default selection or modify this setting.

6. Check the status of the Relocation readiness.

   • If the status is Ready with a green tick, it indicates that the target cluster is ready for relocation to start. Proceed to step 7.
   • If the status is Unknown or Not ready, then wait until the status changes to Ready.
7. Click Initiate. The busybox resources are now created on the target cluster.
8. Close the modal window and track the status using the Data policy column on the Applications page.

9. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, click the View more details link.
   4. Verify that you can see one or more policy names and the ongoing activities (Last sync time and Activity status) associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications.
2. Click the Actions menu at the end of application row to view the list of available actions.
3. Click Relocate application.
4. When the Relocate application modal is shown, select policy and target cluster to which the associated application will relocate to in case of a disaster.
5. Click Initiate. The busybox resources are now created on the target cluster.
6. Close the modal window and track the status using the Data policy column on the Applications page.

7. Verify that the activity status shows as Relocated for the application.

   1. Navigate to the Applications Overview tab.
   2. In the Data policy column, click the policy link for the application you applied the policy to.
   3. On the Data policy popover, verify that you can see one or more policy names and the relocation status associated with the policy in use with the application.

Procedure

1. On the Hub cluster, navigate to Applications Overview tab.

2. In the Data policy column, click the policy link for the application you applied the policy to.

   A Data Policies modal page appears with the number of disaster recovery policies applied to each application along with failover and relocation status.

3. On the Data Policies modal page, click the View more details link.

   A detailed Data Policies modal page is displayed that shows the policy names and the ongoing activities (Last sync, Activity status) associated with the policy that is applied to the application.

   The Last sync time reported in the modal page, represents the most recent sync time of all volumes that are DR protected for the application.