Configuring OpenShift Data Foundation for Regional-DR with Advanced Cluster Management

Procedure

1. Install OpenShift Data Foundation version 4.10 on each of the managed clusters.

   For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).

2. Validate the successful deployment on each managed cluster with the following command:

   $ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   and for the Multicloud Object Gateway (MCG):

   $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue on to enabling mirroring on the managed clusters.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the search filter for OpenShift DR Hub Operator.
2. Follow the screen instructions to Install the operator into the project openshift-dr-system.

3. Verify that the operator Pod is in Running state using the following command:

   $ oc get pods -n openshift-dr-system

   Example output:

   NAME                                 	READY   STATUS    RESTARTS   AGE
   ramen-hub-operator-898c5989b-96k65   	2/2     Running   0          4m14s

Procedure

1. Save the following YAML to filename ocs-storagecluster-ceph-rbdmirror.yaml.

   allowVolumeExpansion: true
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: ocs-storagecluster-ceph-rbdmirror
   parameters:
     clusterID: openshift-storage
     csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
     csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
     csi.storage.k8s.io/fstype: ext4
     csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
     csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
     csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
     csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
     imageFeatures: layering,exclusive-lock,object-map,fast-diff
     imageFormat: "2"
     pool: ocs-storagecluster-cephblockpool
   provisioner: openshift-storage.rbd.csi.ceph.com
   reclaimPolicy: Delete
   volumeBindingMode: Immediate

2. Create the file on both the managed clusters.

   $ oc create -f ocs-storagecluster-ceph-rbdmirror.yaml

   Example output:

   storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbdmirror created

Procedure

1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

   $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

3. Create a new ConfigMap to hold the remote cluster’s certificate bundle with filename cm-clusters-crt.yaml on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   Note

   There could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the primary.crt and secondary.crt files that were created before.

   apiVersion: v1
   data:
     ca-bundle.crt: |
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from primary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 primary.crt here>
       -----END CERTIFICATE----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert1 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert2 from secondary.crt here>
       -----END CERTIFICATE-----
   
       -----BEGIN CERTIFICATE-----
       <copy contents of cert3 from secondary.crt here>
       -----END CERTIFICATE-----
   kind: ConfigMap
   metadata:
     name: user-ca-bundle
     namespace: openshift-config

4. Create the ConfigMap file on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc create -f cm-clusters-crt.yaml

   Example output:

   configmap/user-ca-bundle created

   Important

   For the Hub cluster to verify access to the object buckets using the DRPolicy resource, the same ConfigMap cm-clusters-crt.yaml must also be created on the Hub cluster.

5. Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

   $ oc patch proxy cluster --type=merge  --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'

   Example output:

   proxy.config.openshift.io/cluster patched

Procedure

1. On the Hub cluster, navigate to Installed Operators in the openshift-dr-system project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
2. Click Create instance for DRPolicy and click YAML view.

3. Copy and save the following YAML to filename drpolicy.yaml after replacing <cluster1> and <cluster2> with the correct names of your managed clusters in ACM. Replace <string_value_1> and <string_value_2> with any values as long as they are unique (for example: east and west). The schedulingInterval should be one of the values configured in the MirrorPeer earlier (for example: 5m).

   apiVersion: ramendr.openshift.io/v1alpha1
   kind: DRPolicy
   metadata:
     name: odr-policy-5m
   spec:
     drClusterSet:
     - name: <cluster1>
       region: <string_value_1>
       s3ProfileName: s3profile-<cluster1>-ocs-storagecluster
     - name: <cluster2>
       region: <string_value_2>
       s3ProfileName: s3profile-<cluster2>-ocs-storagecluster
     schedulingInterval: 5m

   Note

   There is no need to specify a namespace to create this resource because DRPolicy is a cluster-scoped resource.

4. Copy the contents of your unique drpolicy.yaml file into the YAML view. You must completely replace the original content.

5. Click Create on the YAML view screen.

   Important

   The DRPolicy schedulingInterval must match one of the values configured in MirroPeer resource (e.g. 5m). To use one of the other schedulingIntervals for volume replication configured in the MirrorPeer requires creating additional DRPolicy resources with the new values (i.e., 15m). Make sure to change the DRPolicy name to be unique and useful in identifying the replication interval (e.g. odr-policy-15m).

6. Verify that the DRPolicy is created successfully by running the command on the Hub cluster for each DRPolicy resource created. This example is for odr-policy-5m:

   $ oc get drpolicy odr-policy-5m -n openshift-dr-system -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

Procedure

1. Edit the ConfigMag ramen-hub-operator-config on the Hub cluster to add deploymentAutomationEnabled=true as follows:

   $ oc edit configmap ramen-hub-operator-config -n openshift-dr-system

   apiVersion: v1
   data:
     ramen_manager_config.yaml: |
       apiVersion: ramendr.openshift.io/v1alpha1
       drClusterOperator:
         deploymentAutomationEnabled: true  ## <-- Add to enable installation of ODR Cluster operator on managed clusters
         catalogSourceName: redhat-operators
         catalogSourceNamespaceName: openshift-marketplace
         channelName: stable-4.10
         clusterServiceVersionName: odr-cluster-operator.v4.10.0
         namespaceName: openshift-dr-system
         packageName: odr-cluster-operator
   [...]

2. Verify that the installation was successful in the Primary managed cluster and the Secondary managed cluster do the following command:

   $ oc get csv,pod -n openshift-dr-system

   Example output:

   NAME                                                                      DISPLAY                         VERSION   REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.10.0   Openshift DR Cluster Operator   4.10.0               Succeeded
   
   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/ramen-dr-cluster-operator-5564f9d669-f6lbc   2/2     Running   0          5m32s

   You can also go to OperatorHub on each of the managed clusters and verify if the OpenShift DR Cluster Operator is installed.

Procedure

1. Edit the ConfigMag ramen-hub-operator-config on the Hub cluster to add s3SecretDistributionEnabled=true as follows:

   $ oc edit configmap ramen-hub-operator-config -n openshift-dr-system

   apiVersion: v1
   data:
     ramen_manager_config.yaml: |
       apiVersion: ramendr.openshift.io/v1alpha1
       drClusterOperator:
         deploymentAutomationEnabled: true
         s3SecretDistributionEnabled: true  ## <-- Add to enable automatic transfer of s3secrets
         catalogSourceName: redhat-operators
         catalogSourceNamespaceName: openshift-marketplace
         channelName: stable-4.10
         clusterServiceVersionName: odr-cluster-operator.v4.10.0
         namespaceName: openshift-dr-system
         packageName: odr-cluster-operator
   [...]

2. Verify that transfer of secrets was successful by running this command in both managed clusters.

   $ oc get secrets -n openshift-dr-system | grep Opaque

   Example output:

   8b3fb9ed90f66808d988c7edfa76eba35647092   Opaque        2      11m
   af5f82f21f8f77faf3de2553e223b535002e480   Opaque        2      11m

Procedure

1. Create a namespace or project on the Hub cluster for a busybox sample application.

   $ oc new-project busybox-sample

   Note

   A different project name other than busybox-sample can be used if desired. Make sure when deploying the sample application via the Advanced Cluster Manager console to use the same project name as what is created in this step.

2. Create DRPlacementControl resource

   DRPlacementControl is an API available after the OpenShift DR Hub Operator is installed on the Hub cluster. It is broadly an Advanced Cluster Manager PlacementRule reconciler that orchestrates placement decisions based on data availability across clusters that are part of a DRPolicy.

   1. On the Hub cluster, navigate to Installed Operators in the busybox-sample project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
   2. Create an instance for DRPlacementControl and then go to the YAML view. Make sure the busybox-sample project is selected.

   3. Save the following YAML to filename busybox-drpc.yaml after replacing <cluster1> with the correct name of your managed cluster in Advanced Cluster Manager. Modify drPolicyRef name for the DRPolicy that has the desired replication interval.

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRPlacementControl
      metadata:
        labels:
          app: busybox-sample
        name: busybox-drpc
      spec:
        drPolicyRef:
          name: odr-policy-5m     ## <-- Modify to specify desired DRPolicy and RPO
        placementRef:
          kind: PlacementRule
          name: busybox-placement
        preferredCluster: <cluster1>
        pvcSelector:
          matchLabels:
            appname: busybox

   4. Copy the contents of your unique busybox-drpc.yaml file into the YAML view (completely replacing original content).

   5. Click Create on the YAML view screen.

      You can also create this resource using the following CLI command:

      $ oc create -f busybox-drpc.yaml -n busybox-sample

      Example output:

      drplacementcontrol.ramendr.openshift.io/busybox-drpc created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

3. Create Placement Rule resource that defines the target clusters where resource templates can be deployed. Use placement rules to facilitate the multicluster deployment of your applications.

   1. Copy and save the following YAML to filename busybox-placementrule.yaml.

      apiVersion: apps.open-cluster-management.io/v1
      kind: PlacementRule
      metadata:
        labels:
          app: busybox-sample
        name: busybox-placement
      spec:
        clusterConditions:
        - status: "True"
          type: ManagedClusterConditionAvailable
        clusterReplicas: 1
        schedulerName: ramen

   2. Create the Placement Rule resource for the busybox-sample application.

      $ oc create -f busybox-placementrule.yaml -n busybox-sample

      Example output:

      placementrule.apps.open-cluster-management.io/busybox-placement created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

4. Create sample application using RHACM console

   1. Log in to the RHACM console using your OpenShift credentials if not already logged in.

      $ oc get route multicloud-console -n open-cluster-management -o jsonpath --template="https://{.spec.host}/multicloud/applications{'\n'}"

      Example Output:

      https://multicloud-console.apps.perf3.example.com/multicloud/applications

   2. Navigate to Applications and click Create application.
   3. Select type as Subscription.
   4. Enter your application Name (for example, busybox) and Namespace (for example, busybox-sample).
   5. In Repository location for resources section, select Repository type Git.

   6. Enter the Git repository URL for the sample application, the github Branch and Path where the resources busybox Pod and PVC will be created.

      Use the sample application repository as https://github.com/RamenDR/ocm-ramen-samples where the Branch is main and Path is busybox-odr.

      Important

      Make sure that the new StorageClass ocs-storagecluster-ceph-rbdmirror is created as detailed in section Create Mirroring StorageClass resource before proceeding.

      Verify that it is created using the following command:

      oc get storageclass | grep rbdmirror | awk '{print $1}'

      Example output:

      ocs-storagecluster-ceph-rbdmirror

   7. Scroll down the form to the section Select clusters to deploy to and click Select an existing placement configuration.
   8. Select an Existing Placement Rule (for example, busybox-placement) from the drop-down list.

   9. Click Save.

      On the follow-on screen scroll to the bottom. You should see that there are all Green checkmarks on the application topology.

      Note

      To get more information, click on any of the topology elements and a window will appear on the right of the topology view.

5. Verify the sample application deployment and replication.

   Now that the busybox application has been deployed to your preferred Cluster (specified in the DRPlacementControl) the deployment can be validated.

   1. Login to your managed cluster where busybox was deployed by RHACM.

      $ oc get pods,pvc -n busybox-sample

      Example output:

      NAME          READY   STATUS    RESTARTS   AGE
      pod/busybox   1/1     Running   0          6m
      
      NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
      persistentvolumeclaim/busybox-pvc   Bound    pvc-a56c138a-a1a9-4465-927f-af02afbbff37   1Gi        RWO            ocs-storagecluster-ceph-rbd   6m

   2. Verify that the replication resources are also created for the busybox PVC.

      $ oc get volumereplication,volumereplicationgroup -n busybox-sample

      Example output:

      NAME                                                             AGE   VOLUMEREPLICATIONCLASS           PVCNAME       DESIREDSTATE   CURRENTSTATE
      volumereplication.replication.storage.openshift.io/busybox-pvc   6m   odf-rbd-volumereplicationclass   busybox-pvc   primary        Primary
      
      NAME                                                       AGE
      volumereplicationgroup.ramendr.openshift.io/busybox-drpc   6m

   3. Verify that the busybox volume has been replicated to the alternate cluster by running the following command on both the Primary managed cluster and the Secondary managed cluster.

      $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

      Example output:

      {"daemon_health":"OK","health":"OK","image_health":"OK","states":{"replaying":2}}

      Note

      Both managed clusters should have the exact same output with a new status of "states":{"replaying":2}`.

Procedure

1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
2. Click DRPlacementControl tab.
3. Click DRPC busybox-drpc and then the YAML view.

4. Add the action and failoverCluster details as shown in below screenshot. The failoverCluster should be the ACM cluster name for the Secondary managed cluster.

   DRPlacementControl add action Failover

   

5. Click Save.

6. Verify that the application busybox is now running in the Secondary managed cluster, the failover cluster ocp4perf2 specified in the YAML file.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   NAME          READY   STATUS    RESTARTS   AGE
   pod/busybox   1/1     Running   0          35s
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   35s

7. Verify that busybox is no longer running on the Primary managed cluster.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   No resources found in busybox-sample namespace.

Procedure

1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
2. Click DRPlacementControl tab.
3. Click DRPC busybox-drpc and then the YAML view.

4. Modify action to Relocate

   DRPlacementControl modify action to Relocate

   

5. Click Save.

6. Verify if the application busybox is now running in the Primary managed cluster.The failback is to the preferredCluster ocp4perf1 as specified in the YAML file, which is where the application was running before the failover operation.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   NAME          READY   STATUS    RESTARTS   AGE
   pod/busybox   1/1     Running   0          60s
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   61s

7. Verify if busybox is running in the Secondary managed cluster. The busybox application should no longer be running on this managed cluster.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   No resources found in busybox-sample namespace.