Chapter 3. Metro-DR solution for OpenShift Data Foundation

1. Register all the nodes to the Red Hat Network or Red Hat Satellite and subscribe to a valid pool:

   subscription-manager register
   subscription-manager subscribe --pool=8a8XXXXXX9e0

2. Enable access for all the nodes in the Ceph cluster for the following repositories:

   • rhel9-for-x86_64-baseos-rpms

   • rhel9-for-x86_64-appstream-rpms

     subscription-manager repos --disable="*" --enable="rhel9-for-x86_64-baseos-rpms" --enable="rhel9-for-x86_64-appstream-rpms"

3. Update the operating system RPMs to the latest version and reboot if needed:

   dnf update -y
   reboot

4. Select a node from the cluster to be your bootstrap node. ceph1 is our bootstrap node in this example going forward.

   Only on the bootstrap node ceph1, enable the ansible-2.9-for-rhel-9-x86_64-rpms and rhceph-6-tools-for-rhel-9-x86_64-rpms repositories:

   subscription-manager repos --enable="ansible-2.9-for-rhel-9-x86_64-rpms" --enable="rhceph-6-tools-for-rhel-9-x86_64-rpms"

5. Configure the hostname using the bare/short hostname in all the hosts.

   hostnamectl set-hostname <short_name>

6. Verify the hostname configuration for deploying Red Hat Ceph Storage with cephadm.

   $ hostname

   Example output:

   ceph1

7. Modify /etc/hosts file and add the fqdn entry to the 127.0.0.1 IP by setting the DOMAIN variable with our DNS domain name.

   DOMAIN="example.domain.com"
   
   cat <<EOF >/etc/hosts
   127.0.0.1 $(hostname).${DOMAIN} $(hostname) localhost localhost.localdomain localhost4 localhost4.localdomain4
   ::1       $(hostname).${DOMAIN} $(hostname) localhost6 localhost6.localdomain6
   EOF

8. Check the long hostname with the fqdn using the hostname -f option.

   $ hostname -f

   Example output:

   ceph1.example.domain.com

   Note

   To know more about why these changes are required, see Fully Qualified Domain Names vs Bare Host Names.

9. Run the following steps on the bootstrap node. In our example, the bootstrap node is ceph1.

   1. Install the cephadm-ansible RPM package:

      $ sudo dnf install -y cephadm-ansible

      Important

      To run the ansible playbooks, you must have ssh passwordless access to all the nodes that are configured to the Red Hat Ceph Storage cluster. Ensure that the configured user (for example, deployment-user) has root privileges to invoke the sudo command without needing a password.

   2. To use a custom key, configure the selected user (for example, deployment-user) ssh config file to specify the id/key that will be used for connecting to the nodes via ssh:

      cat <<EOF > ~/.ssh/config
      Host ceph*
         User deployment-user
         IdentityFile ~/.ssh/ceph.pem
      EOF

   3. Build the ansible inventory

      cat <<EOF > /usr/share/cephadm-ansible/inventory
      ceph1
      ceph2
      ceph3
      ceph4
      ceph5
      ceph6
      ceph7
      [admin]
      ceph1
      ceph4
      EOF

      Note

      Here, the Hosts (Ceph1 and Ceph4) belonging to two different data centers are configured as part of the [admin] group on the inventory file and are tagged as _admin by cephadm. Each of these admin nodes receive the admin ceph keyring during the bootstrap process so that when one data center is down, we can check using the other available admin node.

   4. Verify that ansible can access all nodes using the ping module before running the pre-flight playbook.

      $ ansible -i /usr/share/cephadm-ansible/inventory -m ping all -b

      Example output:

      ceph6 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph4 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph3 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph2 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph5 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph1 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }
      ceph7 | SUCCESS => {
          "ansible_facts": {
              "discovered_interpreter_python": "/usr/libexec/platform-python"
          },
          "changed": false,
          "ping": "pong"
      }

   5. Navigate to the /usr/share/cephadm-ansible directory.

   6. Run ansible-playbook with relative file paths.

      $ ansible-playbook -i /usr/share/cephadm-ansible/inventory /usr/share/cephadm-ansible/cephadm-preflight.yml --extra-vars "ceph_origin=rhcs"

      The preflight playbook Ansible playbook configures the RHCS dnf repository and prepares the storage cluster for bootstrapping. It also installs podman, lvm2, chronyd, and cephadm. The default location for cephadm-ansible and cephadm-preflight.yml is /usr/share/cephadm-ansible. For additional information, see Running the preflight playbook

1. Bootstrap
2. Service deployment

Procedure

1. Create json file to authenticate against the container registry using a json file as follows:

   $ cat <<EOF > /root/registry.json
   {
    "url":"registry.redhat.io",
    "username":"User",
    "password":"Pass"
   }
   EOF

2. Create a cluster-spec.yaml that adds the nodes to the Red Hat Ceph Storage cluster and also sets specific labels for where the services should run following table 3.1.

   cat <<EOF > /root/cluster-spec.yaml
   service_type: host
   addr: 10.0.40.78  ## <XXX.XXX.XXX.XXX>
   hostname: ceph1   ##  <ceph-hostname-1>
   location:
     root: default
     datacenter: DC1
   labels:
     - osd
     - mon
     - mgr
   ---
   service_type: host
   addr: 10.0.40.35
   hostname: ceph2
   location:
     datacenter: DC1
   labels:
     - osd
     - mon
   ---
   service_type: host
   addr: 10.0.40.24
   hostname: ceph3
   location:
     datacenter: DC1
   labels:
     - osd
     - mds
     - rgw
   ---
   service_type: host
   addr: 10.0.40.185
   hostname: ceph4
   location:
     root: default
     datacenter: DC2
   labels:
     - osd
     - mon
     - mgr
   ---
   service_type: host
   addr: 10.0.40.88
   hostname: ceph5
   location:
     datacenter: DC2
   labels:
     - osd
     - mon
   ---
   service_type: host
   addr: 10.0.40.66
   hostname: ceph6
   location:
     datacenter: DC2
   labels:
     - osd
     - mds
     - rgw
   ---
   service_type: host
   addr: 10.0.40.221
   hostname: ceph7
   labels:
     - mon
   ---
   service_type: mon
   placement:
     label: "mon"
   ---
   service_type: mds
   service_id: cephfs
   placement:
     label: "mds"
   ---
   service_type: mgr
   service_name: mgr
   placement:
     label: "mgr"
   ---
   service_type: osd
   service_id: all-available-devices
   service_name: osd.all-available-devices
   placement:
     label: "osd"
   spec:
     data_devices:
       all: true
   ---
   service_type: rgw
   service_id: objectgw
   service_name: rgw.objectgw
   placement:
     count: 2
     label: "rgw"
   spec:
     rgw_frontend_port: 8080
   EOF

3. Retrieve the IP for the NIC with the Red Hat Ceph Storage public network configured from the bootstrap node. After substituting 10.0.40.0 with the subnet that you have defined in your ceph public network, execute the following command.

   $ ip a | grep 10.0.40

   Example output:

   10.0.40.78

4. Run the cephadm bootstrap command as the root user on the node that will be the initial Monitor node in the cluster. The IP_ADDRESS option is the node’s IP address that you are using to run the cephadm bootstrap command.

   Note

   If you have configured a different user instead of root for passwordless SSH access, then use the --ssh-user= flag with the cepadm bootstrap command.

   If you are using non default/id_rsa ssh key names, then use --ssh-private-key and --ssh-public-key options with cephadm command.

   $ cephadm  bootstrap --ssh-user=deployment-user --mon-ip 10.0.40.78 --apply-spec /root/cluster-spec.yaml --registry-json /root/registry.json

   Important

   If the local node uses fully-qualified domain names (FQDN), then add the --allow-fqdn-hostname option to cephadm bootstrap on the command line.

   Once the bootstrap finishes, you will see the following output from the previous cephadm bootstrap command:

   You can access the Ceph CLI with:
   
   	sudo /usr/sbin/cephadm shell --fsid dd77f050-9afe-11ec-a56c-029f8148ea14 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
   
   Consider enabling telemetry to help improve Ceph:
   
   	ceph telemetry on
   
   For more information see:
   
   	https://docs.ceph.com/docs/pacific/mgr/telemetry/

5. Verify the status of Red Hat Ceph Storage cluster deployment using the Ceph CLI client from ceph1:

   $ ceph -s

   Example output:

   cluster:
     id:     3a801754-e01f-11ec-b7ab-005056838602
     health: HEALTH_OK
   
   services:
     mon: 5 daemons, quorum ceph1,ceph2,ceph4,ceph5,ceph7 (age 4m)
     mgr: ceph1.khuuot(active, since 5m), standbys: ceph4.zotfsp
     osd: 12 osds: 12 up (since 3m), 12 in (since 4m)
     rgw: 2 daemons active (2 hosts, 1 zones)
   
   data:
     pools:   5 pools, 107 pgs
     objects: 191 objects, 5.3 KiB
     usage:   105 MiB used, 600 GiB / 600 GiB avail
              105 active+clean

   Note

   It may take several minutes for all the services to start.

   It is normal to get a global recovery event while you do not have any OSDs configured.

   You can use ceph orch ps and ceph orch ls to further check the status of the services.

6. Verify if all the nodes are part of the cephadm cluster.

   $ ceph orch host ls

   Example output:

   HOST   ADDR          LABELS  STATUS
   ceph1  10.0.40.78    _admin osd mon mgr
   ceph2  10.0.40.35    osd mon
   ceph3  10.0.40.24    osd mds rgw
   ceph4  10.0.40.185   osd mon mgr
   ceph5  10.0.40.88    osd mon
   ceph6  10.0.40.66    osd mds rgw
   ceph7  10.0.40.221   mon

   Note

   You can run Ceph commands directly from the host because ceph1 was configured in the cephadm-ansible inventory as part of the [admin] group. The Ceph admin keys were copied to the host during the cephadm bootstrap process.

7. Check the current placement of the Ceph monitor services on the datacenters.

   $ ceph orch ps | grep mon | awk '{print $1 " " $2}'

   Example output:

   mon.ceph1 ceph1
   mon.ceph2 ceph2
   mon.ceph4 ceph4
   mon.ceph5 ceph5
   mon.ceph7 ceph7

8. Check the current placement of the Ceph manager services on the datacenters.

   $ ceph orch ps | grep mgr | awk '{print $1 " " $2}'

   Example output:

   mgr.ceph2.ycgwyz ceph2
   mgr.ceph5.kremtt ceph5

9. Check the ceph osd crush map layout to ensure that each host has one OSD configured and its status is UP. Also, double-check that each node is under the right datacenter bucket as specified in table 3.1

   $ ceph osd tree

   Example output:

   ID   CLASS  WEIGHT   TYPE NAME           STATUS  REWEIGHT  PRI-AFF
   -1          0.87900  root default
   -16         0.43950      datacenter DC1
   -11         0.14650          host ceph1
     2    ssd  0.14650              osd.2       up   1.00000  1.00000
    -3         0.14650          host ceph2
     3    ssd  0.14650              osd.3       up   1.00000  1.00000
   -13         0.14650          host ceph3
     4    ssd  0.14650              osd.4       up   1.00000  1.00000
   -17         0.43950      datacenter DC2
    -5         0.14650          host ceph4
     0    ssd  0.14650              osd.0       up   1.00000  1.00000
    -9         0.14650          host ceph5
     1    ssd  0.14650              osd.1       up   1.00000  1.00000
    -7         0.14650          host ceph6
     5    ssd  0.14650              osd.5       up   1.00000  1.00000

10. Create and enable a new RDB block pool.

    $ ceph osd pool create 32 32
    $ ceph osd pool application enable rbdpool rbd

    Note

    The number 32 at the end of the command is the number of PGs assigned to this pool. The number of PGs can vary depending on several factors like the number of OSDs in the cluster, expected % used of the pool, etc. You can use the following calculator to determine the number of PGs needed: Ceph Placement Groups (PGs) per Pool Calculator.

11. Verify that the RBD pool has been created.

    $ ceph osd lspools | grep rbdpool

    Example output:

     3 rbdpool

12. Verify that MDS services are active and have located one service on each datacenter.

    $ ceph orch ps | grep mds

    Example output:

    mds.cephfs.ceph3.cjpbqo    ceph3               running (17m)   117s ago  17m    16.1M        -  16.2.9
    mds.cephfs.ceph6.lqmgqt    ceph6               running (17m)   117s ago  17m    16.1M        -  16.2.9

13. Create the CephFS volume.

    $ ceph fs volume create cephfs

    Note

    The ceph fs volume create command also creates the needed data and meta CephFS pools. For more information, see Configuring and Mounting Ceph File Systems.

14. Check the Ceph status to verify how the MDS daemons have been deployed. Ensure that the state is active where ceph6 is the primary MDS for this filesystem and ceph3 is the secondary MDS.

    $ ceph fs status

    Example output:

    cephfs - 0 clients
    ======
    RANK  STATE           MDS             ACTIVITY     DNS    INOS   DIRS   CAPS
     0    active  cephfs.ceph6.ggjywj  Reqs:    0 /s    10     13     12      0
           POOL           TYPE     USED  AVAIL
    cephfs.cephfs.meta  metadata  96.0k   284G
    cephfs.cephfs.data    data       0    284G
        STANDBY MDS
    cephfs.ceph3.ogcqkl

15. Verify that RGW services are active.

    $ ceph orch ps | grep rgw

    Example output:

    rgw.objectgw.ceph3.kkmxgb  ceph3  *:8080       running (7m)      3m ago   7m    52.7M        -  16.2.9
    rgw.objectgw.ceph6.xmnpah  ceph6  *:8080       running (7m)      3m ago   7m    53.3M        -  16.2.9

Procedure

1. Check the current election strategy being used by the monitors with the ceph mon dump command. By default in a ceph cluster, the connectivity is set to classic.

   ceph mon dump | grep election_strategy

   Example output:

   dumped monmap epoch 9
   election_strategy: 1

2. Change the monitor election to connectivity.

   ceph mon set election_strategy connectivity

3. Run the previous ceph mon dump command again to verify the election_strategy value.

   $ ceph mon dump | grep election_strategy

   Example output:

   dumped monmap epoch 10
   election_strategy: 3

   To know more about the different election strategies, see Configuring monitor election strategy.

4. Set the location for all our Ceph monitors:

   ceph mon set_location ceph1 datacenter=DC1
   ceph mon set_location ceph2 datacenter=DC1
   ceph mon set_location ceph4 datacenter=DC2
   ceph mon set_location ceph5 datacenter=DC2
   ceph mon set_location ceph7 datacenter=DC3

5. Verify that each monitor has its appropriate location.

   $ ceph mon dump

   Example output:

   epoch 17
   fsid dd77f050-9afe-11ec-a56c-029f8148ea14
   last_changed 2022-03-04T07:17:26.913330+0000
   created 2022-03-03T14:33:22.957190+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   0: [v2:10.0.143.78:3300/0,v1:10.0.143.78:6789/0] mon.ceph1; crush_location {datacenter=DC1}
   1: [v2:10.0.155.185:3300/0,v1:10.0.155.185:6789/0] mon.ceph4; crush_location {datacenter=DC2}
   2: [v2:10.0.139.88:3300/0,v1:10.0.139.88:6789/0] mon.ceph5; crush_location {datacenter=DC2}
   3: [v2:10.0.150.221:3300/0,v1:10.0.150.221:6789/0] mon.ceph7; crush_location {datacenter=DC3}
   4: [v2:10.0.155.35:3300/0,v1:10.0.155.35:6789/0] mon.ceph2; crush_location {datacenter=DC1}

6. Create a CRUSH rule that makes use of this OSD crush topology by installing the ceph-base RPM package in order to use the crushtool command:

   $ dnf -y install ceph-base

   To know more about CRUSH ruleset, see Ceph CRUSH ruleset.

7. Get the compiled CRUSH map from the cluster:

   $ ceph osd getcrushmap > /etc/ceph/crushmap.bin

8. Decompile the CRUSH map and convert it to a text file in order to be able to edit it:

   $ crushtool -d /etc/ceph/crushmap.bin -o /etc/ceph/crushmap.txt

9. Add the following rule to the CRUSH map by editing the text file /etc/ceph/crushmap.txt at the end of the file.

   $ vim /etc/ceph/crushmap.txt

   rule stretch_rule {
           id 1
           type replicated
           min_size 1
           max_size 10
           step take default
           step choose firstn 0 type datacenter
           step chooseleaf firstn 2 type host
           step emit
   }
   # end crush map

   This example is applicable for active applications in both OpenShift Container Platform clusters.

   Note

   The rule id has to be unique. In the example, we only have one more crush rule with id 0 hence we are using id 1. If your deployment has more rules created, then use the next free id.

   The CRUSH rule declared contains the following information:

   • Rule name

     • Description: A unique whole name for identifying the rule.
     • Value: stretch_rule

   • id

     • Description: A unique whole number for identifying the rule.
     • Value: 1

   • type

     • Description: Describes a rule for either a storage drive replicated or erasure-coded.
     • Value: replicated

   • min_size

     • Description: If a pool makes fewer replicas than this number, CRUSH will not select this rule.
     • Value: 1

   • max_size

     • Description: If a pool makes more replicas than this number, CRUSH will not select this rule.
     • Value: 10

   • step take default

     • Description: Takes the root bucket called default, and begins iterating down the tree.

   • step choose firstn 0 type datacenter

     • Description: Selects the datacenter bucket, and goes into its subtrees.

   • step chooseleaf firstn 2 type host

     • Description: Selects the number of buckets of the given type. In this case, it is two different hosts located in the datacenter it entered at the previous level.

   • step emit

     • Description: Outputs the current value and empties the stack. Typically used at the end of a rule, but may also be used to pick from different trees in the same rule.

10. Compile the new CRUSH map from the file /etc/ceph/crushmap.txt and convert it to a binary file called /etc/ceph/crushmap2.bin:

    $ crushtool -c /etc/ceph/crushmap.txt -o /etc/ceph/crushmap2.bin

11. Inject the new crushmap we created back into the cluster:

    $ ceph osd setcrushmap -i /etc/ceph/crushmap2.bin

    Example output:

    17

    Note

    The number 17 is a counter and it will increase (18,19, and so on) depending on the changes you make to the crush map.

12. Verify that the stretched rule created is now available for use.

    ceph osd crush rule ls

    Example output:

    replicated_rule
    stretch_rule

13. Enable the stretch cluster mode.

    $ ceph mon enable_stretch_mode ceph7 stretch_rule datacenter

    In this example, ceph7 is the arbiter node, stretch_rule is the crush rule we created in the previous step and datacenter is the dividing bucket.

14. Verify all our pools are using the stretch_rule CRUSH rule we have created in our Ceph cluster:

    $ for pool in $(rados lspools);do echo -n "Pool: ${pool}; ";ceph osd pool get ${pool} crush_rule;done

    Example output:

    Pool: device_health_metrics; crush_rule: stretch_rule
    Pool: cephfs.cephfs.meta; crush_rule: stretch_rule
    Pool: cephfs.cephfs.data; crush_rule: stretch_rule
    Pool: .rgw.root; crush_rule: stretch_rule
    Pool: default.rgw.log; crush_rule: stretch_rule
    Pool: default.rgw.control; crush_rule: stretch_rule
    Pool: default.rgw.meta; crush_rule: stretch_rule
    Pool: rbdpool; crush_rule: stretch_rule

    This indicates that a working Red Hat Ceph Storage stretched cluster with arbiter mode is now available.

1. Find the IP addresses for all of the OpenShift nodes in the managed clusters by running this command in the Primary managed cluster and the Secondary managed cluster.

   $ oc get nodes -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="ExternalIP")].address}{"\n"}{end}'

   Example output:

   10.70.56.118
   10.70.56.193
   10.70.56.154
   10.70.56.242
   10.70.56.136
   10.70.56.99

   Once you have the IP addresses then the DRCluster resources can be modified for each managed cluster.

2. Find the DRCluster names on the Hub Cluster.

   $ oc get drcluster

   Example output:

   NAME        AGE
   ocp4perf1   5m35s
   ocp4perf2   5m35s

3. Edit each DRCluster to add your unique IP addresses after replacing <drcluster_name> with your unique name.

   $ oc edit drcluster <drcluster_name>

   apiVersion: ramendr.openshift.io/v1alpha1
   kind: DRCluster
   metadata:
   [...]
   spec:
     s3ProfileName: s3profile-<drcluster_name>-ocs-external-storagecluster
     ## Add this section
     cidrs:
       -  <IP_Address1>/32
       -  <IP_Address2>/32
       -  <IP_Address3>/32
       -  <IP_Address4>/32
       -  <IP_Address5>/32
       -  <IP_Address6>/32
   [...]

   Example output:

   drcluster.ramendr.openshift.io/ocp4perf1 edited

Procedure

1. On the RHACM console, navigate to Applications.
2. Search for the sample application to be deleted (for example, busybox).
3. Click the Action Menu (⋮) next to the application you want to delete.

4. Click Delete application.

   When the Delete application is selected a new screen will appear asking if the application related resources should also be deleted.

5. Select Remove application related resources checkbox to delete the Subscription and PlacementRule.
6. Click Delete. This will delete the busybox application on the Primary managed cluster (or whatever cluster the application was running on).

7. In addition to the resources deleted using the RHACM console, delete the DRPlacementControl if it is not auto-deleted after deleting the busybox application.

   1. Log in to the OpenShift Web console for the Hub cluster and navigate to Installed Operators for the project busybox-sample.

      For ApplicationSet applications, select the project as openshift-gitops.

   2. Click OpenShift DR Hub Operator and then click the DRPlacementControl tab.
   3. Click the Action Menu (⋮) next to the busybox application DRPlacementControl that you want to delete.
   4. Click Delete DRPlacementControl.
   5. Click Delete.

1. Ensure that all the DR configurations have been installed on the Primary managed cluster and the Secondary managed cluster.

2. Install the OADP 1.4 operator.

   Note

   Any version before OADP 1.4 will not work for protecting discovered applications.

   1. On the Primary and Secondary managed cluster, navigate to OperatorHub and use the keyword filter to search for OADP.
   2. Click the OADP tile.
   3. Keep all default settings and click Install. Ensure that the operator resources are installed in the openshift-adp project.
   Note

   If OADP 1.4 is installed after DR configuration has been completed then the ramen-dr-cluster-operator pods on the Primary managed cluster and the Secondary managed cluster in namespace openshift-dr-system must be restarted (deleted and recreated).

3. [Optional] Add CACertificates to ramen-hub-operator-config ConfigMap.

   Configure network (SSL) access between the primary and secondary clusters so that metadata can be stored on the alternate cluster in a Multicloud Gateway (MCG) object bucket using a secure transport protocol and in the Hub cluster for verifying access to the object buckets.

   Note

   If all of your OpenShift clusters are deployed using a signed and valid set of certificates for your environment then this section can be skipped.

   If you are using self-signed certificates, then you have already created a ConfigMap named user-ca-bundle in the openshift-config namespace and added this ConfigMap to the default Proxy cluster resource.

   1. Find the encoded value for the CACertificates.

      $ oc get configmap user-ca-bundle -n openshift-config -o jsonpath="{['data']['ca-bundle\.crt']}" |base64 -w 0

   2. Add this base64 encoded value to the configmap ramen-hub-operator-config on the Hub cluster. Example below shows where to add CACertificates.

      $ oc edit configmap ramen-hub-operator-config -n openshift-operators

      [...]
          ramenOpsNamespace: openshift-dr-ops
          s3StoreProfiles:
          - s3Bucket: odrbucket-36bceb61c09c
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.hyper3.vmw.ibmfusion.eu
            s3ProfileName: s3profile-hyper3-ocs-storagecluster
            s3Region: noobaa
            s3SecretRef:
              name: 60f2ea6069e168346d5ad0e0b5faa59bb74946f
            caCertificates: {input base64 encoded value here}
          - s3Bucket: odrbucket-36bceb61c09c
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.hyper4.vmw.ibmfusion.eu
            s3ProfileName: s3profile-hyper4-ocs-storagecluster
            s3Region: noobaa
            s3SecretRef:
              name: cc237eba032ad5c422fb939684eb633822d7900
            caCertificates: {input base64 encoded value here}

4. Verify that there are DR secrets created in the OADP operator default namespace openshift-adp on the Primary managed cluster and the Secondary managed cluster. The DR secrets that were created when the first DRPolicy was created, will be similar to the secrets below. The DR secret name is preceded with the letter v.

   $ oc get secrets -n openshift-adp
   NAME                                       TYPE     DATA   AGE
   v60f2ea6069e168346d5ad0e0b5faa59bb74946f   Opaque   1      3d20h
   vcc237eba032ad5c422fb939684eb633822d7900   Opaque   1      3d20h
   [...]

   Note

   There will be one DR created secret for each managed cluster in the openshift-adp namespace.

5. Verify if the Data Protection Application (DPA) is already installed on each managed cluster in the OADP namespace openshift-adp. If not already created then follow the next step to create this resource.

   1. Create the DPA by copying the following YAML definition content to dpa.yaml.

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        labels:
          app.kubernetes.io/component: velero
        name: velero
        namespace: openshift-adp
      spec:
        backupImages: false
        configuration:
          nodeAgent:
            enable: false
            uploaderType: restic
          velero:
            defaultPlugins:
              - openshift
              - aws
            noDefaultBackupLocation: true

   2. Create the DPA resource.

      $ oc create -f dpa.yaml -n openshift-adp

      dataprotectionapplication.oadp.openshift.io/velero created

   3. Verify that the OADP resources are created and are in Running state.

      $ oc get pods,dpa -n openshift-adp
      NAME                                                    READY   STATUS    RESTARTS   AGE
      pod/openshift-adp-controller-manager-7b64b74fcd-msjbs   1/1     Running   0          5m30s
      pod/velero-694b5b8f5c-b4kwg                             1/1     Running   0          3m31s
      
      
      NAME                                                 AGE
      dataprotectionapplication.oadp.openshift.io/velero   3m31s

Procedure

1. Log in to the Primary managed cluster and clone the sample application repository.

   $ git clone https://github.com/red-hat-storage/ocm-ramen-samples.git

2. Verify that you are on the main branch.

   $ cd ~/ocm-ramen-samples
   $ git branch
   * main

   The correct directory should be used when creating the sample application based on your scenario, metro or regional.

   Note

   Only applications using CephRBD or block volumes are supported for discovered applications.

   $ ls workloads/deployment | egrep -v 'cephfs|k8s|base'
   odr-metro-rbd
   odr-regional-rbd

3. Create a project named busybox-discovered on both the Primary and Secondary managed clusters.

   $ oc new-project busybox-discovered

4. Create the busybox application on the Primary managed cluster. This sample application example is for Metro-DR using a block (Ceph RBD) volume.

   $ oc apply -k workloads/deployment/odr-metro-rbd -n busybox-discovered
   persistentvolumeclaim/busybox-pvc created
   deployment.apps/busybox created

   Note

   OpenShift Data Foundation Disaster Recovery solution now extends protection to discovered applications that span across multiple namespaces.

5. Verify that busybox is running in the correct project on the Primary managed cluster.

   $ oc get pods,pvc,deployment -n busybox-discovered

   NAME                           READY   STATUS    RESTARTS   AGE
   pod/busybox-796fccbb95-qmxjf   1/1     Running   0          18s
   
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  VOLUMEATTRIBUTESCLASS   AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-b20e4129-902d-47c7-b962-040ad64130c4   1Gi        RWO            ocs-storagecluster-ceph-rbd   <unset>                 18s
   
   
   NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/busybox   1/1     1            1           18

Procedure

1. On RHACM console, navigate to Disaster recovery Protected applications tab.
2. Click Enroll application to start configuring existing applications for DR protection.
3. Select ACM discovered applications.
4. In the Namespace page, choose the DR cluster which is the name of the Primary managed cluster where busybox is installed.

5. Select namespace where the application is installed. For example, busybox-discovered.

   Note

   If you have workload spread across multiple namespaces then you can select all of those namespaces to DR protect.

6. Choose a unique Name, for example busybox-rbd, for the discovered application and click Next.
7. In the Configuration page, the Resource label is used to protect your resources where you can set which resources will be included in the kubernetes-object backup and what volume’s persistent data will be replicated. Resource label is selected by default.
8. Provide Label expressions and PVC label selector. Choose the label appname=busybox for both the kubernetes-objects and for the PVC(s).
9. Click Next.

10. In the Replication page, select an existing DR Policy and the kubernetes-objects backup interval.

    Note

    It is recommended to choose the same duration for the PVC data replication and kubernetes-object backup interval (i.e., 5 minutes).

11. Click Next.

12. Review the configuration and click Save.

    Use the Back button to go back to the screen to correct any issues.

13. Verify that the Application volumes (PVCs) and the Kubernetes-objects backup have a Healthy status before proceeding to DR Failover and Relocate testing. You can view the status of your Discovered applications on the Protected applications tab.

    1. To see the status of the DRPC, run the following command on the Hub cluster:

       $ oc get drpc {drpc_name} -o wide -n openshift-dr-ops

       The discovered applications store resources such as DRPlacementControl (DRPC) and Placement on the Hub cluster in a new namespace called openshift-dr-ops. The DRPC name can be identified by the unique Name configured in prior steps (i.e., busybox-rbd).

    2. To see the status of the VolumeReplicationGroup (VRG) for discovered applications, run the following command on the managed cluster where the busybox application was manually installed.

       $ oc get vrg {vrg_name} -n openshift-dr-ops

       The VRG resource is stored in the namespace openshift-dr-ops after a DR Policy is assigned to the discovered application. The VRG name can be identified by the unique Name configured in prior steps (i.e., busybox-rbd).

Procedure

1. Login to the Hub cluster.

2. List the DRPlacementControl (DRPC) resources. Each DRPC resource was created when the application was assigned a DR policy.

   $ oc get drpc -n openshift-dr-ops

3. Find the DRPC that has a name that includes the unique identifier that you chose when assigning a DR policy (for example, busybox-rbd) and delete the DRPC.

   $ oc delete {drpc_name} -n openshift-dr-ops

4. List the Placement resources. Each Placement resource was created when the application was assigned a DR policy.

   $ oc get placements -n openshift-dr-ops

5. Find the Placement that has a name that includes the unique identifier that you chose when assigning a DR policy (for example, busybox-rbd-placement-1) and delete the Placement.

   $ oc delete placements {placement_name} -n openshift-dr-ops

Procedure

1. Ensure that RHACM operator and MultiClusterHub is installed on the passive hub cluster. See RHACM installation guide for instructions.

   After the operator is successfully installed, a popover with a message that the Web console update is available appears on the user interface. Click Refresh web console from this popover for the console changes to reflect.

2. Before hub recovery, configure backup and restore. See Backup and restore topics of RHACM Business continuity guide.
3. Install the multicluster orchestrator (MCO) operator along with Red Hat OpenShift GitOps operator on the passive RHACM hub prior to the restore. For instructions to restore your RHACM hub, see Installing OpenShift Data Foundation Multicluster Orchestrator operator.
4. Ensure that .spec.cleanupBeforeRestore is set to None for the Restore.cluster.open-cluster-management.io resource. For details, see Restoring passive resources while checking for backups chapter of RHACM documentation.
5. If SSL access across clusters was configured manually during setup, then re-configure SSL access across clusters. For instructions, see Configuring SSL access across clusters chapter.

6. On the passive hub, add label to openshift-operators namespace to enable basic monitoring of VolumeSyncronizationDelay alert using this command. For alert details, see Disaster recovery alerts chapter.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

Procedure

1. Restore the backups on the passive hub cluster. For information, see Restoring a hub cluster from backup.

   Important

   Recovering a failed hub to its passive instance will only restore applications and their DR protected state to its last scheduled backup. Any application that was DR protected after the last scheduled backup would need to be protected again on the new hub.

2. During the restore procedure, to avoid eviction of resources when ManifestWorks are not regenerated correctly, you can enlarge the AppliedManifestWork eviction grace period.

   1. Verify that the restore is complete.

      $ oc -n <restore-namespace> wait restore <restore-name> --for=jsonpath='{.status.phase}'=Finished --timeout=120s

   2. After the restore is completed, on the hub cluster, check for existing global KlusterletConfig.

      • If global KlusterletConfig exists then edit and set the value for appliedManifestWorkEvictionGracePeriod parameter to a larger value. For example, 24 hours or more.

      • If global KlusterletConfig does not exist, then create the Klusterletconfig using the following yaml:

        apiVersion: config.open-cluster-management.io/v1alpha1
        kind: KlusterletConfig
        metadata:
          name: global
        spec:
          appliedManifestWorkEvictionGracePeriod: "24h"

        The configuration will be propagated to all the managed clusters automatically.

3. Verify that the Primary and Seconday managed clusters are successfully imported into the RHACM console and they are accessible. If any of the managed clusters are down or unreachable then they will not be successfully imported.
4. Wait until DRPolicy validation succeeds.

5. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with a unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

6. Refresh the RHACM console to make the DR monitoring dashboard tab accessible if it was enabled on the Active hub cluster.
7. Once all components are recovered, edit the global KlusterletConfig on the new hub and remove the parameter appliedManifestWorkEvictionGracePeriod and its value.
8. If only the active hub cluster is down, restore the hub by performing hub recovery, and restoring the backups on the passive hub. If the managed clusters are still accessible, no further action is required.
9. If the primary managed cluster is down, along with the active hub cluster, you need to fail over the workloads from the primary managed cluster to the secondary managed cluster. For failover instructions, based on your workload type, see Subscription-based applications or ApplicationSet-based applications.

10. Verify that the failover is successful. If the Primary managed cluster is also down, then the PROGRESSION status for the workload would be in Cleaning Up phase until the down Primary managed cluster is back online and successfully imported into the RHACM console.

    On the passive hub cluster, run the following command to check the PROGRESSION status.

    $ oc get drpc -o wide -A