3.17. Hub recovery using Red Hat Advanced Cluster Management [Technology preview]

Procedure

1. Ensure that RHACM operator and MultiClusterHub is installed on the passive hub cluster. See RHACM installation guide for instructions.

   After the operator is successfully installed, a popover with a message that the Web console update is available appears on the user interface. Click Refresh web console from this popover for the console changes to reflect.

2. Before hub recovery, configure backup and restore. See Backup and restore topics of RHACM Business continuity guide.
3. Install the multicluster orchestrator (MCO) operator along with Red Hat OpenShift GitOps operator on the passive RHACM hub prior to the restore. For instructions to restore your RHACM hub, see Installing OpenShift Data Foundation Multicluster Orchestrator operator.
4. Ensure that .spec.cleanupBeforeRestore is set to None for the Restore.cluster.open-cluster-management.io resource. For details, see Restoring passive resources while checking for backups chapter of RHACM documentation.
5. If SSL access across clusters was configured manually during setup, then re-configure SSL access across clusters. For instructions, see Configuring SSL access across clusters chapter.

6. On the passive hub, add the monitoring label for collecting the disaster recovery metrics. For alert details, see Disaster recovery alerts chapter.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

Procedure

1. Restore the backups on the passive hub cluster. For information, see Restoring a hub cluster from backup.

   重要

   Recovering a failed hub to its passive instance will only restore applications and their DR protected state to its last scheduled backup. Any application that was DR protected after the last scheduled backup would need to be protected again on the new hub.

2. Verify that the Primary and Seconday managed clusters are successfully imported into the RHACM console and they are accessible. If any of the managed clusters are down or unreachable then they will not be successfully imported.
3. Wait until DRPolicy validation succeeds.

4. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with a unique name.

   $ oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

5. Refresh the RHACM console to make the DR monitoring dashboard tab accessible if it was enabled on the Active hub cluster.
6. If only the active hub cluster is down, restore the hub by performing hub recovery, and restoring the backups on the passive hub. If the managed clusters are still accessible, no further action is required.
7. If the primary managed cluster is down, along with the active hub cluster, you need to fail over the workloads from the primary managed cluster to the secondary managed cluster. For failover instructions, based on your workload type, see Subscription-based applications or ApplicationSet-based applications.

8. Verify that the failover is successful. When the Primary managed cluster is down, then the PROGRESSION status for the workload would be in Cleaning Up phase until the down managed cluster is back online and successfully imported into the RHACM console.

   On the passive hub cluster, run the following command to check the PROGRESSION status.

   $ oc get drpc -o wide -A

   Example output:

   NAMESPACE              NAME                                    AGE    PREFERREDCLUSTER    FAILOVERCLUSTER     DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION        PEER READY
   [...]
   busybox                cephfs-busybox-placement-1-drpc         103m   cluster-1            cluster-2          Failover       FailedOver     Cleaning Up   2024-04-15T09:12:23Z                   False
   busybox                cephfs-busybox-placement-1-drpc         102m   cluster-1                                              Deployed       Completed     2024-04-15T07:40:09Z   37.200569819s   True
   [...]