Chapter 6. Monitoring disaster recovery health

Procedure

1. On the Hub cluster, open a terminal window and perform the next steps.

2. Add label to openshift-operator namespace.

   $ oc label namespace openshift-operators openshift.io/cluster-monitoring='true'

3. Create the configmap file named observability-metrics-custom-allowlist.yaml.

   You can use the following YAML to list the disaster recovery metrics on Hub cluster. For details, see Adding custom metrics. To know more about ramen metrics, see Disaster recovery metrics.

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: observability-metrics-custom-allowlist
     namespace: open-cluster-management-observability
   data:
     metrics_list.yaml: |
       names:
         - ramen_last_sync_timestamp_seconds
         - ramen_policy_schedule_interval_seconds
       matches:
         - __name__="csv_succeeded",exported_namespace="openshift-dr-system",name=~"odr-cluster-operator.*"
         - __name__="csv_succeeded",exported_namespace="openshift-operators",name=~"volsync.*"
       recording_rules:
         - record: count_persistentvolumeclaim_total
           expr: count(kube_persistentvolumeclaim_info)
         - record: ramen_sync_duration_seconds
           expr: sum by (obj_name, obj_namespace, obj_type, job, policyname)(time() - (ramen_last_sync_timestamp_seconds > 0))

4. In the open-cluster-management-observability namespace, run the following command:

   $ oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml

5. After observability-metrics-custom-allowlist yaml is created, RHACM will start collecting the listed OpenShift Data Foundation metrics from all the managed clusters.

   To exclude a specific managed cluster from collecting the observability data, add the following cluster label to the clusters: observability: disabled.

6. Create the configmap file named thanos-ruler-custom-rules.yaml and add the name of the custom alert rules to the custom_rules.yaml parameter.

   You can use the following YAML to create an alert against the ramen metrics on the Hub cluster. For details, see Adding custom metrics. To know more about the alerts, see Disaster Recovery alerts.

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: thanos-ruler-custom-rules
     namespace: open-cluster-management-observability
   data:
     custom_rules.yaml: |
       groups:
         - name: ramen-alerts
           rules:
           - record: ramen_rpo_difference
             expr: ramen_sync_duration_seconds{job="ramen-hub-operator-metrics-service"} / on(policyname, job) group_left() (ramen_policy_schedule_interval_seconds{job="ramen-hub-operator-metrics-service"})
           - alert: VolumeSynchronizationDelay
             expr: ramen_rpo_difference >= 3
             for: 5s
             labels:
               cluster: "{{ $labels.cluster }}"
               severity: critical
             annotations:
               description: "Syncing of volumes (DRPC: {{ $labels.obj_name }}, Namespace: {{ $labels.obj_namespace }}) is taking more than thrice the scheduled snapshot interval. This may cause data loss and a backlog of replication requests."
               alert_type: "DisasterRecovery"
           - alert: VolumeSynchronizationDelay
             expr: ramen_rpo_difference > 2 and ramen_rpo_difference < 3
             for: 5s
             labels:
               cluster: "{{ $labels.cluster }}"
               severity: warning
             annotations:
               description: "Syncing of volumes (DRPC: {{ $labels.obj_name }}, Namespace: {{ $labels.obj_namespace }}) is taking more than twice the scheduled snapshot interval. This may cause data loss and impact replication requests."
               alert_type: "DisasterRecovery"

7. Run the following command in the open-cluster-management-observability namespace:

   $ oc apply -n open-cluster-management-observability -f thanos-ruler-custom-rules.yaml

Procedure

1. On the Hub cluster, ensure All Clusters option is selected.
2. Refresh the console to make the DR monitoring dashboard tab accessible.
3. Navigate to Data Services and click Data policies.
4. On the Overview tab, you can view the health status of the operators, clusters and applications. Green tick indicates that the operators are running and available..
5. Click the Disaster recovery tab to view a list of DR policy details and connected applications.