1.4. Customizing observability

Create a ConfigMap named thanos-rule-custom-rules in the open-cluster-management-observability namespace. The key must be named, thanos-ruler-custom-rules.yaml, as shown in the following example. You can create multiple rules in the configuration:

By default, the out-of-the-box alert rules are defined in the ConfigMap in the open-cluster-management-observability namespace.

For example, you can create a custom alert rule that notifies you when your CPU usage passes your defined value:

data:
  custom_rules.yaml: |
    groups:
      - name: cluster-health
        rules:
        - alert: ClusterCPUHealth-jb
          annotations:
            summary: Notify when CPU utilization on a cluster is greater than the defined utilization limit
            description: "The cluster has a high CPU usage: {{ $value }} core for {{ $labels.cluster }} {{ $labels.clusterID }}."
          expr: |
            max(cluster:cpu_usage_cores:sum) by (clusterID, cluster, prometheus) > 0
          for: 5s
          labels:
            cluster: "{{ $labels.cluster }}"
            prometheus: "{{ $labels.prometheus }}"
            severity: critical

Note: If this is the first new custom rule, it is created immediately. For changes to the ConfigMap, you must restart the observability pods with the following command: kubectl rollout restart statefulset observability-observatorium-thanos-rule -n open-cluster-management-observability.

If you want to verify that the alert rules is functioning appropriately, complete the following steps: