Red Hat Summit Red Hat SummitSuporteConsoleDesenvolvedoresComeçar um testeContato

Este conteúdo não está disponível no idioma selecionado.

Chapter 2. Managing observability alerts

Receive and define alerts for the observability service to be notified of hub cluster and managed cluster changes.

2.1. Configuring Alertmanager
Copiar o link

Integrate external messaging tools such as email, Slack, and PagerDuty to receive notifications from Alertmanager. You must override the alertmanager-config secret in the open-cluster-management-observability namespace to add integrations, and configure routes for Alertmanager. Complete the following steps to update the custom receiver rules:

  1. Extract the data from the alertmanager-config secret. Run the following command: 

    oc -n open-cluster-management-observability get secret alertmanager-config --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml
    Copy to Clipboard Toggle word wrap

  2. Edit and save the alertmanager.yaml file configuration by running the following command: 

    oc -n open-cluster-management-observability create secret generic alertmanager-config --from-file=alertmanager.yaml --dry-run -o=yaml |  oc -n open-cluster-management-observability replace secret --filename=-
    Copy to Clipboard Toggle word wrap

    Your updated secret might resemble the following content: 

    global
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'
templates:
- '/etc/alertmanager/template/*.tmpl'
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-X-mails
  routes:
  - match_re:
      service: ^(foo1|foo2|baz)$
    receiver: team-X-mails
    Copy to Clipboard Toggle word wrap

Your changes are applied immediately after it is modified. For an example of Alertmanager, see prometheus/alertmanager.

2.2. Forwarding alerts
Copiar o link

After you enable observability, alerts from your OpenShift Container Platform managed clusters are automatically sent to the hub cluster. You can use the alertmanager-config YAML file to configure alerts with an external notification system.

View the following example of the alertmanager-config YAML file: 

global:
  slack_api_url: '<slack_webhook_url>'

route:
  receiver: 'slack-notifications'
  group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#alerts'
    text: 'https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}'
Copy to Clipboard Toggle word wrap

If you want to configure a proxy for alert forwarding, add the following global entry to the alertmanager-config YAML file: 

global:
  slack_api_url: '<slack_webhook_url>'
  http_config:
    proxy_url: http://****
Copy to Clipboard Toggle word wrap

2.2.1. Disabling alert forwarding for managed clusters
Copiar o link

To disable alert forwarding for managed clusters, add the following annotation to the MultiClusterObservability custom resource: 

metadata:
      annotations:
        mco-disable-alerting: "true"
Copy to Clipboard Toggle word wrap

When you set the annotation, the alert forwarding configuration on the managed clusters is reverted. Any changes made to the ocp-monitoring-config config map in the openshift-monitoring namespace are also reverted. Setting the annotation ensures that the ocp-monitoring-config config map is no longer managed or updated by the observability operator endpoint. After you update the configuration, the Prometheus instance on your managed cluster restarts.

Important: Metrics on your managed cluster are lost if you have a Prometheus instance with a persistent volume for metrics, and the Prometheus instance restarts. Metrics from the hub cluster are not affected.

When the changes are reverted, a ConfigMap named cluster-monitoring-reverted is created in the open-cluster-management-addon-observability namespace. Any new, manually added alert forward configurations are not reverted from the ConfigMap.

Verify that the hub cluster alert manager is no longer propagating managed cluster alerts to third-party messaging tools. See the previous section, Configuring Alertmanager.

2.3. Silencing alerts
Copiar o link

Add alerts that you do not want to receive. You can silence alerts by the alert name, match label, or time duration. After you add the alert that you want to silence, an ID is created. Your ID for your silenced alert might resemble the following string, d839aca9-ed46-40be-84c4-dca8773671da.

Continue reading for ways to silence alerts:

  • To silence a Red Hat Advanced Cluster Management alert, you must have access to the alertmanager pods in the open-cluster-management-observability namespace. For example, enter the following command in the observability-alertmanager-0 pod terminal to silence SampleAlert: 

    amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" alertname="SampleAlert"
    Copy to Clipboard Toggle word wrap

  • Silence an alert by using multiple match labels. The following command uses match-label-1 and match-label-2: 

    amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" <match-label-1>=<match-value-1> <match-label-2>=<match-value-2>
    Copy to Clipboard Toggle word wrap

  • If you want to silence an alert for a specific period of time, use the --duration flag. Run the following command to silence the SampleAlert for an hour: 

    amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" --duration="1h" alertname="SampleAlert"
    Copy to Clipboard Toggle word wrap

    You can also specify a start or end time for the silenced alert. Enter the following command to silence the SampleAlert at a specific start time: 

    amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" --start="2023-04-14T15:04:05-07:00" alertname="SampleAlert"
    Copy to Clipboard Toggle word wrap

  • To view all silenced alerts that are created, run the following command: 

    amtool silence --alertmanager.url="http://localhost:9093"
    Copy to Clipboard Toggle word wrap

  • If you no longer want an alert to be silenced, end the silencing of the alert by running the following command: 

    amtool silence expire --alertmanager.url="http://localhost:9093" "d839aca9-ed46-40be-84c4-dca8773671da"
    Copy to Clipboard Toggle word wrap

  • To end the silencing of all alerts, run the following command: 

    amtool silence expire --alertmanager.url="http://localhost:9093" $(amtool silence query --alertmanager.url="http://localhost:9093" -q)
    Copy to Clipboard Toggle word wrap

2.3.1. Migrating observability storage
Copiar o link

If you use alert silencers, you can migrate observability storage while retaining the silencers from its earlier state. To do this, migrate your Red Hat Advanced Cluster Management observability storage by creating new StatefulSets and PersistentVolumes (PV) resources that use your chosen StorageClass resource.

Note: The storage for PVs is different from the object storage used to store the metrics collected from your clusters.

When you use StatefulSets and PVs to migrate your observability data to new storage, it stores the following data components:

  • Observatorium or Thanos: Receives data then uploads it to object storage. Some of its components store data in PVs. For this data, the Observatorium or Thanos automatically regenerates the object storage on a startup, so there is no consequence if you lose this data.
  • Alertmanager: Only stores silenced alerts. If you want to keep these silenced alerts, you must migrate that data to the new PV.

To migrate your observability storage, complete the following steps:

  1. In the MultiClusterObservability, set the .spec.storageConfig.storageClass field to the new storage class.
  2. To ensure the data of the earlier PersistentVolumes is retained even when you delete the PersistentVolumeClaim, go to all your existing PersistentVolumes.
  3. Change the reclaimPolicy to "Retain": `oc patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'.
  4. Optional: To avoid losing data, see Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4.

  5. Delete both the StatefulSet and the PersistentVolumeClaim in the following StatefulSet resources:

    1. alertmanager-db-observability-alertmanager
    2. data-observability-thanos-compact
    3. data-observability-thanos-receive-default
    4. data-observability-thanos-store-shard
    5. Important: You might need to delete, then re-create, the MultiClusterObservability operator pod so that you can create the new StatefulSet.
  6. Re-create a new PersistentVolumeClaim with the same name but the correct StorageClass.
  7. Create a new PersistentVolumeClaim referring to the old PersistentVolume.
  8. Verify that the new StatefulSet and PersistentVolumes use the new StorageClass that you chose.

2.4. Suppressing alerts
Copiar o link

Suppress Red Hat Advanced Cluster Management alerts across your clusters globally that are less severe. Suppress alerts by defining an inhibition rule in the alertmanager-config in the open-cluster-management-observability namespace.

An inhibition rule mutes an alert when there is a set of parameter matches that match another set of existing matchers. In order for the rule to take effect, both the target and source alerts must have the same label values for the label names in the equal list. Your inhibit_rules might resemble the following: 

global:
  resolve_timeout: 1h
inhibit_rules:
1 

  - equal:
      - namespace
    source_match:
2 

      severity: critical
    target_match_re:
      severity: warning|info
Copy to Clipboard Toggle word wrap
1 1
The inhibit_rules parameter section is defined to look for alerts in the same namespace. When a critical alert is initiated within a namespace and if there are any other alerts that contain the severity level warning or info in that namespace, only the critical alerts are routed to the Alertmanager receiver. The following alerts might be displayed when there are matches: 
ALERTS{alertname="foo", namespace="ns-1", severity="critical"}
ALERTS{alertname="foo", namespace="ns-1", severity="warning"}
Copy to Clipboard Toggle word wrap
2 2
If the value of the source_match and target_match_re parameters do not match, the alert is routed to the receiver: 
ALERTS{alertname="foo", namespace="ns-1", severity="critical"}
ALERTS{alertname="foo", namespace="ns-2", severity="warning"}
Copy to Clipboard Toggle word wrap

  • To view suppressed alerts in Red Hat Advanced Cluster Management, enter the following command: 

    amtool alert --alertmanager.url="http://localhost:9093" --inhibited
    Copy to Clipboard Toggle word wrap

2.5. Additional resources
Copiar o link

Voltar ao topo
Red Hat logoGithubredditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar. Explore nossas atualizações recentes.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja o Blog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

Theme

© 2025 Red Hat