Chapter 5. Configuring external Alertmanager instances


The OpenShift Container Platform monitoring stack includes a local Alertmanager instance that routes alerts from Prometheus.

You can add external Alertmanager instances to route alerts for core OpenShift Container Platform projects.

If you add the same external Alertmanager configuration for multiple clusters and disable the local instance for each cluster, you can then manage alert routing for multiple clusters by using a single external Alertmanager instance.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have created the cluster-monitoring-config ConfigMap object.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
  2. Add an additionalAlertmanagerConfigs section with configuration details under data/config.yaml/prometheusK8s:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        prometheusK8s:
          additionalAlertmanagerConfigs:
          - <alertmanager_specification> 1
    1
    Substitute <alertmanager_specification> with authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig).

    The following sample config map configures an additional Alertmanager for Prometheus by using a bearer token with client TLS authentication:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        prometheusK8s:
          additionalAlertmanagerConfigs:
          - scheme: https
            pathPrefix: /
            timeout: "30s"
            apiVersion: v1
            bearerToken:
              name: alertmanager-bearer-token
              key: token
            tlsConfig:
              key:
                name: alertmanager-tls
                key: tls.key
              cert:
                name: alertmanager-tls
                key: tls.crt
              ca:
                name: alertmanager-tls
                key: tls.ca
            staticConfigs:
            - external-alertmanager1-remote.com
            - external-alertmanager1-remote2.com
  3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

5.1. Disabling the local Alertmanager

A local Alertmanager that routes alerts from Prometheus instances is enabled by default in the openshift-monitoring project of the OpenShift Container Platform monitoring stack.

If you do not need the local Alertmanager, you can disable it by configuring the cluster-monitoring-config config map in the openshift-monitoring project.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.
  • You have created the cluster-monitoring-config config map.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
  2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        alertmanagerMain:
          enabled: false
  3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.