Chapter 5. Configuring secrets for Alertmanager
The OpenShift Container Platform monitoring stack includes Alertmanager, which routes alerts from Prometheus to endpoint receivers. If you need to authenticate with a receiver so that Alertmanager can send alerts to it, you can configure Alertmanager to use a secret that contains authentication credentials for the receiver.
For example, you can configure Alertmanager to use a secret to authenticate with an endpoint receiver that requires a certificate issued by a private Certificate Authority (CA). You can also configure Alertmanager to use a secret to authenticate with a receiver that requires a password file for Basic HTTP authentication. In either case, authentication details are contained in the Secret
object rather than in the ConfigMap
object.
5.1. Adding a secret to the Alertmanager configuration
You can add secrets to the Alertmanager configuration for core platform monitoring components by editing the cluster-monitoring-config
config map in the openshift-monitoring
project.
After you add a secret to the config map, the secret is mounted as a volume at /etc/alertmanager/secrets/<secret_name>
within the alertmanager
container for the Alertmanager pods.
Prerequisites
If you are configuring core OpenShift Container Platform monitoring components in the
openshift-monitoring
project:-
You have access to the cluster as a user with the
cluster-admin
cluster role. -
You have created the
cluster-monitoring-config
config map. -
You have created the secret to be configured in Alertmanager in the
openshift-monitoring
project.
-
You have access to the cluster as a user with the
If you are configuring components that monitor user-defined projects:
-
You have access to the cluster as a user with the
cluster-admin
cluster role, or as a user with theuser-workload-monitoring-config-edit
role in theopenshift-user-workload-monitoring
project. -
You have created the secret to be configured in Alertmanager in the
openshift-user-workload-monitoring
project. - A cluster administrator has enabled monitoring for user-defined projects.
-
You have access to the cluster as a user with the
-
You have installed the OpenShift CLI (
oc
).
Procedure
Edit the
ConfigMap
object.To add a secret configuration to Alertmanager for core platform monitoring:
Edit the
cluster-monitoring-config
config map in theopenshift-monitoring
project:$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
Add a
secrets:
section underdata/config.yaml/alertmanagerMain
with the following configuration:apiVersion: v1 kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: | alertmanagerMain: secrets: 1 - <secret_name_1> 2 - <secret_name_2>
- 1
- This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.
- 2
- The name of the
Secret
object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.
The following sample config map settings configure Alertmanager to use two
Secret
objects namedtest-secret-basic-auth
andtest-secret-api-token
:apiVersion: v1 kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: | alertmanagerMain: secrets: - test-secret-basic-auth - test-secret-api-token
To add a secret configuration to Alertmanager for user-defined project monitoring:
Edit the
user-workload-monitoring-config
config map in theopenshift-user-workload-monitoring
project:$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
Add a
secrets:
section underdata/config.yaml/alertmanager/secrets
with the following configuration:apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | alertmanager: secrets: 1 - <secret_name_1> 2 - <secret_name_2>
- 1
- This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.
- 2
- The name of the
Secret
object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.
The following sample config map settings configure Alertmanager to use two
Secret
objects namedtest-secret
andtest-secret-api-token
:apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | alertmanager: enabled: true secrets: - test-secret - test-api-receiver-token
- Save the file to apply the changes. The new configuration is applied automatically.
5.2. Attaching additional labels to your time series and alerts
You can attach custom labels to all time series and alerts leaving Prometheus by using the external labels feature of Prometheus.
Prerequisites
If you are configuring core OpenShift Container Platform monitoring components:
-
You have access to the cluster as a user with the
cluster-admin
cluster role. -
You have created the
cluster-monitoring-config
ConfigMap
object.
-
You have access to the cluster as a user with the
If you are configuring components that monitor user-defined projects:
-
You have access to the cluster as a user with the
cluster-admin
cluster role, or as a user with theuser-workload-monitoring-config-edit
role in theopenshift-user-workload-monitoring
project. - A cluster administrator has enabled monitoring for user-defined projects.
-
You have access to the cluster as a user with the
-
You have installed the OpenShift CLI (
oc
).
Procedure
Edit the
ConfigMap
object:To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:
Edit the
cluster-monitoring-config
ConfigMap
object in theopenshift-monitoring
project:$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
Define a map of labels you want to add for every metric under
data/config.yaml
:apiVersion: v1 kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: | prometheusK8s: externalLabels: <key>: <value> 1
- 1
- Substitute
<key>: <value>
with a map of key-value pairs where<key>
is a unique name for the new label and<value>
is its value.
Warning-
Do not use
prometheus
orprometheus_replica
as key names, because they are reserved and will be overwritten. -
Do not use
cluster
ormanaged_cluster
as key names. Using them can cause issues where you are unable to see data in the developer dashboards.
For example, to add metadata about the region and environment to all time series and alerts, use the following example:
apiVersion: v1 kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: | prometheusK8s: externalLabels: region: eu environment: prod
- Save the file to apply the changes. The new configuration is applied automatically.
To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:
Edit the
user-workload-monitoring-config
ConfigMap
object in theopenshift-user-workload-monitoring
project:$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
Define a map of labels you want to add for every metric under
data/config.yaml
:apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | prometheus: externalLabels: <key>: <value> 1
- 1
- Substitute
<key>: <value>
with a map of key-value pairs where<key>
is a unique name for the new label and<value>
is its value.
Warning-
Do not use
prometheus
orprometheus_replica
as key names, because they are reserved and will be overwritten. -
Do not use
cluster
ormanaged_cluster
as key names. Using them can cause issues where you are unable to see data in the developer dashboards.
NoteIn the
openshift-user-workload-monitoring
project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. SettingexternalLabels
forprometheus
in theuser-workload-monitoring-config
ConfigMap
object will only configure external labels for metrics and not for any rules.For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use the following example:
apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | prometheus: externalLabels: region: eu environment: prod
- Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.
Additional resources
- See Preparing to configure the monitoring stack for steps to create monitoring config maps.
- Enabling monitoring for user-defined projects