Chapter 8. Enabling alert routing for user-defined projects
In OpenShift Container Platform 4.16, a cluster administrator can enable alert routing for user-defined projects. This process consists of two general steps:
- Enable alert routing for user-defined projects to use the default platform Alertmanager instance or, optionally, a separate Alertmanager instance only for user-defined projects.
- Grant users permission to configure alert routing for user-defined projects.
After you complete these steps, developers and other users can configure custom alerts and alert routing for their user-defined projects.
8.1. Understanding alert routing for user-defined projects
As a cluster administrator, you can enable alert routing for user-defined projects. With this feature, you can allow users with the alert-routing-edit role to configure alert notification routing and receivers for user-defined projects. These notifications are routed by the default Alertmanager instance or, if enabled, an optional Alertmanager instance dedicated to user-defined monitoring.
Users can then create and configure user-defined alert routing by creating or editing the AlertmanagerConfig
objects for their user-defined projects without the help of an administrator.
After a user has defined alert routing for a user-defined project, user-defined alert notifications are routed as follows:
-
To the
alertmanager-main
pods in theopenshift-monitoring
namespace if using the default platform Alertmanager instance. -
To the
alertmanager-user-workload
pods in theopenshift-user-workload-monitoring
namespace if you have enabled a separate instance of Alertmanager for user-defined projects.
The following are limitations of alert routing for user-defined projects:
-
For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration in namespace
ns1
only applies toPrometheusRules
resources in the same namespace. -
When a namespace is excluded from user-defined monitoring,
AlertmanagerConfig
resources in the namespace cease to be part of the Alertmanager configuration.
8.2. Enabling the platform Alertmanager instance for user-defined alert routing
You can allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role. - A cluster administrator has enabled monitoring for user-defined projects.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Edit the
cluster-monitoring-config
ConfigMap
object:$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
Add
enableUserAlertmanagerConfig: true
in thealertmanagerMain
section underdata/config.yaml
:apiVersion: v1 kind: ConfigMap metadata: name: cluster-monitoring-config namespace: openshift-monitoring data: config.yaml: | # ... alertmanagerMain: enableUserAlertmanagerConfig: true 1 # ...
- 1
- Set the
enableUserAlertmanagerConfig
value totrue
to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.
- Save the file to apply the changes. The new configuration is applied automatically.
8.3. Enabling a separate Alertmanager instance for user-defined alert routing
In some clusters, you might want to deploy a dedicated Alertmanager instance for user-defined projects, which can help reduce the load on the default platform Alertmanager instance and can better separate user-defined alerts from default platform alerts. In these cases, you can optionally enable a separate instance of Alertmanager to send alerts for user-defined projects only.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role. - You have enabled monitoring for user-defined projects.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Edit the
user-workload-monitoring-config
ConfigMap
object:$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
Add
enabled: true
andenableAlertmanagerConfig: true
in thealertmanager
section underdata/config.yaml
:apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | alertmanager: enabled: true 1 enableAlertmanagerConfig: true 2
- 1
- Set the
enabled
value totrue
to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value tofalse
or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value tofalse
or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance. - 2
- Set the
enableAlertmanagerConfig
value totrue
to enable users to define their own alert routing configurations withAlertmanagerConfig
objects.
- Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.
Verification
Verify that the
user-workload
Alertmanager instance has started:# oc -n openshift-user-workload-monitoring get alertmanager
Example output
NAME VERSION REPLICAS AGE user-workload 0.24.0 2 100s
8.4. Granting users permission to configure alert routing for user-defined projects
You can grant users permission to configure alert routing for user-defined projects.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
cluster role. - You have enabled monitoring for user-defined projects.
- The user account that you are assigning the role to already exists.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Assign the
alert-routing-edit
cluster role to a user in the user-defined project:$ oc -n <namespace> adm policy add-role-to-user alert-routing-edit <user> 1
- 1
- For
<namespace>
, substitute the namespace for the user-defined project, such asns1
. For<user>
, substitute the username for the account to which you want to assign the role.