이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 16. Getting information about alerts, silences, and alerting rules from the Developer perspective
The Alerting UI provides detailed information about alerts and their governing alerting rules and silences.
Prerequisites
- You have access to the cluster as a user with view permissions for the project that you are viewing alerts for.
Procedure
To obtain information about alerts, silences, and alerting rules:
-
From the Developer perspective of the OpenShift Container Platform web console, go to the Observe
<project_name> Alerts page. View details for an alert, silence, or an alerting rule:
- Alert details can be viewed by clicking a greater than symbol (>) next to an alert name and then selecting the alert from the list.
Silence details can be viewed by clicking a silence in the Silenced by section of the Alert details page. The Silence details page includes the following information:
- Alert specification
- Start time
- End time
- Silence state
- Number and list of firing alerts
-
Alerting rule details can be viewed by clicking the
menu next to an alert in the Alerts page and then clicking View Alerting Rule.
Only alerts, silences, and alerting rules relating to the selected project are displayed in the Developer perspective.
Additional resources
- Cluster Monitoring Operator runbooks (Cluster Monitoring Operator GitHub repository)
16.1. Managing silences
You can create a silence for an alert in the OpenShift Container Platform web console in the Developer perspective. After you create silences, you can view, edit, and expire them. You also do not receive notifications about a silenced alert when the alert fires.
When you create silences, they are replicated across Alertmanager pods. However, if you do not configure persistent storage for Alertmanager, silences might be lost. This can happen, for example, if all Alertmanager pods restart at the same time.
Additional resources
16.1.1. Silencing alerts from the Developer perspective
You can silence a specific alert or silence alerts that match a specification that you define.
Prerequisites
-
If you are a cluster administrator, you have access to the cluster as a user with the
cluster-admin
role. If you are a non-administrator user, you have access to the cluster as a user with the following user roles:
-
The
cluster-monitoring-view
cluster role, which allows you to access Alertmanager. -
The
monitoring-alertmanager-edit
role, which permits you to create and silence alerts in the Administrator perspective in the web console. -
The
monitoring-rules-edit
cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.
-
The
Procedure
To silence a specific alert:
- From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Alerts tab.
- Select the project that you want to silence an alert for from the Project: list.
- If necessary, expand the details for the alert by clicking a greater than symbol (>) next to the alert name.
- Click the alert message in the expanded view to open the Alert details page for the alert.
- Click Silence alert to open the Silence alert page with a default configuration for the alert.
Optional: Change the default configuration details for the silence.
NoteYou must add a comment before saving a silence.
- To save the silence, click Silence.
To silence a set of alerts:
- From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
- Select the project that you want to silence alerts for from the Project: list.
- Click Create silence.
On the Create silence page, set the duration and label details for an alert.
NoteYou must add a comment before saving a silence.
- To create silences for alerts that match the labels that you entered, click Silence.
16.1.2. Editing silences from the Developer perspective
You can edit a silence, which expires the existing silence and creates a new one with the changed configuration.
Prerequisites
-
If you are a cluster administrator, you have access to the cluster as a user with the
cluster-admin
role. If you are a non-administrator user, you have access to the cluster as a user with the following user roles:
-
The
cluster-monitoring-view
cluster role, which allows you to access Alertmanager. -
The
monitoring-rules-edit
cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.
-
The
Procedure
- From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
- Select the project that you want to edit silences for from the Project: list.
For the silence you want to modify, click
and select Edit silence.
Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.
- On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.
16.1.3. Expiring silences from the Developer perspective
You can expire a single silence or multiple silences. Expiring a silence deactivates it permanently.
You cannot delete expired, silenced alerts. Expired silences older than 120 hours are garbage collected.
Prerequisites
-
If you are a cluster administrator, you have access to the cluster as a user with the
cluster-admin
role. If you are a non-administrator user, you have access to the cluster as a user with the following user roles:
-
The
cluster-monitoring-view
cluster role, which allows you to access Alertmanager. -
The
monitoring-rules-edit
cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.
-
The
Procedure
- From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
- Select the project that you want to expire a silence for from the Project: list.
- For the silence or silences you want to expire, select the checkbox in the corresponding row.
Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.
Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.
16.2. Managing alerting rules for user-defined projects
In OpenShift Container Platform, you can create, view, edit, and remove alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics.
Additional resources
16.2.1. Creating alerting rules for user-defined projects
You can create alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics.
- When you create an alerting rule, a project label is enforced on it even if a rule with the same name exists in another project.
- To help users understand the impact and cause of the alert, ensure that your alerting rule contains an alert message and severity value.
Prerequisites
- You have enabled monitoring for user-defined projects.
-
You are logged in as a cluster administrator or as a user that has the
monitoring-rules-edit
cluster role for the project where you want to create an alerting rule. -
You have installed the OpenShift CLI (
oc
).
Procedure
-
Create a YAML file for alerting rules. In this example, it is called
example-app-alerting-rule.yaml
. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named
example-alert
. The alerting rule fires an alert when theversion
metric exposed by the sample service becomes0
:apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: example-alert namespace: ns1 spec: groups: - name: example rules: - alert: VersionAlert 1 for: 1m 2 expr: version{job="prometheus-example-app"} == 0 3 labels: severity: warning 4 annotations: message: This is an example alert. 5
Apply the configuration file to the cluster:
$ oc apply -f example-app-alerting-rule.yaml
Additional resources
- Monitoring stack architecture
- Alerting (Prometheus documentation)
16.2.2. Accessing alerting rules for user-defined projects
To list alerting rules for a user-defined project, you must have been assigned the monitoring-rules-view
cluster role for the project.
Prerequisites
- You have enabled monitoring for user-defined projects.
-
You are logged in as a user that has the
monitoring-rules-view
cluster role for your project. -
You have installed the OpenShift CLI (
oc
).
Procedure
To list alerting rules in
<project>
:$ oc -n <project> get prometheusrule
To list the configuration of an alerting rule, run the following:
$ oc -n <project> get prometheusrule <rule> -o yaml
16.2.3. Removing alerting rules for user-defined projects
You can remove alerting rules for user-defined projects.
Prerequisites
- You have enabled monitoring for user-defined projects.
-
You are logged in as a cluster administrator or as a user that has the
monitoring-rules-edit
cluster role for the project where you want to create an alerting rule. -
You have installed the OpenShift CLI (
oc
).
Procedure
To remove rule
<foo>
in<namespace>
, run the following:$ oc -n <namespace> delete prometheusrule <foo>
Additional resources
- Alertmanager (Prometheus documentation)