Chapter 9. Managing alerts

1. Navigate to the Observe Alerting Silences page in the OpenShift Container Platform web console.
2. Select Create Silence.
3. Set the schedule, duration, and label details for an alert in the Create Silence form. You must also add a comment for the silence.
4. To create silences for alerts that match the label sectors that you entered in the previous step, select Silence.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Edit silence.

   Alternatively, you can select Actions Edit Silence in the Silence Details page for a silence.

3. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Edit Silence in the Silence Details page for a silence.
6. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Expire silence.

   Alternatively, you can select Actions Expire Silence in the Silence Details page for a silence.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Expire Silence in the Silence Details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert 1
         for: 1m 2
         expr: version{job="prometheus-example-app"} == 0 3
         labels:
           severity: warning 4
         annotations:
           message: This is an example alert. 5

   1

   The name of the alerting rule you want to create.

   2

   The duration for which the condition should be true before an alert is fired.

   3

   The PromQL query expression that defines the new rule.

   4

   The severity that alerting rule assigns to the alert.

   5

   The message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

Procedure

1. In the Administrator perspective, navigate to Observe Alerting Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1alpha1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring 1
   spec:
     configs:
     - sourceLabels: [alertname,severity] 2
       regex: "Watchdog;none" 3
       targetLabel: severity 4
       replacement: critical 5
       action: Replace 6

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The source labels for the values you want to modify.

   3

   The regular expression against which the value of sourceLabels is matched.

   4

   The target label of the value you want to modify.

   5

   The new value to replace the target label.

   6

   The relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

   Important

   You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

   apiVersion: monitoring.openshift.io/v1alpha1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring 1
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert 2
         for: 1m 3
         expr: vector(1) 4
         labels:
           severity: warning 5
         annotations:
           message: This is an example alert. 6

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The name of the alerting rule you want to create.

   3

   The duration for which the condition should be true before an alert is fired.

   4

   The PromQL query expression that defines the new rule.

   5

   The severity that alerting rule assigns to the alert.

   6

   The message associated with the alert.

   Important

   You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

Procedure

1. In the Administrator perspective, go to Administration Cluster Settings Configuration Alertmanager.

   Note

   Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Click Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.
     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.
     3. Select whether TLS is required.
     4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

   1. Add routing label names and values in the Routing labels section of the form.
   2. Select Regular expression if want to use a regular expression.
   3. Click Add label to add further routing labels.
6. Click Create to create the receiver.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

   Note

   For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration defined in the AlertmanagerConfig object for namespace ns1 only applies to PrometheusRules resources in the same namespace.

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.

Procedure

1. Open the Alertmanager YAML configuration file:

   • To open the Alertmanager configuration from the CLI:

     1. Print the currently active Alertmanager configuration from the alertmanager-main secret into alertmanager.yaml file:

        $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

     2. Open the alertmanager.yaml file.

   • To open the Alertmanager configuration from the OpenShift Container Platform web console:

     1. Go to the Administration Cluster Settings Configuration Alertmanager YAML page of the web console.

2. Edit the Alertmanager configuration by updating parameters in the YAML:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s 1
     group_interval: 5m 2
     repeat_interval: 12h 3
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=<your_service>" 4
       routes:
       - matchers:
         - <your_matching_rules> 5
         receiver: <receiver> 6
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
     <receiver_configuration> 7

   1

   Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.

   2

   Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.

   3

   Specify the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

   4

   Specify the name of the service that fires the alerts.

   5

   Specify labels to match your alerts.

   6

   Specify the name of the receiver to use for the alerts.

   7

   Specify the receiver configuration.

   Important
   • Use the matchers key name to indicate the matchers that an alert has to fulfill to match the node. Do not use the match or match_re key names, which are both deprecated and planned for removal in a future release.

   • If you define inhibition rules, use the following key names:

     • target_matchers: to indicate the target matchers
     • source_matchers: to indicate the source matchers

     Do not use the target_match, target_match_re, source_match, or source_match_re key names, which are deprecated and planned for removal in a future release.

   The following Alertmanager configuration example configures PagerDuty as an alert receiver:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=example-app"
       routes:
       - matchers:
         - "severity=critical"
         receiver: team-frontend-page
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page
     pagerduty_configs:
     - service_key: "<your_key>"

   With this configuration, alerts of critical severity that are fired by the example-app service are sent through the team-frontend-page receiver. Typically, these types of alerts would be paged to an individual or a critical response team.

3. Apply the new configuration in the file:

   • To apply the changes from the CLI, run the following command:

     $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

   • To apply the changes from the OpenShift Container Platform web console, click Save.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 1
       receiver: <receiver> 2
   receivers:
   - name: Default
   - name: <receiver>
     <receiver_configuration> 3

   1

   Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.

   2

   Specify the name of the receiver to use for the alerts group.

   3

   Specify the receiver configuration.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-