Chapter 9. Managing alerts

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. For example:

   Note

   When you create an alerting rule, a project label is enforced on it if a rule with the same name exists in another project.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

   This configuration creates an alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0.

   Important

   A user-defined alerting rule can include metrics for its own project and cluster metrics. You cannot include metrics for another user-defined project.

   For example, an alerting rule for the user-defined project ns1 can have metrics from ns1 and cluster metrics, such as the CPU and memory metrics. However, the rule cannot include metrics from ns2.

   Additionally, you cannot create alerting rules for the openshift-* core OpenShift Container Platform projects. OpenShift Container Platform monitoring by default provides a set of alerting rules for these projects.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   It takes some time to create the alerting rule.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file that includes a label with the key openshift.io/prometheus-rule-evaluation-scope and value leaf-prometheus. For example:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
     labels:
       openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

1. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   It takes some time to create the alerting rule.

Procedure

1. You can list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

Procedure

1. In the Administrator perspective, navigate to Observe Alerting Alerting Rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

1. Navigate to the Observe Alerting Silences page in the OpenShift Container Platform web console.
2. Select Create Silence.
3. Set the schedule, duration, and label details for an alert in the Create Silence form. You must also add a comment for the silence.
4. To create silences for alerts that match the label sectors that you entered in the previous step, select Silence.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Edit silence.

   Alternatively, you can select Actions Edit Silence in the Silence Details page for a silence.

3. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Edit Silence in the Silence Details page for a silence.
6. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Expire silence.

   Alternatively, you can select Actions Expire Silence in the Silence Details page for a silence.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Expire Silence in the Silence Details page for a silence.

Procedure

1. In the Administrator perspective, navigate to Administration Cluster Settings Configuration Alertmanager.

   Note

   Alternatively, you can navigate to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Select Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver Name and choose a Receiver Type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Select Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Select Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.
     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.
     3. Choose whether TLS is required.
     4. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors will be sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver:

   1. Add routing label names and values in the Routing Labels section of the form.
   2. Select Regular Expression if want to use a regular expression.
   3. Select Add Label to add further routing labels.
6. Select Create to create the receiver.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1alpha1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

   Note

   For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration defined in the AlertmanagerConfig object for namespace ns1 only applies to PrometheusRules resources in the same namespace.

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.