Chapter 10. Managing alerts

1. Navigate to the Observe Alerting Silences page in the OpenShift Container Platform web console.
2. Select Create Silence.
3. Set the schedule, duration, and label details for an alert in the Create Silence form. You must also add a comment for the silence.
4. To create silences for alerts that match the label sectors that you entered in the previous step, select Silence.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Edit silence.

   Alternatively, you can select Actions Edit Silence in the Silence Details page for a silence.

3. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Edit Silence in the Silence Details page for a silence.
6. In the Edit Silence page, enter your changes and select Silence. This will expire the existing silence and create one with the chosen configuration.

1. Navigate to the Observe Alerting Silences page.

2. For the silence you want to modify, select the in the last column and choose Expire silence.

   Alternatively, you can select Actions Expire Silence in the Silence Details page for a silence.

1. Navigate to the Observe <project_name> Alerts page.
2. Expand the details for an alert by selecting > to the left of the alert name. Select the name of the alert in the expanded view to open the Alert Details page for the alert.
3. Select the name of a silence in the Silenced By section in that page to navigate to the Silence Details page for the silence.
4. Select the name of a silence to navigate to its Silence Details page.
5. Select Actions Expire Silence in the Silence Details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. For example:

   Note

   When you create an alerting rule, a project label is enforced on it if a rule with the same name exists in another project.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

   This configuration creates an alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. You can list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

Procedure

1. In the Administrator perspective, navigate to Observe Alerting Alerting Rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml in the openshift-monitoring namespace.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1alpha1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring
   spec:
     configs:
     - sourceLabels: [alertname,severity] 1
       regex: "Watchdog;none" 2
       targetLabel: severity 3
       replacement: critical 4
       action: Replace 5

   1

   The source labels for the values you want to modify.

   2

   The regular expression against which the value of sourceLabels is matched.

   3

   The target label of the value you want to modify.

   4

   The new value to replace the target label.

   5

   The relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml in the openshift-monitoring namespace.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default watchdog alert:

   apiVersion: monitoring.openshift.io/v1alpha1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert 1
         expr: vector(1) 2

   1

   The name of the alerting rule you want to create.

   2

   The PromQL query expression that defines the new rule.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

Procedure

1. In the Administrator perspective, navigate to Administration Cluster Settings Configuration Alertmanager.

   Note

   Alternatively, you can navigate to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Select Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver Name and choose a Receiver Type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Select Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Select Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.
     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.
     3. Choose whether TLS is required.
     4. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors will be sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver:

   1. Add routing label names and values in the Routing Labels section of the form.
   2. Select Regular Expression if want to use a regular expression.
   3. Select Add Label to add further routing labels.
6. Select Create to create the receiver.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

   Note

   For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration defined in the AlertmanagerConfig object for namespace ns1 only applies to PrometheusRules resources in the same namespace.

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.