Chapter 6. Managing alerts

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe Alerting Alerts.
2. For the alert that you want to silence, click and select Silence alert to open the Silence alert page with a default configuration for the chosen alert.

3. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

4. To save the silence, click Silence.

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe Alerting Silences.
2. Click Create silence.

3. On the Create silence page, set the schedule, duration, and label details for an alert.

   Note

   You must add a comment before saving a silence.

4. To create silences for alerts that match the labels that you entered, click Silence.

Procedure

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe Alerting Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. Go to Observe Alerting Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring 

   1

   
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert 

   2

   
         for: 1m 

   3

   
         expr: vector(1) 

   4

   
         labels:
           severity: warning 

   5

   
         annotations:
           message: This is an example alert. 

   6

   Copy to Clipboard Toggle word wrap

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The name of the alerting rule you want to create.

   3

   The duration for which the condition should be true before an alert is fired.

   4

   The PromQL query expression that defines the new rule.

   5

   The severity that alerting rule assigns to the alert.

   6

   The message associated with the alert.

   Important

   You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring 

   1

   
   spec:
     configs:
     - sourceLabels: [alertname,severity] 

   2

   
       regex: "Watchdog;none" 

   3

   
       targetLabel: severity 

   4

   
       replacement: critical 

   5

   
       action: Replace 

   6

   Copy to Clipboard Toggle word wrap

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The source labels for the values you want to modify.

   3

   The regular expression against which the value of sourceLabels is matched.

   4

   The target label of the value you want to modify.

   5

   The new value to replace the target label.

   6

   The relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

   Important

   You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert 

   1

   
         for: 1m 

   2

   
         expr: version{job="prometheus-example-app"} == 0 

   3

   
         labels:
           severity: warning 

   4

   
         annotations:
           message: This is an example alert. 

   5

   Copy to Clipboard Toggle word wrap

   1

   The name of the alerting rule you want to create.

   2

   The duration for which the condition should be true before an alert is fired.

   3

   The PromQL query expression that defines the new rule.

   4

   The severity that alerting rule assigns to the alert.

   5

   The message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe Alerting Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Alerts tab.
2. Select the project that you want to silence an alert for from the Project: list.
3. If necessary, expand the details for the alert by clicking a greater than symbol (>) next to the alert name.
4. Click the alert message in the expanded view to open the Alert details page for the alert.
5. Click Silence alert to open the Silence alert page with a default configuration for the alert.

6. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

7. To save the silence, click Silence.

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
2. Select the project that you want to silence alerts for from the Project: list.
3. Click Create silence.

4. On the Create silence page, set the duration and label details for an alert.

   Note

   You must add a comment before saving a silence.

5. To create silences for alerts that match the labels that you entered, click Silence.

Procedure

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
2. Select the project that you want to edit silences for from the Project: list.

3. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

4. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
2. Select the project that you want to expire a silence for from the Project: list.
3. For the silence or silences you want to expire, select the checkbox in the corresponding row.

4. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert 

   1

   
         for: 1m 

   2

   
         expr: version{job="prometheus-example-app"} == 0 

   3

   
         labels:
           severity: warning 

   4

   
         annotations:
           message: This is an example alert. 

   5

   Copy to Clipboard Toggle word wrap

   1

   The name of the alerting rule you want to create.

   2

   The duration for which the condition should be true before an alert is fired.

   3

   The PromQL query expression that defines the new rule.

   4

   The severity that alerting rule assigns to the alert.

   5

   The message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

   Copy to Clipboard Toggle word wrap

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

   Copy to Clipboard Toggle word wrap