Managing alerts

Procedure

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. Go to Observe → Alerting → Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert
         for: 1m
         expr: vector(1)
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   namespace

   Specifies the project. Ensure that the namespace is openshift-monitoring.

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

   Important

   You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring
   spec:
     configs:
     - sourceLabels: [alertname,severity]
       regex: "Watchdog;none"
       targetLabel: severity
       replacement: critical
       action: Replace

   where:

   namespace

   Must be set to openshift-monitoring.

   sourceLabels

   Defines the source labels for the values you want to modify.

   regex

   Defines the regular expression against which the value of sourceLabels is matched.

   targetLabel

   Defines the target label of the value you want to modify.

   replacement

   Defines the new value to replace the target label.

   action

   Defines the relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

   Important

   You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-example-app"} == 0
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
2. Select the project that you want to edit silences for from the Project: list.

3. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

4. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. From the Developer perspective of the OpenShift Container Platform web console, go to Observe and go to the Silences tab.
2. Select the project that you want to expire a silence for from the Project: list.
3. For the silence or silences you want to expire, select the checkbox in the corresponding row.

4. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-example-app"} == 0
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml