Managing alerts

Procedure

1. In the OpenShift Container Platform web console, go to Observe → Alerting → Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. Go to Observe → Alerting → Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert
         for: 1m
         expr: vector(1)
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   namespace

   Specifies the project. Ensure that the namespace is openshift-monitoring.

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

   Important

   You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring
   spec:
     configs:
     - sourceLabels: [alertname,severity]
       regex: "Watchdog;none"
       targetLabel: severity
       replacement: critical
       action: Replace

   where:

   namespace

   Must be set to openshift-monitoring.

   sourceLabels

   Defines the source labels for the values you want to modify.

   regex

   Defines the regular expression against which the value of sourceLabels is matched.

   targetLabel

   Defines the target label of the value you want to modify.

   replacement

   Defines the new value to replace the target label.

   action

   Defines the relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

   Important

   You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-example-app"} == 0
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Configure projects in which you want to create alerting rules that are not bound to a specific project:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       namespacesWithoutLabelEnforcement: [ <namespace1>, <namespace2> ]
       # ...

   where:

   namespacesWithoutLabelEnforcement

   Specifies one or more projects in which you want to create cross-project alerting rules. Prometheus and Thanos Ruler for user-defined monitoring do not enforce the namespace label in PrometheusRule objects created in these projects, making the PrometheusRule objects applicable to all projects.

3. Create a YAML file for alerting rules. In this example, it is called example-cross-project-alerting-rule.yaml.

4. Add an alerting rule configuration to the YAML file. The following example creates a new cross-project alerting rule called example-security. The alerting rule fires when a user project does not enforce the restricted pod security policy:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-security
     namespace: ns1
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "ProjectNotEnforcingRestrictedPolicy"
             for: 5m
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"}
             annotations:
               message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy."
             labels:
               severity: warning

   where:

   namespace

   Specifies the project. Ensure that you specify the project that you defined in the namespacesWithoutLabelEnforcement field.

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule. You can use label matchers on the namespace label to filter which projects are included or excluded from the alerting rule.

   annotations.message

   Specifies the message associated with the alert.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   Important

   Ensure that you create a specific cross-project alerting rule in only one of the projects that you specified in the namespacesWithoutLabelEnforcement field. If you create the same cross-project alerting rule in multiple projects, it results in repeated alerts.

5. Apply the configuration file to the cluster:

   $ oc apply -f example-cross-project-alerting-rule.yaml

Procedure

1. In the OpenShift Container Platform web console, go to Observe → Alerting → Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. In the cluster-monitoring-config config map, disable the option to create cross-project alerting rules by setting the rulesWithoutLabelEnforcementAllowed value under data/config.yaml/userWorkload to false:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       userWorkload:
         rulesWithoutLabelEnforcementAllowed: false
       # ...

3. Save the file to apply the changes.

Procedure

1. In the OpenShift Container Platform web console, go to Observe → Alerting → Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. Go to Observe → Alerting → Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         for: 1m
         expr: version{job="prometheus-example-app"} == 0
         labels:
           severity: warning
         annotations:
           message: This is an example alert.

   where:

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   annotations.message

   Specifies the message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Configure projects in which you want to create alerting rules that are not bound to a specific project:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       namespacesWithoutLabelEnforcement: [ <namespace1>, <namespace2> ]
       # ...

   where:

   namespacesWithoutLabelEnforcement

   Specifies one or more projects in which you want to create cross-project alerting rules. Prometheus and Thanos Ruler for user-defined monitoring do not enforce the namespace label in PrometheusRule objects created in these projects, making the PrometheusRule objects applicable to all projects.

3. Create a YAML file for alerting rules. In this example, it is called example-cross-project-alerting-rule.yaml.

4. Add an alerting rule configuration to the YAML file. The following example creates a new cross-project alerting rule called example-security. The alerting rule fires when a user project does not enforce the restricted pod security policy:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-security
     namespace: ns1
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "ProjectNotEnforcingRestrictedPolicy"
             for: 5m
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"}
             annotations:
               message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy."
             labels:
               severity: warning

   where:

   namespace

   Specifies the project. Ensure that you specify the project that you defined in the namespacesWithoutLabelEnforcement field.

   alert

   Specifies the name of the alerting rule you want to create.

   for

   Specifies the duration for which the condition should be true before an alert is fired.

   expr

   Specifies the PromQL query expression that defines the new rule. You can use label matchers on the namespace label to filter which projects are included or excluded from the alerting rule.

   annotations.message

   Specifies the message associated with the alert.

   labels.severity

   Specifies the severity that alerting rule assigns to the alert.

   Important

   Ensure that you create a specific cross-project alerting rule in only one of the projects that you specified in the namespacesWithoutLabelEnforcement field. If you create the same cross-project alerting rule in multiple projects, it results in repeated alerts.

5. Apply the configuration file to the cluster:

   $ oc apply -f example-cross-project-alerting-rule.yaml

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml