Chapter 7. Managing alerts

1. From the Administrator perspective of the Red Hat OpenShift Service on AWS web console, go to Observe Alerting Alerts.
2. For the alert that you want to silence, click and select Silence alert to open the Silence alert page with a default configuration for the chosen alert.

3. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

4. To save the silence, click Silence.

1. From the Administrator perspective of the Red Hat OpenShift Service on AWS web console, go to Observe Alerting Silences.
2. Click Create silence.

3. On the Create silence page, set the schedule, duration, and label details for an alert.

   Note

   You must add a comment before saving a silence.

4. To create silences for alerts that match the labels that you entered, click Silence.

1. From the Developer perspective of the Red Hat OpenShift Service on AWS web console, go to Observe and go to the Alerts tab.
2. Select the project that you want to silence an alert for from the Project: list.
3. If necessary, expand the details for the alert by clicking a greater than symbol (>) next to the alert name.
4. Click the alert message in the expanded view to open the Alert details page for the alert.
5. Click Silence alert to open the Silence alert page with a default configuration for the alert.

6. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

7. To save the silence, click Silence.

1. From the Developer perspective of the Red Hat OpenShift Service on AWS web console, go to Observe and go to the Silences tab.
2. Select the project that you want to silence alerts for from the Project: list.
3. Click Create silence.

4. On the Create silence page, set the duration and label details for an alert.

   Note

   You must add a comment before saving a silence.

5. To create silences for alerts that match the labels that you entered, click Silence.

Procedure

1. From the Administrator perspective of the Red Hat OpenShift Service on AWS web console, go to Observe Alerting Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. From the Developer perspective of the Red Hat OpenShift Service on AWS web console, go to Observe and go to the Silences tab.
2. Select the project that you want to edit silences for from the Project: list.

3. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

4. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Procedure

1. Go to Observe Alerting Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. From the Developer perspective of the Red Hat OpenShift Service on AWS web console, go to Observe and go to the Silences tab.
2. Select the project that you want to expire a silence for from the Project: list.
3. For the silence or silences you want to expire, select the checkbox in the corresponding row.

4. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert 1
         for: 1m 2
         expr: version{job="prometheus-example-app"} == 0 3
         labels:
           severity: warning 4
         annotations:
           message: This is an example alert. 5

   1

   The name of the alerting rule you want to create.

   2

   The duration for which the condition should be true before an alert is fired.

   3

   The PromQL query expression that defines the new rule.

   4

   The severity that alerting rule assigns to the alert.

   5

   The message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Configure projects in which you want to create alerting rules that are not bound to a specific project:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       namespacesWithoutLabelEnforcement: [ <namespace1>, <namespace2> ] 1
       # ...

   1

   Specify one or more projects in which you want to create cross-project alerting rules. Prometheus and Thanos Ruler for user-defined monitoring do not enforce the namespace label in PrometheusRule objects created in these projects, making the PrometheusRule objects applicable to all projects.

3. Create a YAML file for alerting rules. In this example, it is called example-cross-project-alerting-rule.yaml.

4. Add an alerting rule configuration to the YAML file. The following example creates a new cross-project alerting rule called example-security. The alerting rule fires when a user project does not enforce the restricted pod security policy:

   Example cross-project alerting rule

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-security
     namespace: ns1 1
   spec:
     groups:
       - name: pod-security-policy
         rules:
           - alert: "ProjectNotEnforcingRestrictedPolicy" 2
             for: 5m 3
             expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"} 4
             annotations:
               message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy." 5
             labels:
               severity: warning 6

   1

   Ensure that you specify the project that you defined in the namespacesWithoutLabelEnforcement field.

   2

   The name of the alerting rule you want to create.

   3

   The duration for which the condition should be true before an alert is fired.

   4

   The PromQL query expression that defines the new rule. You can use label matchers on the namespace label to filter which projects are included or excluded from the alerting rule.

   5

   The message associated with the alert.

   6

   The severity that alerting rule assigns to the alert.

   Important

   Ensure that you create a specific cross-project alerting rule in only one of the projects that you specified in the namespacesWithoutLabelEnforcement field. If you create the same cross-project alerting rule in multiple projects, it results in repeated alerts.

5. Apply the configuration file to the cluster:

   $ oc apply -f example-cross-project-alerting-rule.yaml

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

Procedure

1. From the Administrator perspective of the Red Hat OpenShift Service on AWS web console, go to Observe Alerting Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. In the cluster-monitoring-config config map, disable the option to create cross-project alerting rules by setting the rulesWithoutLabelEnforcementAllowed value under data/config.yaml/userWorkload to false:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       userWorkload:
         rulesWithoutLabelEnforcementAllowed: false
       # ...

3. Save the file to apply the changes.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   global:
     http_config:
       proxy_from_environment: true 1
   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 2
       receiver: <receiver> 3
   receivers:
   - name: Default
   - name: <receiver>
     <receiver_configuration> 4

   1

   If you configured an HTTP cluster-wide proxy, set the proxy_from_environment parameter to true to enable proxying for all alert receivers.

   2

   Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.

   3

   Specify the name of the receiver to use for the alerts group.

   4

   Specify the receiver configuration.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-