1.2. Configuring the monitoring stack

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

2. Put your PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         volumeClaimTemplate:
           spec:
             storageClassName: <storage_class>
             resources:
               requests:
                 storage: <amount_of_storage>

   Copy to Clipboard Toggle word wrap

   See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

   For example, to configure a PVC that claims local persistent storage for Prometheus, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           spec:
             storageClassName: local-storage
             resources:
               requests:
                 storage: 40Gi

   Copy to Clipboard Toggle word wrap

   In the above example, the storage class created by the Local Storage Operator is called local-storage.

   To configure a PVC that claims local persistent storage for Alertmanager, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         volumeClaimTemplate:
           spec:
             storageClassName: local-storage
             resources:
               requests:
                 storage: 40Gi

   Copy to Clipboard Toggle word wrap
3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

2. Put your retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: <time_specification>

   Copy to Clipboard Toggle word wrap

   Substitute <time_specification> with a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years).

   For example, to configure retention time to be 24 hours, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: 24h

   Copy to Clipboard Toggle word wrap
3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

Procedure

1. Print the currently active Alertmanager configuration into file alertmanager.yaml:

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

   Copy to Clipboard Toggle word wrap

2. Change the configuration in file alertmanager.yaml to your new configuration:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - match:
         alertname: Watchdog
       repeat_interval: 5m
       receiver: watchdog
     - match:
         service: <your_service> 

   1

   
       routes:
       - match:
           <your_matching_rules> 

   2

   
         receiver: <receiver> 

   3

   
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
     <receiver_configuration>

   Copy to Clipboard Toggle word wrap

   1

   service specifies the service that fires the alerts.

   2

   <your_matching_rules> specify the target alerts.

   3

   receiver specifies the receiver to use for the alert.

   For example, this listing configures PagerDuty for notifications:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - match:
         alertname: Watchdog
       repeat_interval: 5m
       receiver: watchdog
     - match: service: example-app routes: - match: severity: critical receiver: team-frontend-page
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page pagerduty_configs: - service_key: "your-key"

   Copy to Clipboard Toggle word wrap

   With this configuration, alerts of critical severity fired by the example-app service are sent using the team-frontend-page receiver, which means that these alerts are paged to a chosen person.

3. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure the necessary port forwarding:

   $ oc -n openshift-monitoring port-forward svc/prometheus-operated 9090

   Copy to Clipboard Toggle word wrap

2. Fetch the JSON object containing acting alerting rules and their properties:

   $ curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")]'

   Copy to Clipboard Toggle word wrap

   Example output

   [
     {
       "name": "ClusterOperatorDown",
       "query": "cluster_operator_up{job=\"cluster-version-operator\"} == 0",
       "duration": 600,
       "labels": {
         "severity": "critical"
       },
       "annotations": {
         "message": "Cluster operator {{ $labels.name }} has not been available for 10 mins. Operator may be down or disabled, cluster will not be kept up to date and upgrades will not be possible."
       },
       "alerts": [],
       "health": "ok",
       "type": "alerting"
     },
     {
       "name": "ClusterOperatorDegraded",
       ...

   Copy to Clipboard Toggle word wrap