Chapter 3. Configuring core platform monitoring

Procedure

1. In the OpenShift Container Platform web console, go to User Management RoleBindings Create binding.
2. In the Binding Type section, select the Namespace Role Binding type.
3. In the Name field, enter a name for the role binding.

4. In the Namespace field, select the project where you want to grant the access.

   Important

   The monitoring role or cluster role permissions that you grant to a user by using this procedure apply only to the project that you select in the Namespace field.

5. Select a monitoring role or cluster role from the Role Name list.
6. In the Subject section, select User.
7. In the Subject Name field, enter the name of the user.
8. Select Create to apply the role binding.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label> 

   1

   Copy to Clipboard

   1

   Replace <node_name> with the name of the node where you want to add the label. Replace <node_label> with the name of the wanted label.

2. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       <component>: 

   1

   
         nodeSelector:
           <node_label_1> 

   2

   
           <node_label_2> 

   3

   
       # ...

   Copy to Clipboard

   1

   Substitute <component> with the appropriate monitoring stack component name.

   2

   Substitute <node_label_1> with the label you added to the node.

   3

   Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Copy to Clipboard

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

   Copy to Clipboard
3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   Example of setting resource limits and requests

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusK8s:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosQuerier:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperator:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       metricsServer:
         resources:
           requests:
             cpu: 10m
             memory: 50Mi
           limits:
             cpu: 50m
             memory: 500Mi
       kubeStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       telemeterClient:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       openshiftStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       nodeExporter:
         resources:
           limits:
             cpu: 50m
             memory: 150Mi
           requests:
             cpu: 20m
             memory: 50Mi
       monitoringPlugin:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperatorAdmissionWebhook:
         resources:
           limits:
             cpu: 50m
             memory: 100Mi
           requests:
             cpu: 20m
             memory: 50Mi

   Copy to Clipboard

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

2. Add your PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>: 

   1

   
         volumeClaimTemplate:
           spec:
             storageClassName: <storage_class> 

   2

   
             resources:
               requests:
                 storage: <amount_of_storage> 

   3

   Copy to Clipboard

   1

   Specify the monitoring component for which you want to configure the PVC.

   2

   Specify an existing storage class. If a storage class is not specified, the default storage class is used.

   3

   Specify the amount of required storage.

   The following example configures a PVC that claims persistent storage for Prometheus:

   Example PVC configuration

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           spec:
             storageClassName: my-storage-class
             resources:
               requests:
                 storage: 40Gi

   Copy to Clipboard

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed and the new storage configuration is applied.

   Warning

   When you update the config map with a PVC configuration, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Manually expand a PVC with the updated storage request. For more information, see "Expanding persistent volume claims (PVCs) with a file system" in Expanding persistent volumes.

2. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

3. Add a new storage size for the PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>: 

   1

   
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: <amount_of_storage> 

   2

   Copy to Clipboard

   1

   The component for which you want to change the storage size.

   2

   Specify the new size for the storage volume. It must be greater than the previous value.

   The following example sets the new PVC request to 100 gigabytes for the Prometheus instance:

   Example storage configuration for prometheusK8s

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: 100Gi

   Copy to Clipboard

4. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

   Warning

   When you update the config map with a new storage size, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

   Copy to Clipboard
3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard

2. Add a secrets: section under data/config.yaml/alertmanagerMain with the following configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         secrets: 

   1

   
         - <secret_name_1> 

   2

   
         - <secret_name_2>

   Copy to Clipboard

   1

   This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.

   2

   The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

   The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         secrets:
         - test-secret-basic-auth
         - test-secret-api-token

   Copy to Clipboard
3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Extract the currently active Alertmanager configuration from the alertmanager-main secret and save it as a local alertmanager.yaml file:

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

   Copy to Clipboard
2. Open the alertmanager.yaml file.

3. Edit the Alertmanager configuration:

   1. Optional: Change the default Alertmanager configuration:

      Example of the default Alertmanager secret YAML

      global:
        resolve_timeout: 5m
        http_config:
          proxy_from_environment: true 

      1

      
      route:
        group_wait: 30s 

      2

      
        group_interval: 5m 

      3

      
        repeat_interval: 12h 

      4

      
        receiver: default
        routes:
        - matchers:
          - "alertname=Watchdog"
          repeat_interval: 2m
          receiver: watchdog
      receivers:
      - name: default
      - name: watchdog

      Copy to Clipboard

      1

      If you configured an HTTP cluster-wide proxy, set the proxy_from_environment parameter to true to enable proxying for all alert receivers.

      2

      Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.

      3

      Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.

      4

      Specify the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

   2. Add your alert receiver configuration:

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: <receiver> 

      1

      
        <receiver_configuration> 

      2

      
      # ...

      Copy to Clipboard

      1

      The name of the receiver.

      2

      The receiver configuration. The supported receivers are PagerDuty, webhook, email, Slack, and Microsoft Teams.

      Example of configuring PagerDuty as an alert receiver

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: team-frontend-page
        pagerduty_configs:
        - routing_key: xxxxxxxxxx 

      1

      
          http_config: 

      2

      
            proxy_from_environment: true
            authorization:
              credentials: xxxxxxxxxx
      # ...

      Copy to Clipboard

      1

      Defines the PagerDuty integration key.

      2

      Optional: Add the custom HTTP configuration for a specific receiver. That receiver does not inherit the global HTTP configuration settings.

      Example of configuring email as an alert receiver

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: team-frontend-page
        email_configs:
          - to: myemail@example.com 

      1

      
            from: alertmanager@example.com 

      2

      
            smarthost: 'smtp.example.com:587' 

      3

      
            auth_username: alertmanager@example.com  

      4

      
            auth_password: password
            hello: alertmanager 

      5

      
      # ...

      Copy to Clipboard

      1

      Specify an email address to send notifications to.

      2

      Specify an email address to send notifications from.

      3

      Specify the SMTP server address used for sending emails, including the port number.

      4

      Specify the authentication credentials that Alertmanager uses to connect to the SMTP server. This example uses username and password.

      5

      Specify the hostname to identify to the SMTP server. If you do not include this parameter, the hostname defaults to localhost.

      Important

      Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

   3. Add the routing configuration:

      # ...
      route:
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: default
        routes:
        - matchers:
          - "alertname=Watchdog"
          repeat_interval: 2m
          receiver: watchdog
        - matchers: 

      1

      
          - "<your_matching_rules>" 

      2

      
          receiver: <receiver> 

      3

      
      # ...

      Copy to Clipboard

      1

      Use the matchers key name to specify the matching rules that an alert has to fulfill to match the node. If you define inhibition rules, use target_matchers key name for target matchers and source_matchers key name for source matchers.

      2

      Specify labels to match your alerts.

      3

      Specify the name of the receiver to use for the alerts.

      Warning

      Do not use the match, match_re, target_match, target_match_re, source_match, and source_match_re key names, which are deprecated and planned for removal in a future release.

      Example of alert routing

      # ...
      route:
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: default
        routes:
        - matchers:
          - "alertname=Watchdog"
          repeat_interval: 2m
          receiver: watchdog
        - matchers: 

      1

      
          - "service=example-app"
          routes: 

      2

      
          - matchers:
            - "severity=critical"
            receiver: team-frontend-page
      # ...

      Copy to Clipboard

      1

      This example matches alerts from the example-app service.

      2

      You can create routes within other routes for more complex alert routing.

      The previous example routes alerts of critical severity that are fired by the example-app service to the team-frontend-page receiver. Typically, these types of alerts are paged to an individual or a critical response team.

4. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

   Copy to Clipboard

5. Verify your routing configuration by visualizing the routing tree:

   $ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool config routes show --alertmanager.url http://localhost:9093

   Copy to Clipboard

   Example output

   Routing tree:
   .
   └── default-route  receiver: default
       ├── {alertname="Watchdog"}  receiver: Watchdog
       └── {service="example-app"}  receiver: default
           └── {severity="critical"}  receiver: team-frontend-page

   Copy to Clipboard

Procedure

1. In the OpenShift Container Platform web console, go to Administration Cluster Settings Configuration Alertmanager.

   Note

   Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Click Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.

     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.

        Important

        Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

     3. Select whether TLS is required.
     4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

   1. Add routing label names and values in the Routing labels section of the form.
   2. Click Add label to add further routing labels.
6. Click Create to create the receiver.