Configuring core platform monitoring

Procedure

1. In the OpenShift Container Platform web console, go to User Management → RoleBindings → Create binding.
2. In the Binding Type section, select the Namespace Role Binding type.
3. In the Name field, enter a name for the role binding.

4. In the Namespace field, select the project where you want to grant the access.

   Important

   The monitoring role or cluster role permissions that you grant to a user by using this procedure apply only to the project that you select in the Namespace field.

5. Select a monitoring role or cluster role from the Role Name list.
6. In the Subject section, select User.
7. In the Subject Name field, enter the name of the user.
8. Select Create to apply the role binding.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label>

   where:

   <node_name>

   Specifies the name of the node where you want to add the label.

   <node_label>

   Specifies the name of the wanted label.

2. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       <component>:
         nodeSelector:
           <node_label_1>
           <node_label_2>
       # ...

   where:

   <component>

   Specifies the monitoring stack component.

   <node_label_1>

   Specifies the label you added to the node.

   <node_label_2>

   Optional: Specifies additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusK8s:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosQuerier:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperator:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       metricsServer:
         resources:
           requests:
             cpu: 10m
             memory: 50Mi
           limits:
             cpu: 50m
             memory: 500Mi
       kubeStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       telemeterClient:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       openshiftStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       nodeExporter:
         resources:
           limits:
             cpu: 50m
             memory: 150Mi
           requests:
             cpu: 20m
             memory: 50Mi
       monitoringPlugin:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperatorAdmissionWebhook:
         resources:
           limits:
             cpu: 50m
             memory: 100Mi
           requests:
             cpu: 20m
             memory: 50Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add your PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         volumeClaimTemplate:
           spec:
             storageClassName: <storage_class>
             resources:
               requests:
                 storage: <amount_of_storage>

   where:

   <component>

   Specifies the monitoring component for which you want to configure the PVC.

   <storage_class>

   Specifies an existing storage class. If a storage class is not specified, the default storage class is used.

   <amount_of_storage>

   Specifies the amount of required storage.

   The following example configures a PVC that claims persistent storage for Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           spec:
             storageClassName: my-storage-class
             resources:
               requests:
                 storage: 40Gi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed and the new storage configuration is applied.

   Warning

   When you update the config map with a PVC configuration, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Manually expand a PVC with the updated storage request. For more information, see "Expanding persistent volume claims (PVCs) with a file system" in Expanding persistent volumes.

2. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Add a new storage size for the PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: <amount_of_storage>

   where:

   <component>

   Specifies the component for which you want to change the storage size.

   <amount_of_storage>

   Specifies the new size for the storage volume. It must be greater than the previous value.

   The following example sets the new PVC request to 100 gigabytes for the Prometheus instance:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: 100Gi

4. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

   Warning

   When you update the config map with a new storage size, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a secrets: section under data/config.yaml/alertmanagerMain with the following configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         secrets:
         - <secret_name_1>
         - <secret_name_2>

   where:

   secrets

   Defines the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.

   <secret_name_1>

   Specifies the name of the Secret object that contains authentication credentials for the receiver.

   <secret_name_2>

   Optional: Specifies additional secrets. If you add additional secrets, place each one on a new line.

   The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         secrets:
         - test-secret-basic-auth
         - test-secret-api-token

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Extract the currently active Alertmanager configuration from the alertmanager-main secret and save it as a local alertmanager.yaml file:

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Open the alertmanager.yaml file.

3. Edit the Alertmanager configuration:

   1. Optional: Change the default Alertmanager configuration.

      Example of the default Alertmanager secret YAML:

      global:
        resolve_timeout: 5m
        http_config:
          proxy_from_environment: true
      route:
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: default
        routes:
        - matchers:
          - "alertname=Watchdog"
          repeat_interval: 2m
          receiver: watchdog
      receivers:
      - name: default
      - name: watchdog

      where:

      proxy_from_environment

      Specifies whether to enable proxying for all alert receivers. If you configured an HTTP cluster-wide proxy, set this parameter to true to enable proxying for all alert receivers.

      group_wait

      Specifies how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.

      group_interval

      Specifies how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.

      repeat_interval

      Specifies the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

   2. Add your alert receiver configuration:

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: <receiver>
        <receiver_configuration>
      # ...

      where:

      <receiver>

      Specifies the name of the receiver.

      <receiver_configuration>

      Specifies the receiver configuration. The supported receivers are PagerDuty, webhook, email, Slack, and Microsoft Teams.

      Example of configuring PagerDuty as an alert receiver:

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: team-frontend-page
        pagerduty_configs:
        - routing_key: xxxxxxxxxx
          http_config:
            proxy_from_environment: true
            authorization:
              credentials: xxxxxxxxxx
      # ...

      where:

      routing_key

      Specifies the PagerDuty integration key.

      http_config

      (Optional) Adds the custom HTTP configuration for a specific receiver. That receiver does not inherit the global HTTP configuration settings.

      Example of configuring email as an alert receiver:

      # ...
      receivers:
      - name: default
      - name: watchdog
      - name: team-frontend-page
        email_configs:
          - to: myemail@example.com
            from: alertmanager@example.com
            smarthost: 'smtp.example.com:587'
            auth_username: alertmanager@example.com
            auth_password: password
            hello: alertmanager
      # ...

      where:

      to

      Specifies an email address to send notifications to.

      from

      Specifies an email address to send notifications from.

      smarthost

      Specifies the SMTP server address used for sending emails, including the port number.

      auth_username, auth_password

      Specify the authentication credentials that Alertmanager uses to connect to the SMTP server. This example uses username and password.

      hello

      Specifies the hostname to identify to the SMTP server. If you do not include this parameter, the hostname defaults to localhost.

      Important

      Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

      Warning

      Alertmanager requires an explicit TLS connection using STARTTLS. This results in the following behavior:

      • Connections that transmit data without encryption to remote SMTP servers (unencrypted SMTP) are not supported.
      • Implicit TLS connections are not supported, for example using smtps:// on port 465.
      • Only explicit TLS connections using STARTTLS are supported, for example using smtp:// on port 587 with STARTTLS enabled.

   3. Add the routing configuration:

      # ...
      route:
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: default
        routes:
        - matchers:
          - "alertname=Watchdog"
          repeat_interval: 2m
          receiver: watchdog
        - matchers:
          - "<your_matching_rules>"
          receiver: <receiver>
      # ...

      where:

      matchers

      Defines the matching rules that an alert has to fulfill to match the node. If you define inhibition rules, use target_matchers key name for target matchers and source_matchers key name for source matchers.

      <your_matching_rules>

      Specifies labels to match your alerts.

      <receiver>

      Specifies the name of the receiver to use for the alerts.

      Warning

      Do not use the match, match_re, target_match, target_match_re, source_match, and source_match_re key names, which are deprecated and planned for removal in a future release.

      Example of alert routing:

      # ...
      route:
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: default
        routes:
        # ...
        - matchers:
          - "service=example-app"
          routes:
          - matchers:
            - "severity=critical"
            receiver: team-frontend-page
      # ...

      • "service=example-app" matching rule matches alerts from the example-app service.
      • You can specify routes within other routes for more complex alert routing. In this example, route.routes.matchers.routes matches alerts of critical severity that are fired by the example-app service to the team-frontend-page receiver. Typically, these types of alerts are paged to an individual or a critical response team.

4. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

Procedure

1. In the OpenShift Container Platform web console, go to Administration → Cluster Settings → Configuration → Alertmanager.

   Note

   Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Click Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.

     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.

        Important

        Alertmanager requires an external SMTP server to send email alerts. To configure email alert receivers, ensure you have the necessary connection details for an external SMTP server.

        Warning

        Alertmanager requires an explicit TLS connection using STARTTLS. This results in the following behavior:

        • Connections that transmit data without encryption to remote SMTP servers (unencrypted SMTP) are not supported.
        • Implicit TLS connections are not supported, for example using smtps:// on port 465.
        • Only explicit TLS connections using STARTTLS are supported, for example using smtp:// on port 587 with STARTTLS enabled.
     3. Select whether TLS is required.
     4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

   1. Add routing label names and values in the Routing labels section of the form.
   2. Click Add label to add further routing labels.
6. Click Create to create the receiver.