Monitoring

Procedure

1. Edit the ConfigMap object.

   • To configure core OpenShift Container Platform monitoring components:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a persistent volume claim (PVC) for Prometheus. This relates to the Prometheus instance that monitors core OpenShift Container Platform components only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s: 1
              volumeClaimTemplate:
               spec:
                 storageClassName: fast
                 volumeMode: Filesystem
                 resources:
                   requests:
                     storage: 40Gi

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

   • To configure components that monitor user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a data retention period and minimum container resource requests for Prometheus. This relates to the Prometheus instance that monitors user-defined projects only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus: 1
              retention: 24h 2
              resources:
                requests:
                  cpu: 200m 3
                  memory: 2Gi 4

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

        2

        Configures a twenty-four hour data retention period for the Prometheus instance that monitors user-defined projects.

        3

        Defines a minimum resource request of 200 millicores for the Prometheus container.

        4

        Defines a minimum pod resource request of 2 GiB of memory for the Prometheus container.

        Note

        The Prometheus config map component is called prometheusK8s in the cluster-monitoring-config ConfigMap object and prometheus in the user-workload-monitoring-config ConfigMap object.

2. Save the file to apply the changes to the ConfigMap object. The pods affected by the new configuration are restarted automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To move a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              nodeSelector:
                <node_key>: <node_value>
                <node_key>: <node_value>
                <...>

        Substitute <component> accordingly and substitute <node_key>: <node_value> with the map of key-value pairs that specifies a group of destination nodes. Often, only a single key-value pair is used.

        The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.

        Important

        Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod logs for errors relating to taints and tolerations.

        For example, to move monitoring components for core OpenShift Container Platform projects to specific nodes that are labeled nodename: controlplane1, nodename: worker1, nodename: worker2, and nodename: worker2, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusOperator:
              nodeSelector:
                nodename: controlplane1
            prometheusK8s:
              nodeSelector:
                nodename: worker1
                nodename: worker2
            alertmanagerMain:
              nodeSelector:
                nodename: worker1
                nodename: worker2
            kubeStateMetrics:
              nodeSelector:
                nodename: worker1
            grafana:
              nodeSelector:
                nodename: worker1
            telemeterClient:
              nodeSelector:
                nodename: worker1
            k8sPrometheusAdapter:
              nodeSelector:
                nodename: worker1
                nodename: worker2
            openshiftStateMetrics:
              nodeSelector:
                nodename: worker1
            thanosQuerier:
              nodeSelector:
                nodename: worker1
                nodename: worker2

   • To move a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              nodeSelector:
                <node_key>: <node_value>
                <node_key>: <node_value>
                <...>

        Substitute <component> accordingly and substitute <node_key>: <node_value> with the map of key-value pairs that specifies the destination nodes. Often, only a single key-value pair is used.

        The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.

        Important

        Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod logs for errors relating to taints and tolerations.

        For example, to move monitoring components for user-defined projects to specific worker nodes labeled nodename: worker1, nodename: worker2, and nodename: worker2, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheusOperator:
              nodeSelector:
                nodename: worker1
            prometheus:
              nodeSelector:
                nodename: worker1
                nodename: worker2
            thanosRuler:
              nodeSelector:
                nodename: worker1
                nodename: worker2

2. Save the file to apply the changes. The components affected by the new configuration are moved to the new nodes automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To assign tolerations to a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

   • To assign tolerations to a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

2. Save the file to apply the changes. The new component placement configuration is applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s.

3. Add an endpoint URL and authentication credentials in this section:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write.endpoint"
           <endpoint_authentication_credentials>

   For endpoint_authentication_credentials substitute the credentials for the endpoint. Currently supported authentication methods are basic authentication (basicAuth) and client TLS (tlsConfig) authentication.

   • The following example configures basic authentication:

     basicAuth:
       username:
         <usernameSecret>
       password:
         <passwordSecret>

     Substitute <usernameSecret> and <passwordSecret> accordingly.

     The following sample shows basic authentication configured with remoteWriteAuth for the name values and user and password for the key values. These values contain the endpoint authentication credentials:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: cluster-monitoring-config
       namespace: openshift-monitoring
     data:
       config.yaml: |
         prometheusK8s:
           remoteWrite:
           - url: "https://remote-write.endpoint"
             basicAuth:
               username:
                 name: remoteWriteAuth
                 key: user
               password:
                 name: remoteWriteAuth
                 key: password

   • The following example configures client TLS authentication:

     tlsConfig:
       ca:
         <caSecret>
       cert:
         <certSecret>
       keySecret:
         <keySecret>

     Substitute <caSecret>, <certSecret>, and <keySecret> accordingly.

     The following sample shows a TLS authentication configuration using selfsigned-mtls-bundle for the name values and ca.crt for the ca key value, client.crt for the cert key value, and client.key for the keySecret key value:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: cluster-monitoring-config
       namespace: openshift-monitoring
     data:
       config.yaml: |
         prometheusK8s:
           remoteWrite:
           - url: "https://remote-write.endpoint"
             tlsConfig:
               ca:
                 secret:
                   name: selfsigned-mtls-bundle
                   key: ca.crt
               cert:
                 secret:
                   name: selfsigned-mtls-bundle
                   key: client.crt
               keySecret:
                 name: selfsigned-mtls-bundle
                 key: client.key

4. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write.endpoint"
           <endpoint_authentication_credentials>
           <write_relabel_configs>

   For <write_relabel_configs> substitute a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   The following sample shows how to forward a single metric called my_metric:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write.endpoint"
           writeRelabelConfigs:
           - sourceLabels: [__name__]
             regex: 'my_metric'
             action: keep

   See the Prometheus relabel_config documentation for information about write relabel configuration options.

5. If required, configure remote write for the Prometheus instance that monitors user-defined projects by changing the name and namespace metadata values as follows:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         remoteWrite:
         - url: "https://remote-write.endpoint"
           <endpoint_authentication_credentials>
           <write_relabel_configs>

   Note

   The Prometheus config map component is called prometheusK8s in the cluster-monitoring-config ConfigMap object and prometheus in the user-workload-monitoring-config ConfigMap object.

6. Save the file to apply the changes to the ConfigMap object. The pods affected by the new configuration restart automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   Saving changes to a monitoring ConfigMap object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.

Procedure

1. Edit the ConfigMap object:

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        For example, to add metadata about the region and environment to all time series and alerts, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                region: eu
                environment: prod

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        Note

        In the openshift-user-workload-monitoring project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting externalLabels for prometheus in the user-workload-monitoring-config ConfigMap object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                region: eu
                environment: prod

2. Save the file to apply the changes. The new configuration is applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To set a log level for a component in the openshift-monitoring project:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

   • To set a log level for a component in the openshift-user-workload-monitoring project:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are prometheus, prometheusOperator, and thanosRuler.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods for the component restarts automatically when you apply the log-level change.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized loglevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the grafana component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       grafana:
         enabled: false

3. Save the file to apply the changes. The resources will begin to be removed automatically when you apply the change.

   Warning

   This change results in some components, including Prometheus and the Thanos Querier, being restarted. This might lead to previously collected metrics being lost if you have not yet followed the steps in the "Configuring persistent storage" section.

4. Check that the Grafana pod is no longer running. The following example lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

   Note

   It may take a few minutes after applying the change for these pods to terminate.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Warning

   When changes are saved to the cluster-monitoring-config ConfigMap object, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also be restarted.

4. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Extract a token to connect to Prometheus:

   $ SECRET=`oc get secret -n openshift-user-workload-monitoring | grep  prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`

   $ TOKEN=`echo $(oc get secret $SECRET -n openshift-user-workload-monitoring -o json | jq -r '.data.token') | base64 -d`

2. Extract your route host:

   $ THANOS_QUERIER_HOST=`oc get route thanos-querier -n openshift-monitoring -o json | jq -r '.spec.host'`

3. Query the metrics of your own services in the command line. For example:

   $ NAMESPACE=ns1

   $ curl -X GET -kG "https://$THANOS_QUERIER_HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}" -H "Authorization: Bearer $TOKEN"

   The output will show you the duration that your application pods have been up.

   Example output

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","endpoint":"web","instance":"10.129.0.46:8080","job":"prometheus-example-app","namespace":"ns1","pod":"prometheus-example-app-68d47c4fb6-jztp2","service":"prometheus-example-app"},"value":[1591881154.748,"1"]}]}}

Procedure

1. Add the label to the project namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring=false'

2. To re-enable monitoring, remove the label from the namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring-'

   Note

   If there were any active monitoring targets for the project, it may take a few minutes for Prometheus to stop scraping them after adding the label.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   1. Set enableUserWorkload: to false under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: false

2. Save the file to apply the changes. Monitoring for user-defined projects is then disabled automatically.

3. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are terminated in the openshift-user-workload-monitoring project. This might take a short while:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   No resources found in openshift-user-workload-monitoring project.

1. Open the OpenShift Container Platform web console and navigate to the Observe → Alerting → Alerts page.
2. Optional: Search for alerts by name using the Name field in the search list.
3. Optional: Filter alerts by state, severity, and source by selecting filters in the Filter list.
4. Optional: Sort the alerts by clicking one or more of the Name, Severity, State, and Source column headers.

5. Select the name of an alert to navigate to its Alert Details page. The page includes a graph that illustrates alert time series data. It also provides information about the alert, including:

   • A description of the alert
   • Messages associated with the alerts
   • Labels attached to the alert
   • A link to its governing alerting rule
   • Silences for the alert, if any exist

1. Navigate to the Observe → Alerting → Silences page.
2. Optional: Filter the silences by name using the Search by name field.
3. Optional: Filter silences by state by selecting filters in the Filter list. By default, Active and Pending filters are applied.
4. Optional: Sort the silences by clicking one or more of the Name, Firing Alerts, and State column headers.

5. Select the name of a silence to navigate to its Silence Details page. The page includes the following details:

   • Alert specification
   • Start time
   • End time
   • Silence state
   • Number and list of firing alerts

1. Navigate to the Observe → Alerting → Alerting Rules page.
2. Optional: Filter alerting rules by state, severity, and source by selecting filters in the Filter list.
3. Optional: Sort the alerting rules by clicking one or more of the Name, Severity, Alert State, and Source column headers.

4. Select the name of an alerting rule to navigate to its Alerting Rule Details page. The page provides the following details about the alerting rule:

   • Alerting rule name, severity, and description
   • The expression that defines the condition for firing the alert
   • The time for which the condition should be true for an alert to fire
   • A graph for each alert governed by the alerting rule, showing the value with which the alert is firing
   • A table of all alerts governed by the alerting rule

1. Navigate to the Observe → <project_name> → Alerts page.

2. View details for an alert, silence, or an alerting rule:

   • Alert Details can be viewed by selecting > to the left of an alert name and then selecting the alert in the list.

   • Silence Details can be viewed by selecting a silence in the Silenced By section of the Alert Details page. The Silence Details page includes the following information:

     • Alert specification
     • Start time
     • End time
     • Silence state
     • Number and list of firing alerts
   • Alerting Rule Details can be viewed by selecting View Alerting Rule in the menu on the right of an alert in the Alerts page.

1. Print the currently active Alertmanager configuration into file alertmanager.yaml:

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 5m
       receiver: watchdog
     - matchers:
       - "service=<your_service>" 1
       routes:
       - matchers:
         - <your_matching_rules> 2
         receiver: <receiver> 3
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
   #  <receiver_configuration>

   1

   service specifies the service that fires the alerts.

   2

   <your_matching_rules> specifies the target alerts.

   3

   receiver specifies the receiver to use for the alert.

   Note

   Use the matchers key name to indicate the matchers that an alert has to fulfill to match the node. Do not use the match or match_re key names, which are both deprecated and planned for removal in a future release.

   In addition, if you define inhibition rules, use the target_matchers key name to indicate the target matchers and the source_matchers key name to indicate the source matchers. Do not use the target_match, target_match_re, source_match, or source_match_re key names, which are deprecated and planned for removal in a future release.

   The following Alertmanager configuration example configures PagerDuty as an alert receiver:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 5m
       receiver: watchdog
     - matchers: - "service=example-app" routes: - matchers: - "severity=critical" receiver: team-frontend-page
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page pagerduty_configs: - service_key: "your-key"

   With this configuration, alerts of critical severity that are fired by the example-app service are sent using the team-frontend-page receiver. Typically these types of alerts would be paged to an individual or a critical response team.

3. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

1. Navigate to the Administration → Cluster Settings → Configuration → Alertmanager → YAML page of the web console.
2. Modify the YAML configuration file.
3. Select Save.

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboards.
2. Choose a dashboard in the Dashboard list. Some dashboards, such as etcd and Prometheus dashboards, produce additional sub-menus when selected.

3. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
4. Optional: Select a Refresh Interval.
5. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. In the Developer perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboard.
2. Select a project from the Project: drop-down list.

3. Select a dashboard from the Dashboard drop-down list to see the filtered metrics.

   Note

   All dashboards produce additional sub-menus when selected, except Kubernetes / Compute Resources / Namespace (Pods).

4. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
5. Optional: Select a Refresh Interval.
6. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. In the Administrator perspective, navigate to Networking → Routes.

   Note

   Access to the third-party Alertmanager, Grafana, Prometheus, and Thanos Querier UIs is not available from the Developer perspective. Instead, use the Metrics UI link in the Developer perspective, which includes some predefined CPU, memory, bandwidth, and network packet queries for the selected project.

2. Select the openshift-monitoring project in the Project list.

3. Access a third-party monitoring UI:

   • Select the URL in the alertmanager-main row to open the login page for the Alertmanager UI.
   • Select the URL in the grafana row to open the login page for the Grafana UI.
   • Select the URL in the prometheus-k8s row to open the login page for the Prometheus UI.
   • Select the URL in the thanos-querier row to open the login page for the Thanos Querier UI.
4. Choose Log in with OpenShift to log in using your OpenShift Container Platform credentials.

Procedure

1. Run the following to list routes for the openshift-monitoring project:

   $ oc -n openshift-monitoring get routes

   Example output

   NAME                HOST/PORT                                                     ...
   alertmanager-main   alertmanager-main-openshift-monitoring.apps._url_.openshift.com ...
   grafana             grafana-openshift-monitoring.apps._url_.openshift.com           ...
   prometheus-k8s      prometheus-k8s-openshift-monitoring.apps._url_.openshift.com    ...
   thanos-querier      thanos-querier-openshift-monitoring.apps._url_.openshift.com    ...

2. Navigate to a HOST/PORT route by using a web browser.
3. Select Log in with OpenShift to log in using your OpenShift Container Platform credentials.

Procedure

1. Check that the corresponding labels match in the service and ServiceMonitor resource configurations.

   1. Obtain the label defined in the service. The following example queries the prometheus-example-app service in the ns1 project:

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Example output

        labels:
          app: prometheus-example-app

   2. Check that the matchLabels app label in the ServiceMonitor resource configuration matches the label output in the preceding step:

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Example output

      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

2. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

3. Review the target status for your project in the Prometheus UI directly.

   1. Establish port-forwarding to the Prometheus instance in the openshift-user-workload-monitoring project:

      $ oc port-forward -n openshift-user-workload-monitoring pod/prometheus-user-workload-0 9090

   2. Open http://localhost:9090/targets in a web browser and review the status of the target for your project directly in the Prometheus UI. Check for error messages relating to the target.

4. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug

   3. Save the file to apply the changes.

      Note

      The prometheus-operator in the openshift-user-workload-monitoring project restarts automatically when you apply the log-level change.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Example output

              - --log-level=debug

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Administrator perspective, navigate to Observe → Metrics.

2. Run the following Prometheus Query Language (PromQL) query in the Expression field. This returns the ten metrics that have the highest number of scrape samples:

   topk(10,count by (job)({__name__=~".+"}))

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts.

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core OpenShift Container Platform project, create a Red Hat support case on the Red Hat Customer Portal.

4. Check the TSDB status in the Prometheus UI.

   1. In the Administrator perspective, navigate to Networking → Routes.
   2. Select the openshift-monitoring project in the Project list.
   3. Select the URL in the prometheus-k8s row to open the login page for the Prometheus UI.
   4. Choose Log in with OpenShift to log in using your OpenShift Container Platform credentials.
   5. In the Prometheus UI, navigate to Status → TSDB Status.