Monitoring

Procedure

1. Edit the ConfigMap object.

   • To configure core OpenShift Container Platform monitoring components:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a persistent volume claim (PVC) for Prometheus. This relates to the Prometheus instance that monitors core OpenShift Container Platform components only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: fast
                  volumeMode: Filesystem
                  resources:
                    requests:
                      storage: 40Gi

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

   • To configure components that monitor user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a data retention period and minimum container resource requests for Prometheus. This relates to the Prometheus instance that monitors user-defined projects only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus: 1
              retention: 24h 2
              resources:
                requests:
                  cpu: 200m 3
                  memory: 2Gi 4

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

        2

        Configures a twenty-four hour data retention period for the Prometheus instance that monitors user-defined projects.

        3

        Defines a minimum resource request of 200 millicores for the Prometheus container.

        4

        Defines a minimum pod resource request of 2 GiB of memory for the Prometheus container.

        Note

        The Prometheus config map component is called prometheusK8s in the cluster-monitoring-config ConfigMap object and prometheus in the user-workload-monitoring-config ConfigMap object.

2. Save the file to apply the changes to the ConfigMap object. The pods affected by the new configuration are restarted automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To assign tolerations to a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

   • To assign tolerations to a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

2. Save the file to apply the changes. The new component placement configuration is applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a value for enforcedBodySizeLimit to data/config.yaml/prometheusK8s to limit the body size that can be accepted per target scrape:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       prometheusK8s:
         enforcedBodySizeLimit: 40MB 1

   1

   Specify the maximum body size for scraped metrics targets. This enforcedBodySizeLimit example limits the uncompressed size per target scrape to 40 megabytes. Valid numeric values use the Prometheus data size format: B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes). The default value is 0, which specifies no limit. You can also set the value to automatic to calculate the limit automatically based on cluster capacity.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to a cluster-monitoring-config config map, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s.

3. Add an endpoint URL and authentication credentials in this section:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 1
           <endpoint_authentication_credentials> 2

   1

   The URL of the remote write endpoint.

   2

   The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

4. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           <write_relabel_configs> 1

   1

   The write relabel configuration settings.

   For <write_relabel_configs> substitute a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   The following sample shows how to forward a single metric called my_metric:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__]
             regex: 'my_metric'
             action: keep

   See the Prometheus relabel_config documentation for information about write relabel configuration options.

5. Save the file to apply the changes to the ConfigMap object. The pods affected by the new configuration restart automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   Saving changes to a monitoring ConfigMap object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.

Procedure

1. Edit the ConfigMap object:

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        For example, to add metadata about the region and environment to all time series and alerts, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                region: eu
                environment: prod

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        Note

        In the openshift-user-workload-monitoring project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting externalLabels for prometheus in the user-workload-monitoring-config ConfigMap object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                region: eu
                environment: prod

2. Save the file to apply the changes. The new configuration is applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To set a log level for a component in the openshift-monitoring project:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

   • To set a log level for a component in the openshift-user-workload-monitoring project:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are prometheus, prometheusOperator, and thanosRuler.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods for the component restarts automatically when you apply the log-level change.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized loglevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a thanosQuerier section under data/config.yaml and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 1
         logLevel: <value> 2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes.

   Warning

   When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the thanos-querier pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the enableRequestLogging value to false in the config map.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Warning

   When changes are saved to the cluster-monitoring-config ConfigMap object, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also be restarted.

4. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Extract a token to connect to Prometheus:

   $ SECRET=`oc get secret -n openshift-user-workload-monitoring | grep  prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`

   $ TOKEN=`echo $(oc get secret $SECRET -n openshift-user-workload-monitoring -o json | jq -r '.data.token') | base64 -d`

2. Extract your route host:

   $ THANOS_QUERIER_HOST=`oc get route thanos-querier -n openshift-monitoring -o json | jq -r '.spec.host'`

3. Query the metrics of your own services in the command line. For example:

   $ NAMESPACE=ns1

   $ curl -X GET -kG "https://$THANOS_QUERIER_HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}" -H "Authorization: Bearer $TOKEN"

   The output will show you the duration that your application pods have been up.

   Example output

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","endpoint":"web","instance":"10.129.0.46:8080","job":"prometheus-example-app","namespace":"ns1","pod":"prometheus-example-app-68d47c4fb6-jztp2","service":"prometheus-example-app"},"value":[1591881154.748,"1"]}]}}

Procedure

1. Add the label to the project namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring=false'

2. To re-enable monitoring, remove the label from the namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring-'

   Note

   If there were any active monitoring targets for the project, it may take a few minutes for Prometheus to stop scraping them after adding the label.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   1. Set enableUserWorkload: to false under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: false

2. Save the file to apply the changes. Monitoring for user-defined projects is then disabled automatically.

3. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are terminated in the openshift-user-workload-monitoring project. This might take a short while:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   No resources found in openshift-user-workload-monitoring project.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserAlertmanagerConfig: true in the alertmanagerMain section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enableUserAlertmanagerConfig: true 1

   1

   Set the enableUserAlertmanagerConfig value to true to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

3. Save the file to apply the changes.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         enabled: true 1
         enableAlertmanagerConfig: true 2

   1

   Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.

   2

   Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

1. Open the OpenShift Container Platform web console and navigate to Observe → Targets.
2. Optional: Filter the targets by status and source by selecting filters in the Filter list.
3. Optional: Search for a target by name or label by using the Text or Label field next to the search box.
4. Optional: Sort the targets by clicking one or more of the Endpoint, Status, Namespace, Last Scrape, and Scrape Duration column headers.

5. Click the URL in the Endpoint column for a target to navigate to its Target details page. This page provides information about the target, including:

   • The endpoint URL being scraped for metrics
   • The current Up or Down status of the target
   • A link to the namespace
   • A link to the ServiceMonitor details
   • Labels attached to the target
   • The most recent time that the target was scraped for metrics

1. Open the OpenShift Container Platform web console and navigate to the Observe → Alerting → Alerts page.
2. Optional: Search for alerts by name using the Name field in the search list.
3. Optional: Filter alerts by state, severity, and source by selecting filters in the Filter list.
4. Optional: Sort the alerts by clicking one or more of the Name, Severity, State, and Source column headers.

5. Select the name of an alert to navigate to its Alert Details page. The page includes a graph that illustrates alert time series data. It also provides information about the alert, including:

   • A description of the alert
   • Messages associated with the alerts
   • Labels attached to the alert
   • A link to its governing alerting rule
   • Silences for the alert, if any exist

1. Navigate to the Observe → Alerting → Silences page.
2. Optional: Filter the silences by name using the Search by name field.
3. Optional: Filter silences by state by selecting filters in the Filter list. By default, Active and Pending filters are applied.
4. Optional: Sort the silences by clicking one or more of the Name, Firing Alerts, and State column headers.

5. Select the name of a silence to navigate to its Silence Details page. The page includes the following details:

   • Alert specification
   • Start time
   • End time
   • Silence state
   • Number and list of firing alerts

1. Navigate to the Observe → Alerting → Alerting Rules page.
2. Optional: Filter alerting rules by state, severity, and source by selecting filters in the Filter list.
3. Optional: Sort the alerting rules by clicking one or more of the Name, Severity, Alert State, and Source column headers.

4. Select the name of an alerting rule to navigate to its Alerting Rule Details page. The page provides the following details about the alerting rule:

   • Alerting rule name, severity, and description
   • The expression that defines the condition for firing the alert
   • The time for which the condition should be true for an alert to fire
   • A graph for each alert governed by the alerting rule, showing the value with which the alert is firing
   • A table of all alerts governed by the alerting rule

1. Navigate to the Observe → <project_name> → Alerts page.

2. View details for an alert, silence, or an alerting rule:

   • Alert Details can be viewed by selecting > to the left of an alert name and then selecting the alert in the list.

   • Silence Details can be viewed by selecting a silence in the Silenced By section of the Alert Details page. The Silence Details page includes the following information:

     • Alert specification
     • Start time
     • End time
     • Silence state
     • Number and list of firing alerts
   • Alerting Rule Details can be viewed by selecting View Alerting Rule in the menu on the right of an alert in the Alerts page.

1. Print the currently active Alertmanager configuration into file alertmanager.yaml:

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s 1
     group_interval: 5m 2
     repeat_interval: 12h 3
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=<your_service>" 4
       routes:
       - matchers:
         - <your_matching_rules> 5
         receiver: <receiver> 6
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
   #  <receiver_configuration>

   1

   The group_wait value specifies how long Alertmanager waits before sending an initial notification for a group of alerts. This value controls how long Alertmanager waits while collecting initial alerts for the same group before sending a notification.

   2

   The group_interval value specifies how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.

   3

   The repeat_interval value specifies the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. However, the repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

   4

   The service value specifies the service that fires the alerts.

   5

   The <your_matching_rules> value specifies the target alerts.

   6

   The receiver value specifies the receiver to use for the alert.

   Note

   Use the matchers key name to indicate the matchers that an alert has to fulfill to match the node. Do not use the match or match_re key names, which are both deprecated and planned for removal in a future release.

   In addition, if you define inhibition rules, use the target_matchers key name to indicate the target matchers and the source_matchers key name to indicate the source matchers. Do not use the target_match, target_match_re, source_match, or source_match_re key names, which are deprecated and planned for removal in a future release.

   The following Alertmanager configuration example configures PagerDuty as an alert receiver:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=example-app"
       routes:
       - matchers:
         - "severity=critical"
         receiver: team-frontend-page*
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page
     pagerduty_configs:
     - service_key: "_your-key_"

   With this configuration, alerts of critical severity that are fired by the example-app service are sent using the team-frontend-page receiver. Typically these types of alerts would be paged to an individual or a critical response team.

3. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

1. Navigate to the Administration → Cluster Settings → Configuration → Alertmanager → YAML page of the web console.
2. Modify the YAML configuration file.
3. Select Save.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 1
       receiver: <receiver> 2
   receivers:
   - name: Default
   - name: <receiver>
   #  <receiver_configuration>

   1

   Specifies which alerts match the route. This example shows all alerts that have the service="prometheus-example-monitor" label.

   2

   Specifies the receiver to use for the alerts group.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboards.
2. Choose a dashboard in the Dashboard list. Some dashboards, such as etcd and Prometheus dashboards, produce additional sub-menus when selected.

3. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
4. Optional: Select a Refresh Interval.
5. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. In the Developer perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboard.
2. Select a project from the Project: drop-down list.

3. Select a dashboard from the Dashboard drop-down list to see the filtered metrics.

   Note

   All dashboards produce additional sub-menus when selected, except Kubernetes / Compute Resources / Namespace (Pods).

4. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
5. Optional: Select a Refresh Interval.
6. Hover over each of the graphs within a dashboard to display detailed information about specific items.

1. Add the Helm repository:

   $ helm repo add rh-ecosystem-edge https://rh-ecosystem-edge.github.io/console-plugin-nvidia-gpu

   $ helm repo update

2. Install the Helm chart in the default NVIDIA GPU operator namespace:

   $ helm install -n nvidia-gpu-operator console-plugin-nvidia-gpu rh-ecosystem-edge/console-plugin-nvidia-gpu

   Example output

   NAME: console-plugin-nvidia-gpu
   LAST DEPLOYED: Tue Aug 23 15:37:35 2022
   NAMESPACE: nvidia-gpu-operator
   STATUS: deployed
   REVISION: 1
   NOTES:
   View the Console Plugin NVIDIA GPU deployed resources by running the following command:
   
   $ oc -n {{ .Release.Namespace }} get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu
   
   Enable the plugin by running the following command:
   
   # Check if a plugins field is specified
   $ oc get consoles.operator.openshift.io cluster --output=jsonpath="{.spec.plugins}"
   
   # if not, then run the following command to enable the plugin
   $ oc patch consoles.operator.openshift.io cluster --patch '{ "spec": { "plugins": ["console-plugin-nvidia-gpu"] } }' --type=merge
   
   # if yes, then run the following command to enable the plugin
   $ oc patch consoles.operator.openshift.io cluster --patch '[{"op": "add", "path": "/spec/plugins/-", "value": "console-plugin-nvidia-gpu" }]' --type=json
   
   # add the required DCGM Exporter metrics ConfigMap to the existing NVIDIA operator ClusterPolicy CR:
   oc patch clusterpolicies.nvidia.com gpu-cluster-policy --patch '{ "spec": { "dcgmExporter": { "config": { "name": "console-plugin-nvidia-gpu" } } } }' --type=merge

   The dashboard relies mostly on Prometheus metrics exposed by the NVIDIA DCGM Exporter, but the default exposed metrics are not enough for the dashboard to render the required gauges. Therefore, the DGCM exporter is configured to expose a custom set of metrics, as shown here.

   apiVersion: v1
   data:
     dcgm-metrics.csv: |
       DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, gpu utilization.
       DCGM_FI_DEV_MEM_COPY_UTIL, gauge, mem utilization.
       DCGM_FI_DEV_ENC_UTIL, gauge, enc utilization.
       DCGM_FI_DEV_DEC_UTIL, gauge, dec utilization.
       DCGM_FI_DEV_POWER_USAGE, gauge, power usage.
       DCGM_FI_DEV_POWER_MGMT_LIMIT_MAX, gauge, power mgmt limit.
       DCGM_FI_DEV_GPU_TEMP, gauge, gpu temp.
       DCGM_FI_DEV_SM_CLOCK, gauge, sm clock.
       DCGM_FI_DEV_MAX_SM_CLOCK, gauge, max sm clock.
       DCGM_FI_DEV_MEM_CLOCK, gauge, mem clock.
       DCGM_FI_DEV_MAX_MEM_CLOCK, gauge, max mem clock.
   kind: ConfigMap
   metadata:
     annotations:
       meta.helm.sh/release-name: console-plugin-nvidia-gpu
       meta.helm.sh/release-namespace: nvidia-gpu-operator
     creationTimestamp: "2022-10-26T19:46:41Z"
     labels:
       app.kubernetes.io/component: console-plugin-nvidia-gpu
       app.kubernetes.io/instance: console-plugin-nvidia-gpu
       app.kubernetes.io/managed-by: Helm
       app.kubernetes.io/name: console-plugin-nvidia-gpu
       app.kubernetes.io/part-of: console-plugin-nvidia-gpu
       app.kubernetes.io/version: latest
       helm.sh/chart: console-plugin-nvidia-gpu-0.2.3
     name: console-plugin-nvidia-gpu
     namespace: nvidia-gpu-operator
     resourceVersion: "19096623"
     uid: 96cdf700-dd27-437b-897d-5cbb1c255068

   Install the ConfigMap and edit the NVIDIA Operator ClusterPolicy CR to add that ConfigMap in the DCGM exporter configuration. The installation of the ConfigMap is done by the new version of the Console Plugin NVIDIA GPU Helm Chart, but the ClusterPolicy CR editing is done by the user.

3. View the deployed resources:

   $ oc -n nvidia-gpu-operator get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/console-plugin-nvidia-gpu-7dc9cfb5df-ztksx   1/1     Running   0          2m6s
   
   NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/console-plugin-nvidia-gpu   ClusterIP   172.30.240.138   <none>        9443/TCP   2m6s
   
   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/console-plugin-nvidia-gpu   1/1     1            1           2m6s
   
   NAME                                                   DESIRED   CURRENT   READY   AGE
   replicaset.apps/console-plugin-nvidia-gpu-7dc9cfb5df   1         1         1       2m6s

Procedure

1. Retrieve the bearer token:

   $ token=`oc whoami -t`

2. Query metrics from the /federate route. The following example queries up metrics:

   $ curl -G -s -k -H "Authorization: Bearer $token" \
       'https:/<federation_host>/federate' \ 1
       --data-urlencode 'match[]=up'

   1

   For <federation_host>, substitute the host URL for the federation route.

   Example output

   # TYPE up untyped
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
   ...

Procedure

1. Check that the corresponding labels match in the service and ServiceMonitor resource configurations.

   1. Obtain the label defined in the service. The following example queries the prometheus-example-app service in the ns1 project:

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Example output

        labels:
          app: prometheus-example-app

   2. Check that the matchLabels app label in the ServiceMonitor resource configuration matches the label output in the preceding step:

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Example output

      apiVersion: v1
      kind: Service
      # ...
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app
      # ...

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

2. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

3. Review the target status for your endpoint on the Metrics targets page in the OpenShift Container Platform web console UI.

   1. Log in to the OpenShift Container Platform web console and navigate to Observe → Targets in the Administrator perspective.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

4. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

   3. Save the file to apply the changes.

      Note

      The prometheus-operator in the openshift-user-workload-monitoring project restarts automatically when you apply the log-level change.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Example output

              - --log-level=debug

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Administrator perspective, navigate to Observe → Metrics.

2. Run the following Prometheus Query Language (PromQL) query in the Expression field. This returns the ten metrics that have the highest number of scrape samples:

   topk(10,count by (job)({__name__=~".+"}))

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts.

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core OpenShift Container Platform project, create a Red Hat support case on the Red Hat Customer Portal.

4. Review the TSDB status using the Prometheus HTTP API by running the following commands as a cluster administrator:

   $ oc login -u <username> -p <password>

   $ host=$(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath={.spec.host})

   $ token=$(oc whoami -t)

   $ curl -H "Authorization: Bearer $token" -k "https://$host/api/v1/status/tsdb"

   Example output

   "status": "success",