Monitoring

Procedure

1. Edit the ConfigMap object.

   • To configure core OpenShift Container Platform monitoring components:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a persistent volume claim (PVC) for Prometheus. This relates to the Prometheus instance that monitors core OpenShift Container Platform components only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: fast
                  volumeMode: Filesystem
                  resources:
                    requests:
                      storage: 40Gi

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

   • To configure components that monitor user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your configuration under data/config.yaml as a key-value pair <component_name>: <component_configuration>:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              <configuration_for_the_component>

        Substitute <component> and <configuration_for_the_component> accordingly.

        The following example ConfigMap object configures a data retention period and minimum container resource requests for Prometheus. This relates to the Prometheus instance that monitors user-defined projects only:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus: 1
              retention: 24h 2
              resources:
                requests:
                  cpu: 200m 3
                  memory: 2Gi 4

        1

        Defines the Prometheus component and the subsequent lines define its configuration.

        2

        Configures a twenty-four hour data retention period for the Prometheus instance that monitors user-defined projects.

        3

        Defines a minimum resource request of 200 millicores for the Prometheus container.

        4

        Defines a minimum pod resource request of 2 GiB of memory for the Prometheus container.

        Note

        The Prometheus config map component is called prometheusK8s in the cluster-monitoring-config ConfigMap object and prometheus in the user-workload-monitoring-config ConfigMap object.

2. Save the file to apply the changes to the ConfigMap object.

   Warning

   Different configuration changes to the ConfigMap object result in different outcomes:

   • The pods are not redeployed. Therefore, there is no service outage.

   • The affected pods are redeployed:

     • For single-node clusters, this results in temporary service outage.
     • For multi-node clusters, because of high-availability, the affected pods are gradually rolled out and the monitoring stack remains available.
     • Configuring and resizing a persistent volume always results in a service outage, regardless of high availability.

   Each procedure that requires a change in the config map includes its expected outcome.

Procedure

1. Edit the ConfigMap object:

   • To assign tolerations to a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

   • To assign tolerations to a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify tolerations for the component:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              tolerations:
                <toleration_specification>

        Substitute <component> and <toleration_specification> accordingly.

        For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              tolerations:
              - key: "key1"
                operator: "Equal"
                value: "value1"
                effect: "NoSchedule"

2. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a value for enforcedBodySizeLimit to data/config.yaml/prometheusK8s to limit the body size that can be accepted per target scrape:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       prometheusK8s:
         enforcedBodySizeLimit: 40MB 1

   1

   Specify the maximum body size for scraped metrics targets. This enforcedBodySizeLimit example limits the uncompressed size per target scrape to 40 megabytes. Valid numeric values use the Prometheus data size format: B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes). The default value is 0, which specifies no limit. You can also set the value to automatic to calculate the limit automatically based on cluster capacity.

3. Save the file to apply the changes. The new configuration is applied automatically.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s.

3. Add an endpoint URL and authentication credentials in this section:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 1
           <endpoint_authentication_credentials> 2

   1

   The URL of the remote write endpoint.

   2

   The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

4. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           <your_write_relabel_configs> 1

   1

   The write relabel configuration settings.

   For <your_write_relabel_configs> substitute a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   The following sample shows how to forward a single metric called my_metric:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__]
             regex: 'my_metric'
             action: keep

   See the Prometheus relabel_config documentation for information about write relabel configuration options.

5. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the ConfigMap object:

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning
        • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
        • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.

        For example, to add metadata about the region and environment to all time series and alerts, use the following example:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                region: eu
                environment: prod

     3. Save the file to apply the changes. The new configuration is applied automatically.

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Define a map of labels you want to add for every metric under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                <key>: <value> 1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning
        • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
        • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.
        Note

        In the openshift-user-workload-monitoring project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting externalLabels for prometheus in the user-workload-monitoring-config ConfigMap object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use the following example:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                region: eu
                environment: prod

     3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values for the following settings under data/config.yaml/prometheusK8s to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: DoNotSchedule 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: prometheus

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values for the following settings under data/config.yaml/alertmanagermain to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: DoNotSchedule 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: alertmanager

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring namespace:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add values for the following settings under data/config.yaml/thanosRuler to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: ScheduleAnyway 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: thanos-ruler

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to the user-workload-monitoring-config config map, the pods and other resources in the openshift-user-workload-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   • To set a log level for a component in the openshift-monitoring project:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

   • To set a log level for a component in the openshift-user-workload-monitoring project:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are prometheus, prometheusOperator, and thanosRuler.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized loglevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a thanosQuerier section under data/config.yaml and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 1
         logLevel: <value> 2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the thanos-querier pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the enableRequestLogging value to false in the config map.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add profile: in the k8sPrometheusAdapter/audit section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         audit:
           profile: <audit_log_level> 1

   1

   The audit log level to apply to the Prometheus Adapter.

3. Set the audit log level by using one of the following values for the profile: parameter:

   • None: Do not log events.
   • Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. Metadata is the default audit log level.
   • Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.
   • RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.
4. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. In the config map, under k8sPrometheusAdapter/audit/profile, set the log level to Request and save the file.

2. Confirm that the pods for the Prometheus Adapter are running. The following example lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

3. Confirm that the audit log level and audit log file path are correctly configured:

   $ oc -n openshift-monitoring get deploy prometheus-adapter -o yaml

   Example output

   ...
     - --audit-policy-file=/etc/audit/request-profile.yaml
     - --audit-log-path=/var/log/adapter/audit.log

4. Confirm that the correct log level has been applied in the prometheus-adapter deployment in the openshift-monitoring project:

   $ oc -n openshift-monitoring exec deploy/prometheus-adapter -c prometheus-adapter -- cat /etc/audit/request-profile.yaml

   Example output

   "apiVersion": "audit.k8s.io/v1"
   "kind": "Policy"
   "metadata":
     "name": "Request"
   "omitStages":
   - "RequestReceived"
   "rules":
   - "level": "Request"

   Note

   If you enter an unrecognized profile value for the Prometheus Adapter in the ConfigMap object, no changes are made to the Prometheus Adapter, and an error is logged by the Cluster Monitoring Operator.

5. Review the audit log for the Prometheus Adapter:

   $ oc -n openshift-monitoring exec -c <prometheus_adapter_pod_name> -- cat /var/log/adapter/audit.log

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Note

   If you enable monitoring for user-defined projects, the user-workload-monitoring-config ConfigMap object is created by default.

4. Verify that the prometheus-operator, prometheus-user-workload, and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Assign the user-workload-monitoring-config-edit role to a user in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring adm policy add-role-to-user \
     user-workload-monitoring-config-edit <user> \
     --role-namespace openshift-user-workload-monitoring

2. Verify that the user is correctly assigned to the user-workload-monitoring-config-edit role by displaying the related role binding:

   $ oc describe rolebinding <role_binding_name> -n openshift-user-workload-monitoring

   Example command

   $ oc describe rolebinding user-workload-monitoring-config-edit -n openshift-user-workload-monitoring

   Example output

   Name:         user-workload-monitoring-config-edit
   Labels:       <none>
   Annotations:  <none>
   Role:
     Kind:  Role
     Name:  user-workload-monitoring-config-edit
   Subjects:
     Kind  Name  Namespace
     ----  ----  ---------
     User  user1           1

   1

   In this example, user1 is assigned to the user-workload-monitoring-config-edit role.

Procedure

1. Extract an authentication token to connect to Prometheus by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the thanos-querier API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath={.status.ingress[].host})

3. Set the namespace to the namespace in which your service is running by using the following command:

   $ NAMESPACE=ns1

4. Query the metrics of your own services in the command line by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}"

   The output shows the status for each application pod that Prometheus is scraping:

   The formatted example output

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "up",
             "endpoint": "web",
             "instance": "10.129.0.46:8080",
             "job": "prometheus-example-app",
             "namespace": "ns1",
             "pod": "prometheus-example-app-68d47c4fb6-jztp2",
             "service": "prometheus-example-app"
           },
           "value": [
             1591881154.748,
             "1"
           ]
         }
       ],
     }
   }

   Note
   • The formatted example output uses a filtering tool, such as jq, to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq.
   • The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time.

Procedure

1. Add the label to the project namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring=false'

2. To re-enable monitoring, remove the label from the namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring-'

   Note

   If there were any active monitoring targets for the project, it may take a few minutes for Prometheus to stop scraping them after adding the label.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   1. Set enableUserWorkload: to false under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: false

2. Save the file to apply the changes. Monitoring for user-defined projects is then disabled automatically.

3. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are terminated in the openshift-user-workload-monitoring project. This might take a short while:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   No resources found in openshift-user-workload-monitoring project.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserAlertmanagerConfig: true in the alertmanagerMain section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       alertmanagerMain:
         enableUserAlertmanagerConfig: true 1
       # ...

   1

   Set the enableUserAlertmanagerConfig value to true to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         enabled: true 1
         enableAlertmanagerConfig: true 2

   1

   Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.

   2

   Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

Procedure

1. If you have not obtained the OpenShift Container Platform API route for Thanos Querier, run the following command:

   $ oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}'

2. Retrieve a list of metrics in JSON format from the Thanos Querier API route by running the following command. This command uses oc to authenticate with a bearer token.

   $ curl -k -H "Authorization: Bearer $(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1

   1

   Replace <thanos_querier_route> with the OpenShift Container Platform API route for Thanos Querier.

1. Open the OpenShift Container Platform web console and navigate to Observe → Targets.
2. Optional: Filter the targets by status and source by selecting filters in the Filter list.
3. Optional: Search for a target by name or label by using the Text or Label field next to the search box.
4. Optional: Sort the targets by clicking one or more of the Endpoint, Status, Namespace, Last Scrape, and Scrape Duration column headers.

5. Click the URL in the Endpoint column for a target to navigate to its Target details page. This page provides information about the target, including:

   • The endpoint URL being scraped for metrics
   • The current Up or Down status of the target
   • A link to the namespace
   • A link to the ServiceMonitor details
   • Labels attached to the target
   • The most recent time that the target was scraped for metrics

1. Open the OpenShift Container Platform web console and go to the Observe → Alerting → Alerts page.
2. Optional: Search for alerts by name by using the Name field in the search list.
3. Optional: Filter alerts by state, severity, and source by selecting filters in the Filter list.
4. Optional: Sort the alerts by clicking one or more of the Name, Severity, State, and Source column headers.

5. Click the name of an alert to view its Alert details page. The page includes a graph that illustrates alert time series data. It also provides the following information about the alert:

   • A description of the alert
   • Messages associated with the alert
   • Labels attached to the alert
   • A link to its governing alerting rule
   • Silences for the alert, if any exist

1. Go to the Observe → Alerting → Silences page.
2. Optional: Filter the silences by name using the Search by name field.
3. Optional: Filter silences by state by selecting filters in the Filter list. By default, Active and Pending filters are applied.
4. Optional: Sort the silences by clicking one or more of the Name, Firing alerts, State, and Creator column headers.

5. Select the name of a silence to view its Silence details page. The page includes the following details:

   • Alert specification
   • Start time
   • End time
   • Silence state
   • Number and list of firing alerts

1. Go to the Observe → Alerting → Alerting rules page.
2. Optional: Filter alerting rules by state, severity, and source by selecting filters in the Filter list.
3. Optional: Sort the alerting rules by clicking one or more of the Name, Severity, Alert state, and Source column headers.

4. Select the name of an alerting rule to view its Alerting rule details page. The page provides the following details about the alerting rule:

   • Alerting rule name, severity, and description.
   • The expression that defines the condition for firing the alert.
   • The time for which the condition should be true for an alert to fire.
   • A graph for each alert governed by the alerting rule, showing the value with which the alert is firing.
   • A table of all alerts governed by the alerting rule.

1. Go to the Observe → <project_name> → Alerts page.

2. View details for an alert, silence, or an alerting rule:

   • Alert details can be viewed by clicking a greater than symbol (>) next to an alert name and then selecting the alert from the list.

   • Silence details can be viewed by clicking a silence in the Silenced by section of the Alert details page. The Silence details page includes the following information:

     • Alert specification
     • Start time
     • End time
     • Silence state
     • Number and list of firing alerts
   • Alerting rule details can be viewed by clicking the menu next to an alert in the Alerts page and then clicking View Alerting Rule.

Procedure

1. In the Administrator perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboards.
2. Choose a dashboard in the Dashboard list. Some dashboards, such as etcd and Prometheus dashboards, produce additional sub-menus when selected.

3. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
4. Optional: Select a Refresh Interval.
5. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. In the Developer perspective in the OpenShift Container Platform web console, navigate to Observe → Dashboard.
2. Select a project from the Project: drop-down list.

3. Select a dashboard from the Dashboard drop-down list to see the filtered metrics.

   Note

   All dashboards produce additional sub-menus when selected, except Kubernetes / Compute Resources / Namespace (Pods).

4. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by selecting Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
5. Optional: Select a Refresh Interval.
6. Hover over each of the graphs within a dashboard to display detailed information about specific items.

1. Add the Helm repository:

   $ helm repo add rh-ecosystem-edge https://rh-ecosystem-edge.github.io/console-plugin-nvidia-gpu

   $ helm repo update

2. Install the Helm chart in the default NVIDIA GPU operator namespace:

   $ helm install -n nvidia-gpu-operator console-plugin-nvidia-gpu rh-ecosystem-edge/console-plugin-nvidia-gpu

   Example output

   NAME: console-plugin-nvidia-gpu
   LAST DEPLOYED: Tue Aug 23 15:37:35 2022
   NAMESPACE: nvidia-gpu-operator
   STATUS: deployed
   REVISION: 1
   NOTES:
   View the Console Plugin NVIDIA GPU deployed resources by running the following command:
   
   $ oc -n {{ .Release.Namespace }} get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu
   
   Enable the plugin by running the following command:
   
   # Check if a plugins field is specified
   $ oc get consoles.operator.openshift.io cluster --output=jsonpath="{.spec.plugins}"
   
   # if not, then run the following command to enable the plugin
   $ oc patch consoles.operator.openshift.io cluster --patch '{ "spec": { "plugins": ["console-plugin-nvidia-gpu"] } }' --type=merge
   
   # if yes, then run the following command to enable the plugin
   $ oc patch consoles.operator.openshift.io cluster --patch '[{"op": "add", "path": "/spec/plugins/-", "value": "console-plugin-nvidia-gpu" }]' --type=json
   
   # add the required DCGM Exporter metrics ConfigMap to the existing NVIDIA operator ClusterPolicy CR:
   oc patch clusterpolicies.nvidia.com gpu-cluster-policy --patch '{ "spec": { "dcgmExporter": { "config": { "name": "console-plugin-nvidia-gpu" } } } }' --type=merge

   The dashboard relies mostly on Prometheus metrics exposed by the NVIDIA DCGM Exporter, but the default exposed metrics are not enough for the dashboard to render the required gauges. Therefore, the DGCM exporter is configured to expose a custom set of metrics, as shown here.

   apiVersion: v1
   data:
     dcgm-metrics.csv: |
       DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, gpu utilization.
       DCGM_FI_DEV_MEM_COPY_UTIL, gauge, mem utilization.
       DCGM_FI_DEV_ENC_UTIL, gauge, enc utilization.
       DCGM_FI_DEV_DEC_UTIL, gauge, dec utilization.
       DCGM_FI_DEV_POWER_USAGE, gauge, power usage.
       DCGM_FI_DEV_POWER_MGMT_LIMIT_MAX, gauge, power mgmt limit.
       DCGM_FI_DEV_GPU_TEMP, gauge, gpu temp.
       DCGM_FI_DEV_SM_CLOCK, gauge, sm clock.
       DCGM_FI_DEV_MAX_SM_CLOCK, gauge, max sm clock.
       DCGM_FI_DEV_MEM_CLOCK, gauge, mem clock.
       DCGM_FI_DEV_MAX_MEM_CLOCK, gauge, max mem clock.
   kind: ConfigMap
   metadata:
     annotations:
       meta.helm.sh/release-name: console-plugin-nvidia-gpu
       meta.helm.sh/release-namespace: nvidia-gpu-operator
     creationTimestamp: "2022-10-26T19:46:41Z"
     labels:
       app.kubernetes.io/component: console-plugin-nvidia-gpu
       app.kubernetes.io/instance: console-plugin-nvidia-gpu
       app.kubernetes.io/managed-by: Helm
       app.kubernetes.io/name: console-plugin-nvidia-gpu
       app.kubernetes.io/part-of: console-plugin-nvidia-gpu
       app.kubernetes.io/version: latest
       helm.sh/chart: console-plugin-nvidia-gpu-0.2.3
     name: console-plugin-nvidia-gpu
     namespace: nvidia-gpu-operator
     resourceVersion: "19096623"
     uid: 96cdf700-dd27-437b-897d-5cbb1c255068

   Install the ConfigMap and edit the NVIDIA Operator ClusterPolicy CR to add that ConfigMap in the DCGM exporter configuration. The installation of the ConfigMap is done by the new version of the Console Plugin NVIDIA GPU Helm Chart, but the ClusterPolicy CR editing is done by the user.

3. View the deployed resources:

   $ oc -n nvidia-gpu-operator get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/console-plugin-nvidia-gpu-7dc9cfb5df-ztksx   1/1     Running   0          2m6s
   
   NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/console-plugin-nvidia-gpu   ClusterIP   172.30.240.138   <none>        9443/TCP   2m6s
   
   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/console-plugin-nvidia-gpu   1/1     1            1           2m6s
   
   NAME                                                   DESIRED   CURRENT   READY   AGE
   replicaset.apps/console-plugin-nvidia-gpu-7dc9cfb5df   1         1         1       2m6s

Procedure

1. Extract an authentication token by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the alertmanager-main API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.status.ingress[].host})

3. Query the service API receivers for Alertmanager by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v2/receivers"

Procedure

1. Retrieve the bearer token by running the following the command:

   $ TOKEN=$(oc whoami -t)

2. Get the Prometheus federation route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath={.status.ingress[].host})

3. Query metrics from the /federate route. The following example command queries up metrics:

   $ curl -G -k -H "Authorization: Bearer $TOKEN" https://$HOST/federate --data-urlencode 'match[]=up'

   Example output

   # TYPE up untyped
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
   ...

Procedure

1. Extract an authentication token to connect to Prometheus by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the thanos-querier API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath={.status.ingress[].host})

3. Set the namespace to the namespace in which your service is running by using the following command:

   $ NAMESPACE=ns1

4. Query the metrics of your own services in the command line by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}"

   The output shows the status for each application pod that Prometheus is scraping:

   The formatted example output

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "up",
             "endpoint": "web",
             "instance": "10.129.0.46:8080",
             "job": "prometheus-example-app",
             "namespace": "ns1",
             "pod": "prometheus-example-app-68d47c4fb6-jztp2",
             "service": "prometheus-example-app"
           },
           "value": [
             1591881154.748,
             "1"
           ]
         }
       ],
     }
   }

   Note
   • The formatted example output uses a filtering tool, such as jq, to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq.
   • The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time.

Procedure

1. Check that the corresponding labels match in the service and ServiceMonitor resource configurations.

   1. Obtain the label defined in the service. The following example queries the prometheus-example-app service in the ns1 project:

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Example output

        labels:
          app: prometheus-example-app

   2. Check that the matchLabels definition in the ServiceMonitor resource configuration matches the label output in the preceding step. The following example queries the prometheus-example-monitor service monitor in the ns1 project:

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Example output

      apiVersion: v1
      kind: ServiceMonitor
      metadata:
        name: prometheus-example-monitor
        namespace: ns1
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

2. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

3. Review the target status for your endpoint on the Metrics targets page in the OpenShift Container Platform web console UI.

   1. Log in to the OpenShift Container Platform web console and navigate to Observe → Targets in the Administrator perspective.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

4. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

   3. Save the file to apply the changes. The affected prometheus-operator pod is automatically redeployed.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Example output

              - --log-level=debug

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.