Chapter 14. Troubleshooting monitoring issues

Procedure

1. Check that the corresponding labels match in the service and ServiceMonitor resource configurations.

   1. Obtain the label defined in the service. The following example queries the prometheus-example-app service in the ns1 project:

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Copy to Clipboard

      Example output

        labels:
          app: prometheus-example-app

      Copy to Clipboard

   2. Check that the matchLabels app label in the ServiceMonitor resource configuration matches the label output in the preceding step:

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Copy to Clipboard

      Example output

      apiVersion: v1
      kind: Service
      # ...
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app
      # ...

      Copy to Clipboard

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

2. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Copy to Clipboard

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

      Copy to Clipboard

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      Copy to Clipboard

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

      Copy to Clipboard

3. Review the target status for your endpoint on the Metrics targets page in the OpenShift Container Platform web console UI.

   1. Log in to the OpenShift Container Platform web console and navigate to Observe Targets in the Administrator perspective.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

4. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      Copy to Clipboard

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

      Copy to Clipboard

   3. Save the file to apply the changes.

      Note

      The prometheus-operator in the openshift-user-workload-monitoring project restarts automatically when you apply the log-level change.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Copy to Clipboard

      Example output

              - --log-level=debug

      Copy to Clipboard

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Copy to Clipboard
      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Administrator perspective, navigate to Observe Metrics.

2. Run the following Prometheus Query Language (PromQL) query in the Expression field. This returns the ten metrics that have the highest number of scrape samples:

   topk(10,count by (job)({__name__=~".+"}))

   Copy to Clipboard

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts.

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core OpenShift Container Platform project, create a Red Hat support case on the Red Hat Customer Portal.

4. Review the TSDB status using the Prometheus HTTP API by running the following commands as a cluster administrator:

   $ oc login -u <username> -p <password>

   Copy to Clipboard

   $ host=$(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath={.spec.host})

   Copy to Clipboard

   $ token=$(oc whoami -t)

   Copy to Clipboard

   $ curl -H "Authorization: Bearer $token" -k "https://$host/api/v1/status/tsdb"

   Copy to Clipboard

   Example output

   "status": "success",

   Copy to Clipboard