Chapter 5. Enabling monitoring for user-defined projects

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Warning

   When changes are saved to the cluster-monitoring-config ConfigMap object, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also be restarted.

4. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Extract a token to connect to Prometheus:

   $ SECRET=`oc get secret -n openshift-user-workload-monitoring | grep  prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`

   $ TOKEN=`echo $(oc get secret $SECRET -n openshift-user-workload-monitoring -o json | jq -r '.data.token') | base64 -d`

2. Extract your route host:

   $ THANOS_QUERIER_HOST=`oc get route thanos-querier -n openshift-monitoring -o json | jq -r '.spec.host'`

3. Query the metrics of your own services in the command line. For example:

   $ NAMESPACE=ns1

   $ curl -X GET -kG "https://$THANOS_QUERIER_HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}" -H "Authorization: Bearer $TOKEN"

   The output will show you the duration that your application pods have been up.

   Example output

   {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","endpoint":"web","instance":"10.129.0.46:8080","job":"prometheus-example-app","namespace":"ns1","pod":"prometheus-example-app-68d47c4fb6-jztp2","service":"prometheus-example-app"},"value":[1591881154.748,"1"]}]}}

Procedure

1. Add the label to the project namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring=false'

2. To re-enable monitoring, remove the label from the namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring-'

   Note

   If there were any active monitoring targets for the project, it may take a few minutes for Prometheus to stop scraping them after adding the label.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   1. Set enableUserWorkload: to false under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: false

2. Save the file to apply the changes. Monitoring for user-defined projects is then disabled automatically.

3. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are terminated in the openshift-user-workload-monitoring project. This might take a short while:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   No resources found in openshift-user-workload-monitoring project.