Chapter 6. Enabling monitoring for user-defined projects

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 

   1

   Copy to Clipboard Toggle word wrap

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Note

   If you enable monitoring for user-defined projects, the user-workload-monitoring-config ConfigMap object is created by default.

4. Verify that the prometheus-operator, prometheus-user-workload, and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

   Copy to Clipboard Toggle word wrap

Procedure

1. Assign the user-workload-monitoring-config-edit role to a user in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring adm policy add-role-to-user \
     user-workload-monitoring-config-edit <user> \
     --role-namespace openshift-user-workload-monitoring

   Copy to Clipboard Toggle word wrap

2. Verify that the user is correctly assigned to the user-workload-monitoring-config-edit role by displaying the related role binding:

   $ oc describe rolebinding <role_binding_name> -n openshift-user-workload-monitoring

   Copy to Clipboard Toggle word wrap

   Example command

   $ oc describe rolebinding user-workload-monitoring-config-edit -n openshift-user-workload-monitoring

   Copy to Clipboard Toggle word wrap

   Example output

   Name:         user-workload-monitoring-config-edit
   Labels:       <none>
   Annotations:  <none>
   Role:
     Kind:  Role
     Name:  user-workload-monitoring-config-edit
   Subjects:
     Kind  Name  Namespace
     ----  ----  ---------
     User  user1           

   1

   Copy to Clipboard Toggle word wrap

   1

   In this example, user1 is assigned to the user-workload-monitoring-config-edit role.

Procedure

1. Extract an authentication token to connect to Prometheus by running the following command:

   $ TOKEN=$(oc whoami -t)

   Copy to Clipboard Toggle word wrap

2. Extract the thanos-querier API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath='{.status.ingress[].host}')

   Copy to Clipboard Toggle word wrap

3. Set the namespace to the namespace in which your service is running by using the following command:

   $ NAMESPACE=ns1

   Copy to Clipboard Toggle word wrap

4. Query the metrics of your own services in the command line by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}"

   Copy to Clipboard Toggle word wrap

   The output shows the status for each application pod that Prometheus is scraping:

   The formatted example output

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "up",
             "endpoint": "web",
             "instance": "10.129.0.46:8080",
             "job": "prometheus-example-app",
             "namespace": "ns1",
             "pod": "prometheus-example-app-68d47c4fb6-jztp2",
             "service": "prometheus-example-app"
           },
           "value": [
             1591881154.748,
             "1"
           ]
         }
       ],
     }
   }

   Copy to Clipboard Toggle word wrap

   Note
   • The formatted example output uses a filtering tool, such as jq, to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq.
   • The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time.

Procedure

1. Add the label to the project namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring=false'

   Copy to Clipboard Toggle word wrap

2. To re-enable monitoring, remove the label from the namespace:

   $ oc label namespace my-project 'openshift.io/user-monitoring-'

   Copy to Clipboard Toggle word wrap
   Note

   If there were any active monitoring targets for the project, it may take a few minutes for Prometheus to stop scraping them after adding the label.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

   1. Set enableUserWorkload: to false under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          enableUserWorkload: false

      Copy to Clipboard Toggle word wrap
2. Save the file to apply the changes. Monitoring for user-defined projects is then disabled automatically.

3. Check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are terminated in the openshift-user-workload-monitoring project. This might take a short while:

   $ oc -n openshift-user-workload-monitoring get pod

   Copy to Clipboard Toggle word wrap

   Example output

   No resources found in openshift-user-workload-monitoring project.

   Copy to Clipboard Toggle word wrap