Chapter 7. Configuring user workload monitoring

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Note

   If you enable monitoring for user-defined projects, the user-workload-monitoring-config ConfigMap object is created by default.

4. Verify that the prometheus-operator, prometheus-user-workload, and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Assign the user-workload-monitoring-config-edit role to a user in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring adm policy add-role-to-user \
     user-workload-monitoring-config-edit <user> \
     --role-namespace openshift-user-workload-monitoring

2. Verify that the user is correctly assigned to the user-workload-monitoring-config-edit role by displaying the related role binding:

   $ oc describe rolebinding <role_binding_name> -n openshift-user-workload-monitoring

   Example command

   $ oc describe rolebinding user-workload-monitoring-config-edit -n openshift-user-workload-monitoring

   Example output

   Name:         user-workload-monitoring-config-edit
   Labels:       <none>
   Annotations:  <none>
   Role:
     Kind:  Role
     Name:  user-workload-monitoring-config-edit
   Subjects:
     Kind  Name  Namespace
     ----  ----  ---------
     User  user1           1

   1

   In this example, user1 is assigned to the user-workload-monitoring-config-edit role.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserAlertmanagerConfig: true in the alertmanagerMain section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       alertmanagerMain:
         enableUserAlertmanagerConfig: true 1
       # ...

   1

   Set the enableUserAlertmanagerConfig value to true to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         enabled: true 1
         enableAlertmanagerConfig: true 2

   1

   Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.

   2

   Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to User Management RoleBindings Create binding.
2. In the Binding Type section, select the Namespace Role Binding type.
3. In the Name field, enter a name for the role binding.

4. In the Namespace field, select the project where you want to grant the access.

   Important

   The monitoring role or cluster role permissions that you grant to a user by using this procedure apply only to the project that you select in the Namespace field.

5. Select a monitoring role or cluster role from the Role Name list.
6. In the Subject section, select User.
7. In the Subject Name field, enter the name of the user.
8. Select Create to apply the role binding.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label> 1

   1

   Replace <node_name> with the name of the node where you want to add the label. Replace <node_label> with the name of the wanted label.

2. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       # ...
       <component>: 1
         nodeSelector:
           <node_label_1> 2
           <node_label_2> 3
       # ...

   1

   Substitute <component> with the appropriate monitoring stack component name.

   2

   Substitute <node_label_1> with the label you added to the node.

   3

   Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   Example of setting resource limits and requests

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheus:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosRuler:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforcedSampleLimit configuration to data/config.yaml to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Add the enforcedLabelLimit, enforcedLabelNameLengthLimit, and enforcedLabelValueLengthLimit configurations to data/config.yaml to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedLabelLimit: 500 1
         enforcedLabelNameLengthLimit: 50 2
         enforcedLabelValueLengthLimit: 600 3

   1

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   2

   Specifies the maximum length in characters of a label name. The default value is 0, which specifies no limit.

   3

   Specifies the maximum length in characters of a label value. The default value is 0, which specifies no limit.

4. Save the file to apply the changes. The limits are applied automatically.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 1
     namespace: ns1 2
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 3
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 4
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 5
         labels:
           severity: warning 6
       - alert: ApproachingEnforcedSamplesLimit 7
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 8
         expr: (scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0)) > 0.9 9
         for: 10m 10
         labels:
           severity: warning 11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule is deployed.

   3

   The TargetDown alert fires if the target cannot be scraped and is not available for the for duration.

   4

   The message that is displayed when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert fires when the defined scrape sample threshold is exceeded and lasts for the specified for duration.

   8

   The message that is displayed when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example, the alert fires when the number of ingested samples exceeds 90% of the configured limit.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml

3. Additionally, you can check if a target has hit the configured limit:

   1. In the Administrator perspective of the web console, go to Observe Targets and select an endpoint with a Down status that you want to check.

      The Scrape failed: sample limit exceeded message is displayed if the endpoint failed because of an exceeded sample limit.