Chapter 4. Configuring user workload monitoring

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserWorkload: true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true 1

   1

   When set to true, the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster.

3. Save the file to apply the changes. Monitoring for user-defined projects is then enabled automatically.

   Note

   If you enable monitoring for user-defined projects, the user-workload-monitoring-config ConfigMap object is created by default.

4. Verify that the prometheus-operator, prometheus-user-workload, and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project. It might take a short while for the pods to start:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             4/4     Running       1          3h
   prometheus-user-workload-1             4/4     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Assign the user-workload-monitoring-config-edit role to a user in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring adm policy add-role-to-user \
     user-workload-monitoring-config-edit <user> \
     --role-namespace openshift-user-workload-monitoring

2. Verify that the user is correctly assigned to the user-workload-monitoring-config-edit role by displaying the related role binding:

   $ oc describe rolebinding <role_binding_name> -n openshift-user-workload-monitoring

   Example command

   $ oc describe rolebinding user-workload-monitoring-config-edit -n openshift-user-workload-monitoring

   Example output

   Name:         user-workload-monitoring-config-edit
   Labels:       <none>
   Annotations:  <none>
   Role:
     Kind:  Role
     Name:  user-workload-monitoring-config-edit
   Subjects:
     Kind  Name  Namespace
     ----  ----  ---------
     User  user1           1

   1

   In this example, user1 is assigned to the user-workload-monitoring-config-edit role.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enableUserAlertmanagerConfig: true in the alertmanagerMain section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       alertmanagerMain:
         enableUserAlertmanagerConfig: true 1
       # ...

   1

   Set the enableUserAlertmanagerConfig value to true to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         enabled: true 1
         enableAlertmanagerConfig: true 2

   1

   Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.

   2

   Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to User Management RoleBindings Create binding.
2. In the Binding Type section, select the Namespace Role Binding type.
3. In the Name field, enter a name for the role binding.

4. In the Namespace field, select the project where you want to grant the access.

   Important

   The monitoring role or cluster role permissions that you grant to a user by using this procedure apply only to the project that you select in the Namespace field.

5. Select a monitoring role or cluster role from the Role Name list.
6. In the Subject section, select User.
7. In the Subject Name field, enter the name of the user.
8. Select Create to apply the role binding.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label> 1

   1

   Replace <node_name> with the name of the node where you want to add the label. Replace <node_label> with the name of the wanted label.

2. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       # ...
       <component>: 1
         nodeSelector:
           <node_label_1> 2
           <node_label_2> 3
       # ...

   1

   Substitute <component> with the appropriate monitoring stack component name.

   2

   Substitute <node_label_1> with the label you added to the node.

   3

   Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   Example of setting resource limits and requests

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheus:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosRuler:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforcedSampleLimit configuration to data/config.yaml to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Add the enforcedLabelLimit, enforcedLabelNameLengthLimit, and enforcedLabelValueLengthLimit configurations to data/config.yaml to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedLabelLimit: 500 1
         enforcedLabelNameLengthLimit: 50 2
         enforcedLabelValueLengthLimit: 600 3

   1

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   2

   Specifies the maximum length in characters of a label name. The default value is 0, which specifies no limit.

   3

   Specifies the maximum length in characters of a label value. The default value is 0, which specifies no limit.

4. Save the file to apply the changes. The limits are applied automatically.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 1
     namespace: ns1 2
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 3
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 4
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 5
         labels:
           severity: warning 6
       - alert: ApproachingEnforcedSamplesLimit 7
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 8
         expr: scrape_samples_scraped/50000 > 0.8 9
         for: 10m 10
         labels:
           severity: warning 11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule will be deployed.

   3

   The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.

   4

   The message that will be output when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.

   8

   The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add your PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>: 1
         volumeClaimTemplate:
           spec:
             storageClassName: <storage_class> 2
             resources:
               requests:
                 storage: <amount_of_storage> 3

   1

   Specify the monitoring component for which you want to configure the PVC.

   2

   Specify an existing storage class. If a storage class is not specified, the default storage class is used.

   3

   Specify the amount of required storage.

   The following example configures a PVC that claims persistent storage for Thanos Ruler:

   Example PVC configuration

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         volumeClaimTemplate:
           spec:
             storageClassName: my-storage-class
             resources:
               requests:
                 storage: 10Gi

   Note

   Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed and the new storage configuration is applied.

   Warning

   When you update the config map with a PVC configuration, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Manually expand a PVC with the updated storage request. For more information, see "Expanding persistent volume claims (PVCs) with a file system" in Expanding persistent volumes.

2. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

3. Add a new storage size for the PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>: 1
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: <amount_of_storage> 2

   1

   The component for which you want to change the storage size.

   2

   Specify the new size for the storage volume. It must be greater than the previous value.

   The following example sets the new PVC request to 20 gigabytes for Thanos Ruler:

   Example storage configuration for thanosRuler

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         volumeClaimTemplate:
           spec:
             resources:
               requests:
                 storage: 20Gi

   Note

   Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

4. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

   Warning

   When you update the config map with a new storage size, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 1

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Add the following deployment and service configuration details to the file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   This configuration deploys a service named prometheus-example-app in the user-defined ns1 project. This service exposes the custom version metric.

3. Apply the configuration to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   It takes some time to deploy the service.

4. You can check that the pod is running:

   $ oc -n ns1 get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

Procedure

1. Create a new YAML configuration file named example-app-service-monitor.yaml.

2. Add a ServiceMonitor resource to the YAML file. The following example creates a service monitor named prometheus-example-monitor to scrape metrics exposed by the prometheus-example-app service in the ns1 namespace:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: prometheus-example-monitor
     namespace: ns1 1
   spec:
     endpoints:
     - interval: 30s
       port: web 2
       scheme: http
     selector: 3
       matchLabels:
         app: prometheus-example-app

   1

   Specify a user-defined namespace where your service runs.

   2

   Specify endpoint ports to be scraped by Prometheus.

   3

   Configure a selector to match your service based on its metadata labels.

   Note

   A ServiceMonitor resource in a user-defined namespace can only discover services in the same namespace. That is, the namespaceSelector field of the ServiceMonitor resource is always ignored.

3. Apply the configuration to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   It takes some time to deploy the ServiceMonitor resource.

4. Verify that the ServiceMonitor resource is running:

   $ oc -n <namespace> get servicemonitor

   Example output

   NAME                         AGE
   prometheus-example-monitor   81m

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add a secrets: section under data/config.yaml/alertmanager with the following configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         secrets: 1
         - <secret_name_1> 2
         - <secret_name_2>

   1

   This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.

   2

   The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

   The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         secrets:
         - test-secret-basic-auth
         - test-secret-api-token

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 1
       receiver: <receiver> 2
   receivers:
   - name: Default
   - name: <receiver>
     <receiver_configuration> 3

   1

   Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.

   2

   Specify the name of the receiver to use for the alerts group.

   3

   Specify the receiver configuration.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-