Chapter 3. Configuring user workload monitoring

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         enabled: true 

   1

   
         enableAlertmanagerConfig: true 

   2

   Copy to Clipboard

   1

   Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.

   2

   Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label> 

   1

   Copy to Clipboard

   1

   Replace <node_name> with the name of the node where you want to add the label. Replace <node_label> with the name of the wanted label.

2. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       # ...
       <component>: 

   1

   
         nodeSelector:
           <node_label_1> 

   2

   
           <node_label_2> 

   3

   
       # ...

   Copy to Clipboard

   1

   Substitute <component> with the appropriate monitoring stack component name.

   2

   Substitute <node_label_1> with the label you added to the node.

   3

   Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Copy to Clipboard

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

   Copy to Clipboard
3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   Example of setting resource limits and requests

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheus:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosRuler:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi

   Copy to Clipboard

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add the enforced limit and time interval configurations to data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 

   1

   
         enforcedLabelLimit: 500 

   2

   
         enforcedLabelNameLengthLimit: 50 

   3

   
         enforcedLabelValueLengthLimit: 600 

   4

   
         scrapeInterval: 1m30s 

   5

   
         evaluationInterval: 1m15s 

   6

   Copy to Clipboard

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

   2

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   3

   Specifies the maximum character length for a label name. The default value is 0, which specifies no limit.

   4

   Specifies the maximum character length for a label value. The default value is 0, which specifies no limit.

   5

   Specifies the interval between consecutive scrapes. The interval must be set between 5 seconds and 5 minutes. The default value is 30s.

   6

   Specifies the interval between Prometheus rule evaluations. The interval must be set between 5 seconds and 5 minutes. The default value for Prometheus is 30s.

   Note

   You can also configure the evaluationInterval property for Thanos Ruler through the data/config.yaml/thanosRuler field. The default value for Thanos Ruler is 15s.

3. Save the file to apply the changes. The limits are applied automatically.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add your PVC configuration for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>: 

   1

   
         volumeClaimTemplate:
           spec:
             storageClassName: <storage_class> 

   2

   
             resources:
               requests:
                 storage: <amount_of_storage> 

   3

   Copy to Clipboard

   1

   Specify the monitoring component for which you want to configure the PVC.

   2

   Specify an existing storage class. If a storage class is not specified, the default storage class is used.

   3

   Specify the amount of required storage.

   The following example configures a PVC that claims persistent storage for Thanos Ruler:

   Example PVC configuration

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         volumeClaimTemplate:
           spec:
             storageClassName: my-storage-class
             resources:
               requests:
                 storage: 10Gi

   Copy to Clipboard

   Note

   Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed and the new storage configuration is applied.

   Warning

   When you update the config map with a PVC configuration, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add the retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 

   1

   Copy to Clipboard

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

   Copy to Clipboard
3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Add the following deployment and service configuration details to the file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   Copy to Clipboard

   This configuration deploys a service named prometheus-example-app in the user-defined ns1 project. This service exposes the custom version metric.

3. Apply the configuration to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   Copy to Clipboard

   It takes some time to deploy the service.

4. You can check that the pod is running:

   $ oc -n ns1 get pod

   Copy to Clipboard

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

   Copy to Clipboard

Procedure

1. Create a new YAML configuration file named example-app-service-monitor.yaml.

2. Add a ServiceMonitor resource to the YAML file. The following example creates a service monitor named prometheus-example-monitor to scrape metrics exposed by the prometheus-example-app service in the ns1 namespace:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: prometheus-example-monitor
     namespace: ns1 

   1

   
   spec:
     endpoints:
     - interval: 30s
       port: web 

   2

   
       scheme: http
     selector: 

   3

   
       matchLabels:
         app: prometheus-example-app

   Copy to Clipboard

   1

   Specify a user-defined namespace where your service runs.

   2

   Specify endpoint ports to be scraped by Prometheus.

   3

   Configure a selector to match your service based on its metadata labels.

   Note

   A ServiceMonitor resource in a user-defined namespace can only discover services in the same namespace. That is, the namespaceSelector field of the ServiceMonitor resource is always ignored.

3. Apply the configuration to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   Copy to Clipboard

   It takes some time to deploy the ServiceMonitor resource.

4. Verify that the ServiceMonitor resource is running:

   $ oc -n <namespace> get servicemonitor

   Copy to Clipboard

   Example output

   NAME                         AGE
   prometheus-example-monitor   81m

   Copy to Clipboard

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   Copy to Clipboard

2. Add a secrets: section under data/config.yaml/alertmanager with the following configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         secrets: 

   1

   
         - <secret_name_1> 

   2

   
         - <secret_name_2>

   Copy to Clipboard

   1

   This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.

   2

   The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

   The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         secrets:
         - test-secret-basic-auth
         - test-secret-api-token

   Copy to Clipboard
3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

   Copy to Clipboard
3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   Copy to Clipboard

   The configuration is automatically applied to the Alertmanager pods.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

   Copy to Clipboard

2. Edit the configuration in alertmanager.yaml:

   global:
     http_config:
       proxy_from_environment: true 

   1

   
   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 

   2

   
       receiver: <receiver> 

   3

   
   receivers:
   - name: Default
   - name: <receiver>
     <receiver_configuration> 

   4

   Copy to Clipboard

   1

   If you configured an HTTP cluster-wide proxy, set the proxy_from_environment parameter to true to enable proxying for all alert receivers.

   2

   Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.

   3

   Specify the name of the receiver to use for the alerts group.

   4

   Specify the receiver configuration.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-

   Copy to Clipboard