Chapter 2. Configuring performance and scalability for user workload monitoring

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label>

   where:

   <node_name>

   Specifies the name of the node where you want to add the label.

   <node_label>

   Specifies the name of the wanted label.

2. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       # ...
       <component>:
         nodeSelector:
           <node_label_1>
           <node_label_2>
       # ...

   where:

   <component>

   Specifies the monitoring stack component.

   <node_label_1>

   Specifies the label you added to the node.

   <node_label_2>

   Optional: Specifies additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the thanosRuler component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       alertmanager:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheus:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosRuler:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforced limit and time interval configurations to data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000
         enforcedLabelLimit: 500
         enforcedLabelNameLengthLimit: 50
         enforcedLabelValueLengthLimit: 600
         scrapeInterval: 1m30s
         evaluationInterval: 1m15s

   where:

   enforcedSampleLimit

   Defines the maximum number of samples that can be accepted per target scrape. A value is required if this parameter is specified. This example limits the number to 50,000.

   enforcedLabelLimit

   Defines the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   enforcedLabelNameLengthLimit

   Defines the maximum character length for a label name. The default value is 0, which specifies no limit.

   enforcedLabelValueLengthLimit

   Defines the maximum character length for a label value. The default value is 0, which specifies no limit.

   scrapeInterval

   Defines the interval between consecutive scrapes. The interval must be set between 5 seconds and 5 minutes. The default value is 30s.

   evaluationInterval

   Defines the interval between Prometheus rule evaluations. The interval must be set between 5 seconds and 5 minutes. The default value for Prometheus is 30s.

   Note

   You can also configure the evaluationInterval property for Thanos Ruler through the data/config.yaml/thanosRuler field. The default value for Thanos Ruler is 15s.

3. Save the file to apply the changes. The limits are applied automatically.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts
     namespace: ns1
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.'
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m
         labels:
           severity: warning
       - alert: ApproachingEnforcedSamplesLimit
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.'
         expr: (scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0)) > 0.9
         for: 10m
         labels:
           severity: warning

   where:

   metadata.name

   Specifies the name of the alerting rule.

   metadata.namespace

   Specifies the user-defined project where the alerting rule is deployed.

   TargetDown

   Specifies an alert that fires if the target cannot be scraped and is not available for the for duration.

   ApproachingEnforcedSamplesLimit

   Specifies an alert that fires when the defined scrape sample threshold is exceeded and lasts for the specified for duration.

   annotations.message

   Specifies the message that is displayed when the alert fires.

   expr

   Specifies the PromQL query expression that defines the new rule. For example, the ApproachingEnforcedSamplesLimit alert fires when the number of ingested samples exceeds 90% of the configured limit.

   for

   Specifies that the conditions for the alert must be true for this duration before the alert is fired.

   labels.severity

   Specifies the severity for the alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml

3. Additionally, you can check if a target has hit the configured limit:

   1. In the OpenShift Container Platform web console, go to Observe Targets and select an endpoint with a Down status that you want to check.

      The Scrape failed: sample limit exceeded message is displayed if the endpoint failed because of an exceeded sample limit.