Chapter 2. Configuring the monitoring stack

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Check whether the user-workload-monitoring-config ConfigMap object exists:

   $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

2. If the user-workload-monitoring-config ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called user-workload-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f user-workload-monitoring-config.yaml

      Note

      Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node-name> <node-label>

2. Edit the ConfigMap object:

   • To move a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

   • To move a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

3. Save the file to apply the changes. The components specified in the new configuration are moved to the new nodes automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add an enabled: true key-value pair as shown in the following sample:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         dedicatedServiceMonitors:
           enabled: true 1

   1

   Set the value of the enabled field to true to deploy a dedicated service monitor that exposes the kubelet /metrics/resource endpoint.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to a cluster-monitoring-config config map, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   • To configure a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called local-storage.

        The following example configures a PVC that claims local persistent storage for Alertmanager:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

   • To configure a PVC for a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called local-storage.

        The following example configures a PVC that claims local persistent storage for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To resize a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add a new storage size for the PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures a PVC that sets the local persistent storage to 100 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example configures a PVC that sets the local persistent storage for Alertmanager to 40 gigabytes:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

   • To resize a PVC for a component that monitors user-defined projects:

     Note

     You can resize the volumes for the Thanos Ruler and Prometheus instances that monitor user-defined projects.

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Update the PVC configuration for the monitoring component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures the PVC size to 100 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example sets the PVC size to 20 gigabytes for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 20Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the related project might be redeployed. The monitoring processes running in that project might also be restarted.

3. Manually patch every PVC with the updated storage request. The following example resizes the storage size for the Prometheus component in the openshift-monitoring namespace to 100Gi:

   $ for p in $(oc -n openshift-monitoring get pvc -l app.kubernetes.io/name=prometheus -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \
     oc -n openshift-monitoring patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"100Gi"}}}}'; \
     done

4. Delete the underlying StatefulSet with the --cascade=orphan parameter:

   $ oc delete statefulset -l app.kubernetes.io/name=prometheus --cascade=orphan

Procedure

1. Edit the ConfigMap object:

   • To modify the retention time for the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your retention time configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: <time_specification>

        Substitute <time_specification> with a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years).

        The following example sets the retention time to 24 hours for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: 24h

   • To modify the retention time for the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your retention time configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: <time_specification>

        Substitute <time_specification> with a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years).

        The following example sets the retention time to 24 hours for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: 24h

2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 1

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

3. Save the file to apply the changes. The pods affected by the new configuration automatically restart.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforcedSampleLimit configuration to data/config.yaml to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Save the file to apply the changes. The limit is applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to the user-workload-monitoring-config ConfigMap object, the pods and other resources in the openshift-user-workload-monitoring project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 1
     namespace: ns1 2
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 3
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 4
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 5
         labels:
           severity: warning 6
       - alert: ApproachingEnforcedSamplesLimit 7
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 8
         expr: scrape_samples_scraped/50000 > 0.8 9
         for: 10m 10
         labels:
           severity: warning 11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule will be deployed.

   3

   The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.

   4

   The message that will be output when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.

   8

   The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml