Chapter 2. Configuring the monitoring stack

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Check whether the user-workload-monitoring-config ConfigMap object exists:

   $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

2. If the user-workload-monitoring-config ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called user-workload-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f user-workload-monitoring-config.yaml

      Note

      Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node-name> <node-label>

2. Edit the ConfigMap object:

   • To move a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

   • To move a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

3. Save the file to apply the changes. The components specified in the new configuration are moved to the new nodes automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add an enabled: true key-value pair as shown in the following sample:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         dedicatedServiceMonitors:
           enabled: true 1

   1

   Set the value of the enabled field to true to deploy a dedicated service monitor that exposes the kubelet /metrics/resource endpoint.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to a cluster-monitoring-config config map, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   • To configure a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called local-storage.

        The following example configures a PVC that claims local persistent storage for Alertmanager:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

   • To configure a PVC for a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called local-storage.

        The following example configures a PVC that claims local persistent storage for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the ConfigMap object:

   • To resize a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add a new storage size for the PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures a PVC that sets the local persistent storage to 100 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example configures a PVC that sets the local persistent storage for Alertmanager to 40 gigabytes:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

   • To resize a PVC for a component that monitors user-defined projects:

     Note

     You can resize the volumes for the Thanos Ruler and Prometheus instances that monitor user-defined projects.

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Update the PVC configuration for the monitoring component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures the PVC size to 100 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example sets the PVC size to 20 gigabytes for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 20Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the related project might be redeployed. The monitoring processes running in that project might also be restarted.

3. Manually patch every PVC with the updated storage request. The following example resizes the storage size for the Prometheus component in the openshift-monitoring namespace to 100Gi:

   $ for p in $(oc -n openshift-monitoring get pvc -l app.kubernetes.io/name=prometheus -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \
     oc -n openshift-monitoring patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"100Gi"}}}}'; \
     done

4. Delete the underlying StatefulSet with the --cascade=orphan parameter:

   $ oc delete statefulset -l app.kubernetes.io/name=prometheus --cascade=orphan

Procedure

1. Edit the ConfigMap object:

   • To modify the retention time and size for the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add the retention time and size configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: <time_specification> 1
              retentionSize: <size_specification> 2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: 24h
              retentionSize: 10GB

   • To modify the retention time and size for the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add the retention time and size configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: <time_specification> 1
              retentionSize: <size_specification> 2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), or EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: 24h
              retentionSize: 10GB

2. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 1

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

3. Save the file to apply the changes. The pods affected by the new configuration automatically restart.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Note

   If you configure cluster ID labels for metrics for the Prometheus instance that monitors user-defined projects, edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring namespace. Note that the Prometheus component is called prometheus in this config map and not prometheusK8s, which is the name used in the cluster-monitoring-config config map.

2. In the writeRelabelConfigs: section under data/config.yaml/prometheusK8s/remoteWrite, add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 1
             - <relabel_config> 2

   1

   Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   2

   Substitute the label configuration for the metrics sent to the remote write endpoint.

   The following sample shows how to forward a metric with the cluster ID label cluster_id in default platform monitoring:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels:
             - __tmp_openshift_cluster_id__ 1
             targetLabel: cluster_id 2
             action: replace 3

   1

   The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

   2

   Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

   3

   The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes to the ConfigMap object. The pods affected by the updated configuration automatically restart.

   Warning

   Saving changes to a monitoring ConfigMap object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforcedSampleLimit configuration to data/config.yaml to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Add the enforcedLabelLimit, enforcedLabelNameLengthLimit, and enforcedLabelValueLengthLimit configurations to data/config.yaml to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedLabelLimit: 500 1
         enforcedLabelNameLengthLimit: 50 2
         enforcedLabelValueLengthLimit: 600 3

   1

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   2

   Specifies the maximum length in characters of a label name. The default value is 0, which specifies no limit.

   3

   Specifies the maximum length in characters of a label value. The default value is 0, which specifies no limit.

4. Save the file to apply the changes. The limits are applied automatically.

   Note

   Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to the user-workload-monitoring-config ConfigMap object, the pods and other resources in the openshift-user-workload-monitoring project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 1
     namespace: ns1 2
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 3
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 4
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 5
         labels:
           severity: warning 6
       - alert: ApproachingEnforcedSamplesLimit 7
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 8
         expr: scrape_samples_scraped/50000 > 0.8 9
         for: 10m 10
         labels:
           severity: warning 11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule will be deployed.

   3

   The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.

   4

   The message that will be output when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.

   8

   The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml