Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 2. Configuring the monitoring stack

The OpenShift Container Platform 4 installation program provides only a low number of configuration options before installation. Configuring most OpenShift Container Platform framework components, including the cluster monitoring stack, happens post-installation.

This section explains what configuration is supported, shows how to configure the monitoring stack, and demonstrates several common configuration scenarios.

2.1. Prerequisites
Copiar enlace

  • The monitoring stack imposes additional resource requirements. Consult the computing resources recommendations in Scaling the Cluster Monitoring Operator and verify that you have sufficient resources.

2.2. Maintenance and support for monitoring
Copiar enlace

The supported way of configuring OpenShift Container Platform Monitoring is by configuring it using the options described in this document. Do not use other configurations, as they are unsupported. Configuration paradigms might change across Prometheus releases, and such cases can only be handled gracefully if all configuration possibilities are controlled. If you use configurations other than those described in this section, your changes will disappear because the 

cluster-monitoring-operator
reconciles any differences. The Operator resets everything to the defined state by default and by design.

2.2.1. Support considerations for monitoring
Copiar enlace

The following modifications are explicitly not supported:

  • Creating additional ServiceMonitor, PodMonitor, and PrometheusRule objects in the openshift-* and kube-* projects.

  • Modifying any resources or objects deployed in the openshift-monitoring or openshift-user-workload-monitoring projects. The resources created by the OpenShift Container Platform monitoring stack are not meant to be used by any other resources, as there are no guarantees about their backward compatibility.

    Note

    The Alertmanager configuration is deployed as a secret resource in the 

    openshift-monitoring
    project. To configure additional routes for Alertmanager, you need to decode, modify, and then encode that secret. This procedure is a supported exception to the preceding statement.

  • Modifying resources of the stack. The OpenShift Container Platform monitoring stack ensures its resources are always in the state it expects them to be. If they are modified, the stack will reset them.
  • Deploying user-defined workloads to openshift-*, and kube-* projects. These projects are reserved for Red Hat provided components and they should not be used for user-defined workloads.
  • Modifying the monitoring stack Grafana instance.
  • Installing custom Prometheus instances on OpenShift Container Platform. A custom instance is a Prometheus custom resource (CR) managed by the Prometheus Operator.
  • Enabling symptom based monitoring by using the Probe custom resource definition (CRD) in Prometheus Operator.
  • Modifying Alertmanager configurations by using the AlertmanagerConfig CRD in Prometheus Operator.
Note

Backward compatibility for metrics, recording rules, or alerting rules is not guaranteed.

2.2.2. Support policy for monitoring Operators
Copiar enlace

Monitoring Operators ensure that OpenShift Container Platform monitoring resources function as designed and tested. If Cluster Version Operator (CVO) control of an Operator is overridden, the Operator does not respond to configuration changes, reconcile the intended state of cluster objects, or receive updates.

While overriding CVO control for an Operator can be helpful during debugging, this is unsupported and the cluster administrator assumes full control of the individual component configurations and upgrades.

Overriding the Cluster Version Operator

The 

spec.overrides
parameter can be added to the configuration for the CVO to allow administrators to provide a list of overrides to the behavior of the CVO for a component. Setting the 
spec.overrides[].unmanaged
parameter to 
true
for a component blocks cluster upgrades and alerts the administrator after a CVO override has been set: 

Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.
Warning

Setting a CVO override puts the entire cluster in an unsupported state and prevents the monitoring stack from being reconciled to its intended state. This impacts the reliability features built into Operators and prevents updates from being received. Reported issues must be reproduced after removing any overrides for support to proceed.

2.3. Preparing to configure the monitoring stack
Copiar enlace

You can configure the monitoring stack by creating and updating monitoring config maps.

2.3.1. Creating a cluster monitoring config map
Copiar enlace

To configure core OpenShift Container Platform monitoring components, you must create the 

cluster-monitoring-config
 
ConfigMap
object in the 
openshift-monitoring
project.

Note

When you save your changes to the 

cluster-monitoring-config
 
ConfigMap
object, some or all of the pods in the 
openshift-monitoring
project might be redeployed. It can sometimes take a while for these components to redeploy.

Prerequisites

  • You have access to the cluster as a user with the 
    cluster-admin
    role.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Check whether the 

    cluster-monitoring-config
     
    ConfigMap
    object exists: 

    $ oc -n openshift-monitoring get configmap cluster-monitoring-config

  2. If the 

    ConfigMap
    object does not exist:

    1. Create the following YAML manifest. In this example the file is called 

      cluster-monitoring-config.yaml
      : 

      apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |

    2. Apply the configuration to create the 

      ConfigMap
      object: 

      $ oc apply -f cluster-monitoring-config.yaml

2.3.2. Creating a user-defined workload monitoring config map
Copiar enlace

To configure the components that monitor user-defined projects, you must create the 

user-workload-monitoring-config
 
ConfigMap
object in the 
openshift-user-workload-monitoring
project.

Note

When you save your changes to the 

user-workload-monitoring-config
 
ConfigMap
object, some or all of the pods in the 
openshift-user-workload-monitoring
project might be redeployed. It can sometimes take a while for these components to redeploy. You can create and configure the config map before you first enable monitoring for user-defined projects, to prevent having to redeploy the pods often.

Prerequisites

  • You have access to the cluster as a user with the 
    cluster-admin
    role.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Check whether the 

    user-workload-monitoring-config
     
    ConfigMap
    object exists: 

    $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

  2. If the 

    user-workload-monitoring-config
     
    ConfigMap
    object does not exist:

    1. Create the following YAML manifest. In this example the file is called 

      user-workload-monitoring-config.yaml
      : 

      apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |

    2. Apply the configuration to create the 

      ConfigMap
      object: 

      $ oc apply -f user-workload-monitoring-config.yaml
      Note

      Configurations applied to the 

      user-workload-monitoring-config
       
      ConfigMap
      object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

2.4. Configuring the monitoring stack
Copiar enlace

In OpenShift Container Platform 4.8, you can configure the monitoring stack using the 

cluster-monitoring-config
or 
user-workload-monitoring-config
 
ConfigMap
objects. Config maps configure the Cluster Monitoring Operator (CMO), which in turn configures the components of the stack.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object.

    • To configure core OpenShift Container Platform monitoring components:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add your configuration under 

        data/config.yaml
        as a key-value pair 
        <component_name>: <component_configuration>
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>:
      <configuration_for_the_component>

        Substitute 

        <component>
        and 
        <configuration_for_the_component>
        accordingly.

        The following example 

        ConfigMap
        object configures a persistent volume claim (PVC) for Prometheus. This relates to the Prometheus instance that monitors core OpenShift Container Platform components only: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
        1 
        
      volumeClaimTemplate:
       spec:
         storageClassName: fast
         volumeMode: Filesystem
         resources:
           requests:
             storage: 40Gi
        1
        Defines the Prometheus component and the subsequent lines define its configuration.

    • To configure components that monitor user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add your configuration under 

        data/config.yaml
        as a key-value pair 
        <component_name>: <component_configuration>
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
      <configuration_for_the_component>

        Substitute 

        <component>
        and 
        <configuration_for_the_component>
        accordingly.

        The following example 

        ConfigMap
        object configures a data retention period and minimum container resource requests for Prometheus. This relates to the Prometheus instance that monitors user-defined projects only: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
        1 
        
      retention: 24h
        2 
        
      resources:
        requests:
          cpu: 200m
        3 
        
          memory: 2Gi
        4
        1
        Defines the Prometheus component and the subsequent lines define its configuration.
        2
        Configures a twenty-four hour data retention period for the Prometheus instance that monitors user-defined projects.
        3
        Defines a minimum resource request of 200 millicores for the Prometheus container.
        4
        Defines a minimum pod resource request of 2 GiB of memory for the Prometheus container.
        Note

        The Prometheus config map component is called 

        prometheusK8s
        in the 
        cluster-monitoring-config
         
        ConfigMap
        object and 
        prometheus
        in the 
        user-workload-monitoring-config
         
        ConfigMap
        object.

  2. Save the file to apply the changes to the 

    ConfigMap
    object. The pods affected by the new configuration are restarted automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

2.5. Configurable monitoring components
Copiar enlace

This table shows the monitoring components you can configure and the keys used to specify the components in the 

cluster-monitoring-config
and 
user-workload-monitoring-config
 
ConfigMap
objects:

Expand
Table 2.1. Configurable monitoring components
Componentcluster-monitoring-config config map keyuser-workload-monitoring-config config map key

Prometheus Operator 

prometheusOperator
 

prometheusOperator

Prometheus 

prometheusK8s
 

prometheus

Alertmanager 

alertmanagerMain

 

kube-state-metrics 

kubeStateMetrics

 

openshift-state-metrics 

openshiftStateMetrics

 

Grafana 

grafana

 

Telemeter Client 

telemeterClient

 

Prometheus Adapter 

k8sPrometheusAdapter

 

Thanos Querier 

thanosQuerier

 

Thanos Ruler

  

thanosRuler

Note

The Prometheus key is called 

prometheusK8s
in the 
cluster-monitoring-config
 
ConfigMap
object and 
prometheus
in the 
user-workload-monitoring-config
 
ConfigMap
object.

2.6. Moving monitoring components to different nodes
Copiar enlace

You can move any of the monitoring stack components to specific nodes.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To move a component that monitors core OpenShift Container Platform projects:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Specify the 

        nodeSelector
        constraint for the component under 
        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>:
      nodeSelector:
        <node_key>: <node_value>
        <node_key>: <node_value>
        <...>

        Substitute 

        <component>
        accordingly and substitute 
        <node_key>: <node_value>
        with the map of key-value pairs that specifies a group of destination nodes. Often, only a single key-value pair is used.

        The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.

        Important

        Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.

        Note

        If monitoring components remain in a 

        Pending
        state after configuring the 
        nodeSelector
        constraint, check the pod logs for errors relating to taints and tolerations.

        For example, to move monitoring components for core OpenShift Container Platform projects to specific nodes that are labeled 

        nodename: controlplane1
        , 
        nodename: worker1
        , 
        nodename: worker2
        , and 
        nodename: worker2
        , use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusOperator:
      nodeSelector:
        nodename: controlplane1
    prometheusK8s:
      nodeSelector:
        nodename: worker1
        nodename: worker2
    alertmanagerMain:
      nodeSelector:
        nodename: worker1
        nodename: worker2
    kubeStateMetrics:
      nodeSelector:
        nodename: worker1
    grafana:
      nodeSelector:
        nodename: worker1
    telemeterClient:
      nodeSelector:
        nodename: worker1
    k8sPrometheusAdapter:
      nodeSelector:
        nodename: worker1
        nodename: worker2
    openshiftStateMetrics:
      nodeSelector:
        nodename: worker1
    thanosQuerier:
      nodeSelector:
        nodename: worker1
        nodename: worker2

    • To move a component that monitors user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Specify the 

        nodeSelector
        constraint for the component under 
        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
      nodeSelector:
        <node_key>: <node_value>
        <node_key>: <node_value>
        <...>

        Substitute 

        <component>
        accordingly and substitute 
        <node_key>: <node_value>
        with the map of key-value pairs that specifies the destination nodes. Often, only a single key-value pair is used.

        The component can only run on nodes that have each of the specified key-value pairs as labels. The nodes can have additional labels as well.

        Important

        Many of the monitoring components are deployed by using multiple pods across different nodes in the cluster to maintain high availability. When moving monitoring components to labeled nodes, ensure that enough matching nodes are available to maintain resilience for the component. If only one label is specified, ensure that enough nodes contain that label to distribute all of the pods for the component across separate nodes. Alternatively, you can specify multiple labels each relating to individual nodes.

        Note

        If monitoring components remain in a 

        Pending
        state after configuring the 
        nodeSelector
        constraint, check the pod logs for errors relating to taints and tolerations.

        For example, to move monitoring components for user-defined projects to specific worker nodes labeled 

        nodename: worker1
        , 
        nodename: worker2
        , and 
        nodename: worker2
        , use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheusOperator:
      nodeSelector:
        nodename: worker1
    prometheus:
      nodeSelector:
        nodename: worker1
        nodename: worker2
    thanosRuler:
      nodeSelector:
        nodename: worker1
        nodename: worker2

  2. Save the file to apply the changes. The components affected by the new configuration are moved to the new nodes automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

2.7. Assigning tolerations to monitoring components
Copiar enlace

You can assign tolerations to any of the monitoring stack components to enable moving them to tainted nodes.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To assign tolerations to a component that monitors core OpenShift Container Platform projects:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Specify 

        tolerations
        for the component: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>:
      tolerations:
        <toleration_specification>

        Substitute 

        <component>
        and 
        <toleration_specification>
        accordingly.

        For example, 

        oc adm taint nodes node1 key1=value1:NoSchedule
        adds a taint to 
        node1
        with the key 
        key1
        and the value 
        value1
        . This prevents monitoring components from deploying pods on 
        node1
        unless a toleration is configured for that taint. The following example configures the 
        alertmanagerMain
        component to tolerate the example taint: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      tolerations:
      - key: "key1"
        operator: "Equal"
        value: "value1"
        effect: "NoSchedule"

    • To assign tolerations to a component that monitors user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Specify 

        tolerations
        for the component: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
      tolerations:
        <toleration_specification>

        Substitute 

        <component>
        and 
        <toleration_specification>
        accordingly.

        For example, 

        oc adm taint nodes node1 key1=value1:NoSchedule
        adds a taint to 
        node1
        with the key 
        key1
        and the value 
        value1
        . This prevents monitoring components from deploying pods on 
        node1
        unless a toleration is configured for that taint. The following example configures the 
        thanosRuler
        component to tolerate the example taint: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    thanosRuler:
      tolerations:
      - key: "key1"
        operator: "Equal"
        value: "value1"
        effect: "NoSchedule"

  2. Save the file to apply the changes. The new component placement configuration is applied automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

2.8. Configuring persistent storage
Copiar enlace

Running cluster monitoring with persistent storage means that your metrics are stored to a persistent volume (PV) and can survive a pod being restarted or recreated. This is ideal if you require your metrics or alerting data to be guarded from data loss. For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage.

Note

2.8.1. Persistent storage prerequisites
Copiar enlace

  • Dedicate sufficient local persistent storage to ensure that the disk does not become full. How much storage you need depends on the number of pods. For information on system requirements for persistent storage, see Prometheus database storage requirements.
  • Make sure you have a persistent volume (PV) ready to be claimed by the persistent volume claim (PVC), one PV for each replica. Because Prometheus has two replicas and Alertmanager has three replicas, you need five PVs to support the entire monitoring stack. The PVs should be available from the Local Storage Operator. This does not apply if you enable dynamically provisioned storage.
  • Use 
    Filesystem
    as the storage type value for the 
    volumeMode
    parameter when you configure the persistent volume.

  • Configure local persistent storage.

    Note

    If you use a local volume for persistent storage, do not use a raw block volume, which is described with 

    volumeMode: Block
    in the 
    LocalVolume
    object. Prometheus cannot use raw block volumes.

2.8.2. Configuring a local persistent volume claim
Copiar enlace

For monitoring components to use a persistent volume (PV), you must configure a persistent volume claim (PVC).

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To configure a PVC for a component that monitors core OpenShift Container Platform projects:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add your PVC configuration for the component under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>:
      volumeClaimTemplate:
        spec:
          storageClassName: <storage_class>
          resources:
            requests:
              storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify 

        volumeClaimTemplate
        .

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors core OpenShift Container Platform components: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      volumeClaimTemplate:
        spec:
          storageClassName: local-storage
          resources:
            requests:
              storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called 

        local-storage
        .

        The following example configures a PVC that claims local persistent storage for Alertmanager: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      volumeClaimTemplate:
        spec:
          storageClassName: local-storage
          resources:
            requests:
              storage: 10Gi

    • To configure a PVC for a component that monitors user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add your PVC configuration for the component under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
      volumeClaimTemplate:
        spec:
          storageClassName: <storage_class>
          resources:
            requests:
              storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify 

        volumeClaimTemplate
        .

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors user-defined projects: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      volumeClaimTemplate:
        spec:
          storageClassName: local-storage
          resources:
            requests:
              storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called 

        local-storage
        .

        The following example configures a PVC that claims local persistent storage for Thanos Ruler: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    thanosRuler:
      volumeClaimTemplate:
        spec:
          storageClassName: local-storage
          resources:
            requests:
              storage: 10Gi
        Note

        Storage requirements for the 

        thanosRuler
        component depend on the number of rules that are evaluated and how many samples each rule generates.

  2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

2.8.3. Modifying the retention time for Prometheus metrics data
Copiar enlace

By default, the OpenShift Container Platform monitoring stack configures the retention time for Prometheus data to be 15 days. You can modify the retention time to change how soon the data is deleted.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To modify the retention time for the Prometheus instance that monitors core OpenShift Container Platform projects:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add your retention time configuration under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      retention: <time_specification>

        Substitute 

        <time_specification>
        with a number directly followed by 
        ms
        (milliseconds), 
        s
        (seconds), 
        m
        (minutes), 
        h
        (hours), 
        d
        (days), 
        w
        (weeks), or 
        y
        (years).

        The following example sets the retention time to 24 hours for the Prometheus instance that monitors core OpenShift Container Platform components: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      retention: 24h

    • To modify the retention time for the Prometheus instance that monitors user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add your retention time configuration under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      retention: <time_specification>

        Substitute 

        <time_specification>
        with a number directly followed by 
        ms
        (milliseconds), 
        s
        (seconds), 
        m
        (minutes), 
        h
        (hours), 
        d
        (days), 
        w
        (weeks), or 
        y
        (years).

        The following example sets the retention time to 24 hours for the Prometheus instance that monitors user-defined projects: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      retention: 24h

  2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Developers can create labels to define attributes for metrics in the form of key-value pairs. The number of potential key-value pairs corresponds to the number of possible values for an attribute. An attribute that has an unlimited number of potential values is called an unbound attribute. For example, a 

customer_id
attribute is unbound because it has an infinite number of possible values.

Every assigned key-value pair has a unique time series. The use of many unbound attributes in labels can result in an exponential increase in the number of time series created. This can impact Prometheus performance and can consume a lot of disk space.

Cluster administrators can use the following measures to control the impact of unbound metrics attributes in user-defined projects:

  • Limit the number of samples that can be accepted per target scrape in user-defined projects
  • Create alerts that fire when a scrape sample threshold is reached or when the target cannot be scraped
Note

Limiting scrape samples can help prevent the issues caused by adding many unbound attributes to labels. Developers can also prevent the underlying cause by limiting the number of unbound attributes that they define for metrics. Using attributes that are bound to a limited set of possible values reduces the number of potential key-value pair combinations.

2.9.1. Setting a scrape sample limit for user-defined projects
Copiar enlace

You can limit the number of samples that can be accepted per target scrape in user-defined projects.

Warning

If you set a sample limit, no further sample data is ingested for that target scrape after the limit is reached.

Prerequisites

  • You have access to the cluster as a user with the 
    cluster-admin
    role, or as a user with the 
    user-workload-monitoring-config-edit
    role in the 
    openshift-user-workload-monitoring
    project.
  • You have created the 
    user-workload-monitoring-config
     
    ConfigMap
    object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    user-workload-monitoring-config
     
    ConfigMap
    object in the 
    openshift-user-workload-monitoring
    project: 

    $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

  2. Add the 

    enforcedSampleLimit
    configuration to 
    data/config.yaml
    to limit the number of samples that can be accepted per target scrape in user-defined projects: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      enforcedSampleLimit: 50000
    1
    1
    A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

  3. Save the file to apply the changes. The limit is applied automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to the 

    user-workload-monitoring-config
     
    ConfigMap
    object, the pods and other resources in the 
    openshift-user-workload-monitoring
    project might be redeployed. The running monitoring processes in that project might also be restarted.

2.9.2. Creating scrape sample alerts
Copiar enlace

You can create alerts that notify you when:

  • The target cannot be scraped or is not available for the specified 
    for
    duration
  • A scrape sample threshold is reached or is exceeded for the specified 
    for
    duration

Prerequisites

  • You have access to the cluster as a user with the 
    cluster-admin
    role, or as a user with the 
    user-workload-monitoring-config-edit
    role in the 
    openshift-user-workload-monitoring
    project.
  • You have enabled monitoring for user-defined projects.
  • You have created the 
    user-workload-monitoring-config
     
    ConfigMap
    object.
  • You have limited the number of samples that can be accepted per target scrape in user-defined projects, by using 
    enforcedSampleLimit
    .
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called 

    monitoring-stack-alerts.yaml
    : 

    apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: monitoring-stack-alerts
    1 
    
  namespace: ns1
    2 
    
spec:
  groups:
  - name: general.rules
    rules:
    - alert: TargetDown
    3 
    
      annotations:
        message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
          }} targets in {{ $labels.namespace }} namespace are down.'
    4 
    
      expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
        namespace, service)) > 10
      for: 10m
    5 
    
      labels:
        severity: warning
    6 
    
    - alert: ApproachingEnforcedSamplesLimit
    7 
    
      annotations:
        message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.'
    8 
    
      expr: scrape_samples_scraped/50000 > 0.8
    9 
    
      for: 10m
    10 
    
      labels:
        severity: warning
    11
    1
    Defines the name of the alerting rule.
    2
    Specifies the user-defined project where the alerting rule will be deployed.
    3
    The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.
    4
    The message that will be output when the TargetDown alert fires.
    5
    The conditions for the TargetDown alert must be true for this duration before the alert is fired.
    6
    Defines the severity for the TargetDown alert.
    7
    The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.
    8
    The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.
    9
    The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.
    10
    The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.
    11
    Defines the severity for the ApproachingEnforcedSamplesLimit alert.

  2. Apply the configuration to the user-defined project: 

    $ oc apply -f monitoring-stack-alerts.yaml

2.10. Attaching additional labels to your time series and alerts
Copiar enlace

Using the external labels feature of Prometheus, you can attach custom labels to all time series and alerts leaving Prometheus.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Define a map of labels you want to add for every metric under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        <key>: <value>
        1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning

        Do not use 

        prometheus
        or 
        prometheus_replica
        as key names, because they are reserved and will be overwritten.

        For example, to add metadata about the region and environment to all time series and alerts, use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        region: eu
        environment: prod

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Define a map of labels you want to add for every metric under 

        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        <key>: <value>
        1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning

        Do not use 

        prometheus
        or 
        prometheus_replica
        as key names, because they are reserved and will be overwritten.

        Note

        In the 

        openshift-user-workload-monitoring
        project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting 
        externalLabels
        for 
        prometheus
        in the 
        user-workload-monitoring-config
         
        ConfigMap
        object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        region: eu
        environment: prod

  2. Save the file to apply the changes. The new configuration is applied automatically.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

2.11. Setting log levels for monitoring components
Copiar enlace

You can configure the log level for Prometheus Operator, Prometheus, Thanos Querier, and Thanos Ruler.

Note

You cannot use this procedure to configure the log level for the Alertmanager component.

The following log levels can be applied to each of those components in the 

cluster-monitoring-config
and 
user-workload-monitoring-config
 
ConfigMap
objects: 

  • debug
    . Log debug, informational, warning, and error messages. 
  • info
    . Log informational, warning, and error messages. 
  • warn
    . Log warning and error messages only. 
  • error
    . Log error messages only.

The default log level is 

info
.

Prerequisites

  • If you are setting a log level for Prometheus Operator, Prometheus, or Thanos Querier in the openshift-monitoring project:

    • You have access to the cluster as a user with the 
      cluster-admin
      role.
    • You have created the 
      cluster-monitoring-config
       
      ConfigMap
      object.

  • If you are setting a log level for Prometheus Operator, Prometheus, or Thanos Ruler in the openshift-user-workload-monitoring project:

    • You have access to the cluster as a user with the 
      cluster-admin
      role, or as a user with the 
      user-workload-monitoring-config-edit
      role in the 
      openshift-user-workload-monitoring
      project.
    • You have created the 
      user-workload-monitoring-config
       
      ConfigMap
      object.
  • You have installed the OpenShift CLI (
    oc
    ).

Procedure

  1. Edit the 

    ConfigMap
    object:

    • To set a log level for a component in the openshift-monitoring project:

      1. Edit the 

        cluster-monitoring-config
         
        ConfigMap
        object in the 
        openshift-monitoring
        project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add 

        logLevel: <log_level>
        for a component under 
        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>:
        1 
        
      logLevel: <log_level>
        2
        1
        The monitoring component that you are applying a log level to.
        2
        The log level to apply to the component.

    • To set a log level for a component in the openshift-user-workload-monitoring project:

      1. Edit the 

        user-workload-monitoring-config
         
        ConfigMap
        object in the 
        openshift-user-workload-monitoring
        project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add 

        logLevel: <log_level>
        for a component under 
        data/config.yaml
        : 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
        1 
        
      logLevel: <log_level>
        2
        1
        The monitoring component that you are applying a log level to.
        2
        The log level to apply to the component.

  2. Save the file to apply the changes. The pods for the component restarts automatically when you apply the log-level change.

    Note

    Configurations applied to the 

    user-workload-monitoring-config
     
    ConfigMap
    object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

  3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the 

    prometheus-operator
    deployment in the 
    openshift-user-workload-monitoring
    project: 

    $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

    Example output

            - --log-level=debug

  4. Check that the pods for the component are running. The following example lists the status of pods in the 

    openshift-user-workload-monitoring
    project: 

    $ oc -n openshift-user-workload-monitoring get pods
    Note

    If an unrecognized 

    loglevel
    value is included in the 
    ConfigMap
    object, the pods for the component might not restart successfully.

2.12. Next steps
Copiar enlace

Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2026 Red HatVolver arriba