Chapter 3. Configuring core platform monitoring

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a value for enforcedBodySizeLimit to data/config.yaml/prometheusK8s to limit the body size that can be accepted per target scrape:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |-
       prometheusK8s:
         enforcedBodySizeLimit: 40MB 1

   1

   Specify the maximum body size for scraped metrics targets. This enforcedBodySizeLimit example limits the uncompressed size per target scrape to 40 megabytes. Valid numeric values use the Prometheus data size format: B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes). The default value is 0, which specifies no limit. You can also set the value to automatic to calculate the limit automatically based on cluster capacity.

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the metrics collection profile setting under data/config.yaml/prometheusK8s:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         collectionProfile: <metrics_collection_profile_name> 1

   1

   The name of the metrics collection profile. The available values are full or minimal. If you do not specify a value or if the collectionProfile key name does not exist in the config map, the default setting of full is used.

   The following example sets the metrics collection profile to minimal for the core platform instance of Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         collectionProfile: minimal

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the following settings under the data/config.yaml field to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>: 1
         topologySpreadConstraints:
         - maxSkew: <n> 2
           topologyKey: <key> 3
           whenUnsatisfiable: <value> 4
           labelSelector: 5
             <match_option>

   1

   Specify a name of the component for which you want to set up pod topology spread constraints.

   2

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed.

   3

   Specify a key of node labels for topologyKey. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler tries to put a balanced number of pods into each domain.

   4

   Specify a value for whenUnsatisfiable. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   5

   Specify labelSelector to find matching pods. Pods that match this label selector are counted to determine the number of pods in their corresponding topology domain.

   Example configuration for Prometheus

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: monitoring
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels:
               app.kubernetes.io/name: prometheus

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the retention time and size configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: <time_specification> 1
         retentionSize: <size_specification> 2

   1

   The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

   2

   The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).

   The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance:

   Example of setting retention time for Prometheus

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: 24h
         retentionSize: 10GB

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add logLevel: <log_level> for a component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>: 1
         logLevel: <log_level> 2

   1

   The monitoring stack component for which you are setting a log level. Available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

   2

   The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

4. Confirm that the log level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level for the prometheus-operator deployment:

   $ oc -n openshift-monitoring get deploy prometheus-operator -o yaml | grep "log-level"

   Example output

           - --log-level=debug

5. Check that the pods for the component are running. The following example lists the status of pods:

   $ oc -n openshift-monitoring get pods

   Note

   If an unrecognized logLevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the queryLogFile parameter for Prometheus under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         queryLogFile: <path> 1

   1

   Add the full path to the file in which queries will be logged.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

4. Verify that the pods for the component are running. The following sample command lists the status of pods:

   $ oc -n openshift-monitoring get pods

   Example output

   ...
   prometheus-operator-567c9bc75c-96wkj   2/2     Running   0          62m
   prometheus-k8s-0                       6/6     Running   1          57m
   prometheus-k8s-1                       6/6     Running   1          57m
   thanos-querier-56c76d7df4-2xkpc        6/6     Running   0          57m
   thanos-querier-56c76d7df4-j5p29        6/6     Running   0          57m
   ...

5. Read the query log:

   $ oc -n openshift-monitoring exec prometheus-k8s-0 -- cat <path>

   Important

   Revert the setting in the config map after you have examined the logged query information.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a thanosQuerier section under data/config.yaml and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 1
         logLevel: <value> 2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the thanos-querier pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the enableRequestLogging value to false in the config map.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s, as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 1
           <endpoint_authentication_credentials> 2

   1

   The URL of the remote write endpoint.

   2

   The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

3. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs:
           - <your_write_relabel_configs> 1

   1

   Add configuration for metrics that you want to send to the remote endpoint.

   Example of forwarding a single metric called my_metric

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__]
             regex: 'my_metric'
             action: keep

   Example of forwarding metrics called my_metric_1 and my_metric_2 in my_namespace namespace

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__,namespace]
             regex: '(my_metric_1|my_metric_2);my_namespace'
             action: keep

4. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. In the writeRelabelConfigs: section under data/config.yaml/prometheusK8s/remoteWrite, add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 1
             - <relabel_config> 2

   1

   Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   2

   Substitute the label configuration for the metrics sent to the remote write endpoint.

   The following sample shows how to forward a metric with the cluster ID label cluster_id:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels:
             - __tmp_openshift_cluster_id__ 1
             targetLabel: cluster_id 2
             action: replace 3

   1

   The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

   2

   Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

   3

   The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add an additionalAlertmanagerConfigs section with configuration details under data/config.yaml/prometheusK8s:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         additionalAlertmanagerConfigs:
         - <alertmanager_specification> 1

   1

   Substitute <alertmanager_specification> with authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig).

   The following sample config map configures an additional Alertmanager for Prometheus by using a bearer token with client TLS authentication:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         additionalAlertmanagerConfigs:
         - scheme: https
           pathPrefix: /
           timeout: "30s"
           apiVersion: v1
           bearerToken:
             name: alertmanager-bearer-token
             key: token
           tlsConfig:
             key:
               name: alertmanager-tls
               key: tls.key
             cert:
               name: alertmanager-tls
               key: tls.crt
             ca:
               name: alertmanager-tls
               key: tls.ca
           staticConfigs:
           - external-alertmanager1-remote.com
           - external-alertmanager1-remote2.com

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Define labels you want to add for every metric under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           <key>: <value> 1

   1

   Substitute <key>: <value> with key-value pairs where <key> is a unique name for the new label and <value> is its value.

   Warning
   • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
   • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.

   For example, to add metadata about the region and environment to all time series and alerts, use the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           region: eu
           environment: prod

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.