Chapter 3. Configuring external alertmanager instances

Procedure

1. Edit the

   ConfigMap

   object:

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Define a map of labels you want to add for every metric under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                <key>: <value> 

        1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use

        prometheus

        or

        prometheus_replica

        as key names, because they are reserved and will be overwritten.

        For example, to add metadata about the region and environment to all time series and alerts, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              externalLabels:
                region: eu
                environment: prod

   • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Define a map of labels you want to add for every metric under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                <key>: <value> 

        1

        1

        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.

        Warning

        Do not use

        prometheus

        or

        prometheus_replica

        as key names, because they are reserved and will be overwritten.

        Note

        In the

        openshift-user-workload-monitoring

        project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting

        externalLabels

        for

        prometheus

        in the

        user-workload-monitoring-config

        ConfigMap

        object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              externalLabels:
                region: eu
                environment: prod

2. Save the file to apply the changes. The new configuration is applied automatically.

   Note

   Configurations applied to the

   user-workload-monitoring-config

   ConfigMap

   object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the

   ConfigMap

   object:

   • To set a log level for a component in the openshift-monitoring project:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add

        logLevel: <log_level>

        for a component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              logLevel: <log_level> 

        2

        1

        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

   • To set a log level for a component in the openshift-user-workload-monitoring project:

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add

        logLevel: <log_level>

        for a component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              logLevel: <log_level> 

        2

        1

        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are prometheus, prometheusOperator, and thanosRuler.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods for the component restarts automatically when you apply the log-level change.

   Note

   Configurations applied to the

   user-workload-monitoring-config

   ConfigMap

   object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the

   prometheus-operator

   deployment in the

   openshift-user-workload-monitoring

   project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the

   openshift-user-workload-monitoring

   project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized

   loglevel

   value is included in the

   ConfigMap

   object, the pods for the component might not restart successfully.

1. Edit the

   cluster-monitoring-config

   ConfigMap

   object in the

   openshift-monitoring

   project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a

   thanosQuerier

   section under

   data/config.yaml

   and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 

   1

   
         logLevel: <value> 

   2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes.

   Warning

   When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the

   openshift-monitoring

   project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the

   thanos-querier

   pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the

   enableRequestLogging

   value to

   false

   in the config map.