Chapter 4. Configuring pod topology spread constraints

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the retention time and size configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: <time_specification> 1
         retentionSize: <size_specification> 2

   1

   The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

   2

   The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).

   The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance:

   Example of setting retention time for Prometheus

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: 24h
         retentionSize: 10GB

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add logLevel: <log_level> for a component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>: 1
         logLevel: <log_level> 2

   1

   The monitoring stack component for which you are setting a log level. Available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

   2

   The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

4. Confirm that the log level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level for the prometheus-operator deployment:

   $ oc -n openshift-monitoring get deploy prometheus-operator -o yaml | grep "log-level"

   Example output

           - --log-level=debug

5. Check that the pods for the component are running. The following example lists the status of pods:

   $ oc -n openshift-monitoring get pods

   Note

   If an unrecognized logLevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the queryLogFile parameter for Prometheus under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         queryLogFile: <path> 1

   1

   Add the full path to the file in which queries will be logged.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

4. Verify that the pods for the component are running. The following sample command lists the status of pods:

   $ oc -n openshift-monitoring get pods

   Example output

   ...
   prometheus-operator-567c9bc75c-96wkj   2/2     Running   0          62m
   prometheus-k8s-0                       6/6     Running   1          57m
   prometheus-k8s-1                       6/6     Running   1          57m
   thanos-querier-56c76d7df4-2xkpc        6/6     Running   0          57m
   thanos-querier-56c76d7df4-j5p29        6/6     Running   0          57m
   ...

5. Read the query log:

   $ oc -n openshift-monitoring exec prometheus-k8s-0 -- cat <path>

   Important

   Revert the setting in the config map after you have examined the logged query information.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a thanosQuerier section under data/config.yaml and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 1
         logLevel: <value> 2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the thanos-querier pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the enableRequestLogging value to false in the config map.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s, as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 1
           <endpoint_authentication_credentials> 2

   1

   The URL of the remote write endpoint.

   2

   The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

3. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs:
           - <your_write_relabel_configs> 1

   1

   Add configuration for metrics that you want to send to the remote endpoint.

   Example of forwarding a single metric called my_metric

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__]
             regex: 'my_metric'
             action: keep

   Example of forwarding metrics called my_metric_1 and my_metric_2 in my_namespace namespace

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels: [__name__,namespace]
             regex: '(my_metric_1|my_metric_2);my_namespace'
             action: keep

4. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. In the writeRelabelConfigs: section under data/config.yaml/prometheusK8s/remoteWrite, add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 1
             - <relabel_config> 2

   1

   Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   2

   Substitute the label configuration for the metrics sent to the remote write endpoint.

   The following sample shows how to forward a metric with the cluster ID label cluster_id:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels:
             - __tmp_openshift_cluster_id__ 1
             targetLabel: cluster_id 2
             action: replace 3

   1

   The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

   2

   Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

   3

   The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes. The new configuration is applied automatically.