Chapter 6. Using pod topology spread constraints for monitoring

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the following settings under the data/config.yaml field to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       <component>: 1
         topologySpreadConstraints:
         - maxSkew: <n> 2
           topologyKey: <key> 3
           whenUnsatisfiable: <value> 4
           labelSelector: 5
             <match_option>

   1

   Specify a name of the component for which you want to set up pod topology spread constraints.

   2

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed.

   3

   Specify a key of node labels for topologyKey. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler tries to put a balanced number of pods into each domain.

   4

   Specify a value for whenUnsatisfiable. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   5

   Specify labelSelector to find matching pods. Pods that match this label selector are counted to determine the number of pods in their corresponding topology domain.

   Example configuration for Thanos Ruler

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: monitoring
           whenUnsatisfiable: ScheduleAnyway
           labelSelector:
             matchLabels:
               app.kubernetes.io/name: thanos-ruler

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to the user-workload-monitoring-config config map, the pods and other resources in the openshift-user-workload-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: <log_level> for a component under data/config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          <component>: 1
            logLevel: <log_level> 2

      1

      The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are alertmanager, prometheus, prometheusOperator, and thanosRuler.

      2

      The log level to apply to the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods for the component restart automatically when you apply the log-level change.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml | grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized logLevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add queryLogFile: <path> for prometheus under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         queryLogFile: <path> 1

   1

   The full path to the file in which queries will be logged.

3. Save the file to apply the changes.

   Warning

   When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

4. Verify that the pods for the component are running. The following example command lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

5. Read the query log:

   $ oc -n openshift-user-workload-monitoring exec prometheus-user-workload-0 -- cat <path>

   Important

   Revert the setting in the config map after you have examined the logged query information.