Chapter 5. Configuring pod topology spread constraints for monitoring

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values for the following settings under data/config.yaml/prometheusK8s to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: DoNotSchedule 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: prometheus

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values for the following settings under data/config.yaml/alertmanagermain to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: DoNotSchedule 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: alertmanager

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring namespace:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add values for the following settings under data/config.yaml/thanosRuler to configure pod topology spread constraints:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         topologySpreadConstraints:
         - maxSkew: 1 1
           topologyKey: monitoring 2
           whenUnsatisfiable: ScheduleAnyway 3
           labelSelector:
             matchLabels: 4
               app.kubernetes.io/name: thanos-ruler

   1

   Specify a numeric value for maxSkew, which defines the degree to which pods are allowed to be unevenly distributed. This field is required, and the value must be greater than zero. The value specified has a different effect depending on what value you specify for whenUnsatisfiable.

   2

   Specify a key of node labels for topologyKey. This field is required. Nodes that have a label with this key and identical values are considered to be in the same topology. The scheduler will try to put a balanced number of pods into each domain.

   3

   Specify a value for whenUnsatisfiable. This field is required. Available options are DoNotSchedule and ScheduleAnyway. Specify DoNotSchedule if you want the maxSkew value to define the maximum difference allowed between the number of matching pods in the target topology and the global minimum. Specify ScheduleAnyway if you want the scheduler to still schedule the pod but to give higher priority to nodes that might reduce the skew.

   4

   Specify a value for matchLabels. This value is used to identify the set of matching pods to which to apply the constraints.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to the user-workload-monitoring-config config map, the pods and other resources in the openshift-user-workload-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   • To set a log level for a component in the openshift-monitoring project:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.

        2

        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

   • To set a log level for a component in the openshift-user-workload-monitoring project:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add logLevel: <log_level> for a component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              logLevel: <log_level> 2

        1

        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are alertmanager, prometheus, prometheusOperator, and thanosRuler.

        2

        The log level to apply to the component. The available values are error, warn, info, and debug. The default value is info.

2. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml | grep "log-level"

   Example output

           - --log-level=debug

4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring get pods

   Note

   If an unrecognized logLevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a thanosQuerier section under data/config.yaml and add values as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       thanosQuerier:
         enableRequestLogging: <value> 1
         logLevel: <value> 2

   1

   Set the value to true to enable logging and false to disable logging. The default value is false.

   2

   Set the value to debug, info, warn, or error. If no value exists for logLevel, the log level defaults to error.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. Verify that the Thanos Querier pods are running. The following sample command lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

2. Run a test query using the following sample commands as a model:

   $ token=`oc create token prometheus-k8s -n openshift-monitoring`
   $ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster_version'

3. Run the following command to read the query log:

   $ oc -n openshift-monitoring logs <thanos_querier_pod_name> -c thanos-query

   Note

   Because the thanos-querier pods are highly available (HA) pods, you might be able to see logs in only one pod.

4. After you examine the logged query information, disable query logging by changing the enableRequestLogging value to false in the config map.

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add profile: in the k8sPrometheusAdapter/audit section under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         audit:
           profile: <audit_log_level> 1

   1

   The audit log level to apply to the Prometheus Adapter.

3. Set the audit log level by using one of the following values for the profile: parameter:

   • None: Do not log events.
   • Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. Metadata is the default audit log level.
   • Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.
   • RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.
4. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Verification

1. In the config map, under k8sPrometheusAdapter/audit/profile, set the log level to Request and save the file.

2. Confirm that the pods for the Prometheus Adapter are running. The following example lists the status of pods in the openshift-monitoring project:

   $ oc -n openshift-monitoring get pods

3. Confirm that the audit log level and audit log file path are correctly configured:

   $ oc -n openshift-monitoring get deploy prometheus-adapter -o yaml

   Example output

   ...
     - --audit-policy-file=/etc/audit/request-profile.yaml
     - --audit-log-path=/var/log/adapter/audit.log

4. Confirm that the correct log level has been applied in the prometheus-adapter deployment in the openshift-monitoring project:

   $ oc -n openshift-monitoring exec deploy/prometheus-adapter -c prometheus-adapter -- cat /etc/audit/request-profile.yaml

   Example output

   "apiVersion": "audit.k8s.io/v1"
   "kind": "Policy"
   "metadata":
     "name": "Request"
   "omitStages":
   - "RequestReceived"
   "rules":
   - "level": "Request"

   Note

   If you enter an unrecognized profile value for the Prometheus Adapter in the ConfigMap object, no changes are made to the Prometheus Adapter, and an error is logged by the Cluster Monitoring Operator.

5. Review the audit log for the Prometheus Adapter:

   $ oc -n openshift-monitoring exec -c <prometheus_adapter_pod_name> -- cat /var/log/adapter/audit.log

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add enabled: false for the alertmanagerMain component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         enabled: false

3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.