1.2. Configuring the monitoring stack

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Put your configuration under data/config.yaml as key-value pair <component_name>: <component_configuration>:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         <configuration_for_the_component>

   Substitute <component> and <configuration_for_the_component> accordingly.

   For example, create this ConfigMap object to configure a Persistent Volume Claim (PVC) for Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate: spec: storageClassName: fast volumeMode: Filesystem resources: requests: storage: 40Gi

   Here, prometheusK8s defines the Prometheus component and the following lines define its configuration.

3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         nodeSelector:
           <node_key>: <node_value>
           <node_key>: <node_value>
           <...>

   Substitute <component> accordingly and substitute <node_key>: <node_value> with the map of key-value pairs that specifies the destination node. Often, only a single key-value pair is used.

   The component can only run on a node that has each of the specified key-value pairs as labels. The node can have additional labels as well.

   For example, to move components to the node that is labeled foo: bar, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusOperator: nodeSelector: foo: bar prometheusK8s: nodeSelector: foo: bar alertmanagerMain: nodeSelector: foo: bar kubeStateMetrics: nodeSelector: foo: bar grafana: nodeSelector: foo: bar telemeterClient: nodeSelector: foo: bar k8sPrometheusAdapter: nodeSelector: foo: bar
       openshiftStateMetrics:
         nodeSelector:
           foo: bar
       thanosQuerier:
         nodeSelector:
           foo: bar

3. Save the file to apply the changes. The components affected by the new configuration are moved to new nodes automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, a oc adm taint nodes node1 key1=value1:NoSchedule taint prevents the scheduler from placing pods in the foo: bar node. To make the alertmanagerMain component ignore that taint and to place alertmanagerMain in foo: bar normally, use this toleration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         nodeSelector:
           foo: bar
         tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule"

3. Save the file to apply the changes. The new component placement configuration is applied automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Define a map of labels you want to add for every metric under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           <key>: <value> 1

   1

   Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name of the new label and <value> is its value.

   警告

   Do not use prometheus or prometheus_replica as key names, because they are reserved and would be overwritten.

   For example, to add metadata about the region and environment to all time series and alerts, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           region: eu
           environment: prod

3. Save the file to apply the changes. The new configuration is applied automatically.