Chapter 2. Configuring performance and scalability for core platform monitoring

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node_name> <node_label>

   where:

   <node_name>

   Specifies the name of the node where you want to add the label.

   <node_label>

   Specifies the name of the wanted label.

2. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

3. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       # ...
       <component>:
         nodeSelector:
           <node_label_1>
           <node_label_2>
       # ...

   where:

   <component>

   Specifies the monitoring stack component.

   <node_label_1>

   Specifies the label you added to the node.

   <node_label_2>

   Optional: Specifies additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

   Note

   If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

4. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, oc adm taint nodes node1 key1=value1:NoSchedule adds a taint to node1 with the key key1 and the value value1. This prevents monitoring components from deploying pods on node1 unless a toleration is configured for that taint. The following example configures the alertmanagerMain component to tolerate the example taint:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         tolerations:
         - key: "key1"
           operator: "Equal"
           value: "value1"
           effect: "NoSchedule"

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values to define resource limits and requests for each component you want to configure.

   Important

   Ensure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusK8s:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosQuerier:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperator:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       metricsServer:
         resources:
           requests:
             cpu: 10m
             memory: 50Mi
           limits:
             cpu: 50m
             memory: 500Mi
       kubeStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       telemeterClient:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       openshiftStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       nodeExporter:
         resources:
           limits:
             cpu: 50m
             memory: 150Mi
           requests:
             cpu: 20m
             memory: 50Mi
       monitoringPlugin:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperatorAdmissionWebhook:
         resources:
           limits:
             cpu: 50m
             memory: 100Mi
           requests:
             cpu: 20m
             memory: 50Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.