Chapter 1. Cluster monitoring

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Put your configuration under data/config.yaml as key-value pair <component_name>: <component_configuration>:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         <configuration_for_the_component>

   Substitute <component> and <configuration_for_the_component> accordingly.

   For example, create this ConfigMap object to configure a Persistent Volume Claim (PVC) for Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate: spec: storageClassName: fast volumeMode: Filesystem resources: requests: storage: 40Gi

   Here, prometheusK8s defines the Prometheus component and the following lines define its configuration.

3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         nodeSelector:
           <node_key>: <node_value>
           <node_key>: <node_value>
           <...>

   Substitute <component> accordingly and substitute <node_key>: <node_value> with the map of key-value pairs that specifies the destination node. Often, only a single key-value pair is used.

   The component can only run on a node that has each of the specified key-value pairs as labels. The node can have additional labels as well.

   For example, to move components to the node that is labeled foo: bar, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusOperator: nodeSelector: foo: bar prometheusK8s: nodeSelector: foo: bar alertmanagerMain: nodeSelector: foo: bar kubeStateMetrics: nodeSelector: foo: bar grafana: nodeSelector: foo: bar telemeterClient: nodeSelector: foo: bar k8sPrometheusAdapter: nodeSelector: foo: bar
       openshiftStateMetrics:
         nodeSelector:
           foo: bar
       thanosQuerier:
         nodeSelector:
           foo: bar

3. Save the file to apply the changes. The components affected by the new configuration are moved to new nodes automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify tolerations for the component:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         tolerations:
           <toleration_specification>

   Substitute <component> and <toleration_specification> accordingly.

   For example, a oc adm taint nodes node1 key1=value1:NoSchedule taint prevents the scheduler from placing pods in the foo: bar node. To make the alertmanagerMain component ignore that taint and to place alertmanagerMain in foo: bar normally, use this toleration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         nodeSelector:
           foo: bar
         tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule"

3. Save the file to apply the changes. The new component placement configuration is applied automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Define a map of labels you want to add for every metric under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           <key>: <value> 1

   1

   Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name of the new label and <value> is its value.

   Warning

   Do not use prometheus or prometheus_replica as key names, because they are reserved and would be overwritten.

   For example, to add metadata about the region and environment to all time series and alerts, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         externalLabels:
           region: eu
           environment: prod

3. Save the file to apply the changes. The new configuration is applied automatically.

1. Filtering alerts by their names.
2. Filtering the alerts by their states. To fire, some alerts need a certain condition to be true for the duration of a timeout. If a condition of an alert is currently true, but the timeout has not been reached, such an alert is in the Pending state.
3. Alert name.
4. Description of an alert.
5. Value of the Severity label of the alert.
6. Current state of the alert and when the alert went into this state.
7. Actions you can do with the alert.

1. Creating a silence for an alert.
2. Filtering silences by their name.
3. Filtering silences by their states. If a silence is pending, it is currently not active because it is scheduled to start at a later time. If a silence expired, it is no longer active because it has reached its end time.
4. Description of a silence. It includes the silence name and the specification of alerts that it matches.
5. Number of alerts that are being silenced by the silence.
6. Current state of the silence. For active silences, it shows when the silence ends. For pending silences, it shows when the silence starts.
7. User that created the silence.
8. Actions you can do with a silence.

1. Filtering Alerting Rules by their name.
2. Filtering Alerting Rules by their states.
3. Alerting Rule name.
4. Value of the Severity label of the Alerting Rule.
5. State of the Alert defined by the Alerting Rule.

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring Alerting Alerts page.
2. Optional: Filter the alerts by name using the Filter Alerts by name field.
3. Optional: Filter the alerts by state using one or more of the state buttons Firing, Silenced, Pending.
4. Optional: Sort the alerts by clicking one or more of the Name, Severity, and State column headers.

5. After you see the alert you need, you can see either details of the alert or details of its governing alerting rule.

   To see alert details, click on the name of the alert. The page that opens has the graph with timeseries of the alert. It also has information about the alert, including:

   • A link to its governing alerting rule
   • Description of the alert
   • Labels attached to the alert

   To see alerting rule details, click the button in the last column and select View Alerting Rule. The page that opens has information about the alerting rule, including:

   • Alerting rule name, severity, and description
   • The expression that defines the condition for firing the alert
   • The time for which the condition should be true for an alert to fire
   • Graph for each alert governed by the alerting rule, showing the value with which the alert is firing
   • Table of all alerts governed by the alerting rule

1. Navigate to the Monitoring Alerting Silences page of the OpenShift Container Platform web console.
2. Click Create Silence.
3. Populate the Create Silence form.
4. To create the silence, click Silence.

1. Navigate to the Monitoring Alerting Alerts page of the OpenShift Container Platform web console.
2. For the alert that you want to silence, click the button in the last column and click Silence Alert. The Create Silence form will appear with prepopulated specification of the chosen alert.
3. Optional: Modify the silence.
4. To create the silence, click Silence.

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring Alerting Silences page.
2. Optional: Filter the silences by name using the Filter Silences by name field.
3. Optional: Filter the silences by state using one or more of the state buttons Active, Pending, Expired.
4. Optional: Sort the silences by clicking one or more of the Name, Firing Alerts, State, and Creator column headers.

5. After you see the silence, you can click its name to see the details, including:

   • Alert specification
   • State
   • Start time
   • End time
   • Number and list of firing alerts

Procedure

1. Navigate to the Monitoring Alerting Silences page.

2. For the silence you want to modify, click the button in the last column and click Edit silence.

   Alternatively, you can click Actions Edit Silence in the Silence Overview screen for a particular silence.

3. In the Edit Silence screen, enter your changes and click the Save button. This will expire the existing silence and create one with the chosen configuration.

Procedure

1. Navigate to the Monitoring Alerting Silences page.

2. For the silence you want to expire, click the button in the last column and click Expire Silence.

   Alternatively, you can click the Actions Expire Silence button in the Silence Overview page for a particular silence.

3. Confirm by clicking Expire Silence. This expires the silence.

1. Navigate to the Administration Cluster Settings Global Configuration Alertmanager YAML page of the web console.
2. Modify the YAML configuration file.
3. Click Save.

1. Actions.

   • Add query.
   • Expand or collapse all query tables.
   • Delete all queries.
2. Hide the plot.
3. The interactive plot.
4. The catalog of available metrics.
5. Add query.
6. Run queries.
7. Query forms.
8. Expand or collapse the form.
9. The query.
10. Clear query.
11. Enable or disable query.

12. Actions for a specific query.

    • Enable or disable query.
    • Show or hide all series of the query from the plot.
    • Delete query.
13. The metrics table for a query.
14. Color assigned to the graph of the metric. Clicking the square shows or hides the metric’s graph.

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring Metrics page.

2. In the query field, enter your PromQL query.

   • To show all available metrics and PromQL functions, click Insert Metric at Cursor.
3. For multiple queries, click Add Query.
4. For deleting queries, click for the query, then select Delete query.
5. For keeping but not running a query, click the Disable query button.

6. Once you finish creating queries, click the Run Queries button. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note

   Queries that operate on large amounts of data might timeout or overload the browser when drawing timeseries graphs. To avoid this, hide the graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.

7. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

Procedure

1. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown.

   • To hide all metrics from a query, click for the query and click Hide all series.
   • To hide a specific metric, go to the query table and click the colored square near the metric name.

2. To zoom into the plot and change the shown time range, do one of the following:

   • Visually select the time range by clicking and dragging on the plot horizontally.
   • Use the menu in the left upper corner to select the time range.

   To reset the time range, click Reset Zoom.

3. To display outputs of all queries at a specific point in time, hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
4. For more detailed information about metrics of a specific query, expand the table of that query using the drop-down button. Every metric is shown with its current value.
5. To hide the plot, click Hide Graph.

Procedure

1. Navigate to the OpenShift Container Platform web console and authenticate.

2. To access Prometheus, navigate to the "Monitoring" "Metrics" page.

   To access the Alerting UI, navigate to the "Monitoring" "Alerting" page.

   To access Grafana, navigate to the "Monitoring" "Dashboards" page.

Procedure

1. Run:

   $ oc -n openshift-monitoring get routes

   Example output

   NAME                HOST/PORT                                                     ...
   alertmanager-main   alertmanager-main-openshift-monitoring.apps._url_.openshift.com ...
   grafana             grafana-openshift-monitoring.apps._url_.openshift.com           ...
   prometheus-k8s      prometheus-k8s-openshift-monitoring.apps._url_.openshift.com    ...

2. Prepend https:// to the address, you cannot access web UIs using unencrypted connection.

   For example, this is the resulting URL for Alertmanager:

   https://alertmanager-main-openshift-monitoring.apps._url_.openshift.com

3. Navigate to the address using a web browser and authenticate.