Chapter 1. Cluster monitoring

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If it does not exist, create it:

   $ oc -n openshift-monitoring create configmap cluster-monitoring-config

3. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

4. Create the data section if it does not exist yet:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Put your configuration under data/config.yaml as key-value pair <component_name>: <component_configuration>:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       component:
         configuration for the component

   Substitute <component> and <configuration for the component> accordingly.

   For example, create this ConfigMap to configure a Persistent Volume Claim (PVC) for Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate: spec: storageClassName: fast volumeMode: filesystem resources: requests: storage: 40Gi

   Here, prometheusK8s defines the Prometheus component and the following lines define its configuration.

3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Specify the nodeSelector constraint for the component under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       component:
         nodeSelector:
           node key: node value
           node key: node value
           ...

   Substitute <component> accordingly and substitute <node key>: <node value> with the map of key-value pairs that specifies the destination node. Often, only a single key-value pair is used.

   The component can only run on a node that has each of the specified key-value pairs as labels. The node can have additional labels as well.

   For example, to move components to the node that is labeled foo: bar, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusOperator: nodeSelector: foo: bar prometheusK8s: nodeSelector: foo: bar alertmanagerMain: nodeSelector: foo: bar kubeStateMetrics: nodeSelector: foo: bar grafana: nodeSelector: foo: bar telemeterClient: nodeSelector: foo: bar k8sPrometheusAdapter: nodeSelector: foo: bar

3. Save the file to apply the changes. The components affected by the new configuration are moved to new nodes automatically.

1. Filtering alerts by their names.
2. Filtering the alerts by their states. To fire, some alerts need a certain condition to be true for the duration of a timeout. If a condition of an alert is currently true, but the timeout has not been reached, such an alert is in the Pending state.
3. Alert name.
4. Description of an alert.
5. Current state of the alert and when the alert went into this state.
6. Value of the Severity label of the alert.
7. Actions you can do with the alert.

1. Creating a silence for an alert.
2. Filtering silences by their name.
3. Filtering silences by their states. If a silence is pending, it is currently not active because it is scheduled to start at a later time. If a silence expired, it is no longer active because it has reached its end time.
4. Description of a silence. It includes the specification of alerts that it matches.
5. Current state of the silence. For active silences, it shows when it ends, and for pending silences, it shows when it starts.
6. Number of alerts that are being silenced by the silence.
7. Actions you can do with a silence.

Procedure

1. Open the OpenShift Container Platform web console and navigate to Monitoring Alerts.
2. Optional: Filter the alerts by name using the Filter alerts by name field.
3. Optional: Filter the alerts by state using one or more of the state buttons Firing, Silenced, Pending, Not firing.
4. Optional: Sort the alerts by clicking one or more of the Name, State, and Severity column headers.

5. After you see the alert, you can see either details of the alert or details of its governing alerting rule.

   To see alert details, click on the name of the alert. This is the page with alert details:

   

   The page has the graph with timeseries of the alert. It also has information about the alert, including:

   • A link to its governing alerting rule
   • Description of the alert

   To see alerting rule details, click the button in the last column and select View Alerting Rule. This is the page with alerting rule details:

   

   The page has information about the alerting rule, including:

   • Alerting rule name, severity, and description
   • The expression that defines the condition for firing the alert
   • The time for which the condition should be true for an alert to fire
   • Graph for each alert governed by the alerting rule, showing the value with which the alert is firing
   • Table of all alerts governed by the alerting rule

1. Navigate to the Monitoring Silences page of the OpenShift Container Platform web console.
2. Click Create Silence.
3. Populate the Create Silence form.
4. To create the silence, click Create.

1. Navigate to the Monitoring Alerts page of the OpenShift Container Platform web console.
2. For the alert that you want to silence, click the button in the last column and click Silence Alert. The Create Silence form will appear with prepopulated specification of the chosen alert.
3. Optional: Modify the silence.
4. To create the silence, click Create.

Procedure

1. Open the OpenShift Container Platform web console and navigate to Monitoring Silences.
2. Optional: Filter the silences by name using the Filter Silences by name field.
3. Optional: Filter the silences by state using one or more of the state buttons Active, Pending, Expired.
4. Optional: Sort the silences by clicking one or more of the Name, State, and Firing alerts column headers.

5. After you see the silence, you can click its name to see the details, including:

   • Alert specification
   • State
   • Start time
   • End time
   • Number and list of firing alerts

Procedure

1. Navigate to the Monitoring Silences screen.

2. For the silence you want to modify, click the button in the last column and click Edit silence.

   Alternatively, you can click Actions Edit Silence in the Silence Overview screen for a particular silence.

3. In the Edit Silence screen, enter your changes and click the Save button. This will expire the existing silence and create one with the chosen configuration.

Procedure

1. Navigate to the Monitoring Silences page.

2. For the silence you want to expire, click the button in the last column and click Expire Silence.

   Alternatively, you can click the Actions Expire Silence button in the Silence Overview page for a particular silence.

3. Confirm by clicking Expire Silence. This expires the silence.

Procedure

1. Navigate to the OpenShift Container Platform Web console and authenticate.

2. To access Prometheus, navigate to "Monitoring" "Metrics".

   To access the Alerting UI, navigate to "Monitoring" "Alerts" or "Monitoring" "Silences".

   To access Grafana, navigate to "Monitoring" "Dashboards".

Procedure

1. Run:

   $ oc -n openshift-monitoring get routes
   NAME                HOST/PORT                                                     ...
   alertmanager-main   alertmanager-main-openshift-monitoring.apps.url.openshift.com ...
   grafana             grafana-openshift-monitoring.apps.url.openshift.com           ...
   prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.url.openshift.com    ...

2. Prepend https:// to the address, you cannot access web UIs using unencrypted connection.

   For example, this is the resulting URL for Alertmanager:

   https://alertmanager-main-openshift-monitoring.apps.url.openshift.com

3. Navigate to the address using a Web browser and authenticate.