Monitoring

Procedure

1. Check whether the cluster-monitoring-config ConfigMap object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called cluster-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Check whether the user-workload-monitoring-config ConfigMap object exists:

   $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

2. If the user-workload-monitoring-config ConfigMap object does not exist:

   1. Create the following YAML manifest. In this example the file is called user-workload-monitoring-config.yaml:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the ConfigMap object:

      $ oc apply -f user-workload-monitoring-config.yaml

      Note

      Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node-name> <node-label>

2. Edit the ConfigMap object:

   • To move a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

   • To move a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify the node labels for the nodeSelector constraint for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              nodeSelector:
                <node-label-1> 2
                <node-label-2> 3
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a Pending state after configuring the nodeSelector constraint, check the pod events for errors relating to taints and tolerations.

3. Save the file to apply the changes. The components specified in the new configuration are automatically moved to the new nodes, and the pods affected by the new configuration are redeployed.

Procedure

1. To configure core platform monitoring components, edit the cluster-monitoring-config config map object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add values to define resource limits and requests for each core platform monitoring component you want to configure.

   Important

   Make sure that the value set for a limit is always higher than the value set for a request. Otherwise, an error will occur, and the container will not run.

   Example

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusK8s:
         resources:
           limits:
             cpu: 500m
             memory: 3Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperator:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       k8sPrometheusAdapter:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       kubeStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       telemeterClient:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       openshiftStateMetrics:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       thanosQuerier:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       nodeExporter:
         resources:
           limits:
             cpu: 50m
             memory: 150Mi
           requests:
             cpu: 20m
             memory: 50Mi
       monitoringPlugin:
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 200m
             memory: 500Mi
       prometheusOperatorAdmissionWebhook:
         resources:
           limits:
             cpu: 50m
             memory: 100Mi
           requests:
             cpu: 20m
             memory: 50Mi

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add an enabled: true key-value pair as shown in the following sample:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         dedicatedServiceMonitors:
           enabled: true 1

   1

   Set the value of the enabled field to true to deploy a dedicated service monitor that exposes the kubelet /metrics/resource endpoint.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to a cluster-monitoring-config config map, the pods and other resources in the openshift-monitoring project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the ConfigMap object:

   • To configure a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the core monitoring component for which you want to configure the PVC.

        2

        Specify an existing storage class. If a storage class is not specified, the default storage class is used.

        3

        Specify the amount of required storage.

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims persistent storage for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: my-storage-class
                  resources:
                    requests:
                      storage: 40Gi

   • To configure a PVC for a component that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 2
                  resources:
                    requests:
                      storage: <amount_of_storage> 3

        1

        Specify the component for user-defined monitoring for which you want to configure the PVC.

        2

        Specify an existing storage class. If a storage class is not specified, the default storage class is used.

        3

        Specify the amount of required storage.

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate.

        The following example configures a PVC that claims persistent storage for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: my-storage-class
                  resources:
                    requests:
                      storage: 10Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed and the new storage configuration is applied.

   Warning

   When you update the config map with a PVC configuration, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Manually expand a PVC with the updated storage request. For more information, see "Expanding persistent volume claims (PVCs) with a file system" in Expanding persistent volumes.

2. Edit the ConfigMap object:

   • If you are configuring core OpenShift Container Platform monitoring components:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add a new storage size for the PVC configuration for the component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: <amount_of_storage> 2

        1

        The component for which you want to change the storage size.

        2

        Specify the new size for the storage volume. It must be greater than the previous value.

        The following example sets the new PVC request to 100 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: 100Gi

   • If you are configuring components that monitor user-defined projects:

     Note

     You can resize the volumes for the Thanos Ruler and for instances of Alertmanager and Prometheus that monitor user-defined projects.

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Update the PVC configuration for the monitoring component under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 1
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: <amount_of_storage> 2

        1

        The component for which you want to change the storage size.

        2

        Specify the new size for the storage volume. It must be greater than the previous value.

        The following example sets the new PVC request to 20 gigabytes for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: 20Gi

        Note

        Storage requirements for the thanosRuler component depend on the number of rules that are evaluated and how many samples each rule generates.

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

   Warning

   When you update the config map with a new storage size, the affected StatefulSet object is recreated, resulting in a temporary service outage.

Procedure

1. Edit the ConfigMap object:

   • To modify the retention time and size for the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add the retention time and size configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: <time_specification> 1
              retentionSize: <size_specification> 2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: 24h
              retentionSize: 10GB

   • To modify the retention time and size for the Prometheus instance that monitors user-defined projects:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add the retention time and size configuration under data/config.yaml:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: <time_specification> 1
              retentionSize: <size_specification> 2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), or EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: 24h
              retentionSize: 10GB

2. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the retention time configuration under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 1

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

3. Save the file to apply the changes. The pods affected by the new configuration are automatically redeployed.

Procedure

1. Edit the ConfigMap object:

   • To create cluster ID labels for core OpenShift Container Platform metrics:

     1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. In the writeRelabelConfigs: section under data/config.yaml/prometheusK8s/remoteWrite, add cluster ID relabel configuration values:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              remoteWrite:
              - url: "https://remote-write-endpoint.example.com"
                <endpoint_authentication_credentials>
                writeRelabelConfigs: 1
                  - <relabel_config> 2

        1

        Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

        2

        Substitute the label configuration for the metrics sent to the remote write endpoint.

        The following sample shows how to forward a metric with the cluster ID label cluster_id in default platform monitoring:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              remoteWrite:
              - url: "https://remote-write-endpoint.example.com"
                writeRelabelConfigs:
                - sourceLabels:
                  - __tmp_openshift_cluster_id__ 1
                  targetLabel: cluster_id 2
                  action: replace 3

        1

        The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

        2

        Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

        3

        The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

   • To create cluster ID labels for user-defined project metrics:

     1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. In the writeRelabelConfigs: section under data/config.yaml/prometheus/remoteWrite, add cluster ID relabel configuration values:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              remoteWrite:
              - url: "https://remote-write-endpoint.example.com"
                <endpoint_authentication_credentials>
                writeRelabelConfigs: 1
                  - <relabel_config> 2

        1

        Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

        2

        Substitute the label configuration for the metrics sent to the remote write endpoint.

        The following sample shows how to forward a metric with the cluster ID label cluster_id in user-workload monitoring:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              remoteWrite:
              - url: "https://remote-write-endpoint.example.com"
                writeRelabelConfigs:
                - sourceLabels:
                  - __tmp_openshift_cluster_id__ 1
                  targetLabel: cluster_id 2
                  action: replace 3

        1

        The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

        2

        Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

        3

        The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

2. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add the metrics collection profile setting under data/config.yaml/prometheusK8s:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         collectionProfile: <metrics_collection_profile_name> 1

   1

   The name of the metrics collection profile. The available values are full or minimal. If you do not specify a value or if the collectionProfile key name does not exist in the config map, the default setting of full is used.

   The following example sets the metrics collection profile to minimal for the core platform instance of Prometheus:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         collectionProfile: minimal

3. Save the file to apply the changes. The new configuration is applied automatically.

Procedure

1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the enforcedSampleLimit configuration to data/config.yaml to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Add the enforcedLabelLimit, enforcedLabelNameLengthLimit, and enforcedLabelValueLengthLimit configurations to data/config.yaml to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedLabelLimit: 500 1
         enforcedLabelNameLengthLimit: 50 2
         enforcedLabelValueLengthLimit: 600 3

   1

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   2

   Specifies the maximum length in characters of a label name. The default value is 0, which specifies no limit.

   3

   Specifies the maximum length in characters of a label value. The default value is 0, which specifies no limit.

4. Save the file to apply the changes. The limits are applied automatically.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called monitoring-stack-alerts.yaml:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 1
     namespace: ns1 2
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 3
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 4
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 5
         labels:
           severity: warning 6
       - alert: ApproachingEnforcedSamplesLimit 7
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 8
         expr: scrape_samples_scraped/50000 > 0.8 9
         for: 10m 10
         labels:
           severity: warning 11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule will be deployed.

   3

   The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.

   4

   The message that will be output when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.

   8

   The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to User Management → RoleBindings → Create binding.
2. In the Binding Type section, select the Namespace Role Binding type.
3. In the Name field, enter a name for the role binding.

4. In the Namespace field, select the project where you want to grant the access.

   Important

   The monitoring role or cluster role permissions that you grant to a user by using this procedure apply only to the project that you select in the Namespace field.

5. Select a monitoring role or cluster role from the Role Name list.
6. In the Subject section, select User.
7. In the Subject Name field, enter the name of the user.
8. Select Create to apply the role binding.

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Add the following deployment and service configuration details to the file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   This configuration deploys a service named prometheus-example-app in the user-defined ns1 project. This service exposes the custom version metric.

3. Apply the configuration to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   It takes some time to deploy the service.

4. You can check that the pod is running:

   $ oc -n ns1 get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

Procedure

1. Create a new YAML configuration file named example-app-service-monitor.yaml.

2. Add a ServiceMonitor resource to the YAML file. The following example creates a service monitor named prometheus-example-monitor to scrape metrics exposed by the prometheus-example-app service in the ns1 namespace:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: prometheus-example-monitor
     namespace: ns1 1
   spec:
     endpoints:
     - interval: 30s
       port: web 2
       scheme: http
     selector: 3
       matchLabels:
         app: prometheus-example-app

   1

   Specify a user-defined namespace where your service runs.

   2

   Specify endpoint ports to be scraped by Prometheus.

   3

   Configure a selector to match your service based on its metadata labels.

   Note

   A ServiceMonitor resource in a user-defined namespace can only discover services in the same namespace. That is, the namespaceSelector field of the ServiceMonitor resource is always ignored.

3. Apply the configuration to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   It takes some time to deploy the ServiceMonitor resource.

4. Verify that the ServiceMonitor resource is running:

   $ oc -n <namespace> get servicemonitor

   Example output

   NAME                         AGE
   prometheus-example-monitor   81m

Procedure

1. From the Administrator perspective in the OpenShift Container Platform web console, select Observe → Metrics.

2. To add one or more queries, do any of the following:

   Option	Description
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. You can use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. You can also move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Select Add query.
   Duplicate an existing query.	Select the Options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Select the Options menu next to the query and choose Disable query.

3. To run queries that you created, select Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note

   Queries that operate on large amounts of data might time out or overload the browser when drawing time series graphs. To avoid this, select Hide graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.

   Note

   By default, the query table shows an expanded view that lists every metric and its current value. You can select ˅ to minimize the expanded view for a query.

4. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown by doing any of the following:

   Option	Description
   Hide all metrics from a query.	Click the Options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box.
   Hide the plot.	Select Hide graph.

Procedure

1. From the Developer perspective in the OpenShift Container Platform web console, select Observe → Metrics.
2. Select the project that you want to view metrics for in the Project: list.

3. Select a query from the Select query list, or create a custom PromQL query based on the selected query by selecting Show PromQL. The metrics from the queries are visualized on the plot.

   Note

   In the Developer perspective, you can only run one query at a time.

4. Explore the visualized metrics by doing any of the following:

   Option	Description
   Zoom into the plot and change the time range.	Either: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range.
   Reset the time range.	Select Reset zoom.
   Display outputs for all queries at a specific point in time.	Hold the mouse cursor on the plot at that point. The query outputs appear in a pop-up box.

1. Go to Observe → Alerting → Alerts in the OpenShift Container Platform web console.
2. For the alert that you want to silence, click and select Silence alert to open the Silence alert page with a default configuration for the chosen alert.

3. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

4. To save the silence, click Silence.

1. Go to Observe → <project_name> → Alerts in the OpenShift Container Platform web console.
2. If necessary, expand the details for the alert by selecting a greater than symbol (>) next to the alert name.
3. Click the alert message in the expanded view to open the Alert details page for the alert.
4. Click Silence alert to open the Silence alert page with a default configuration for the alert.

5. Optional: Change the default configuration details for the silence.

   Note

   You must add a comment before saving a silence.

6. To save the silence, click Silence.

1. Go to Observe → Alerting → Silences in the OpenShift Container Platform web console.
2. Click Create silence.

3. On the Create silence page, set the schedule, duration, and label details for an alert.

   Note

   You must add a comment before saving a silence.

4. To create silences for alerts that match the labels that you entered, click Silence.

1. Go to Observe → <project_name> → Silences in the OpenShift Container Platform web console.
2. Click Create silence.

3. On the Create silence page, set the duration and label details for an alert.

   Note

   You must add a comment before saving a silence.

4. To create silences for alerts that match the labels that you entered, click Silence.

1. Go to Observe → Alerting → Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

1. Go to Observe → <project_name> → Silences.

2. For the silence you want to modify, click and select Edit silence.

   Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

1. Go to Observe → Alerting → Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

1. Go to Observe → <project_name> → Silences.
2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

   Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Procedure

1. Create a new YAML configuration file named example-alerting-rule.yaml.

2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertingRule
   metadata:
     name: example
     namespace: openshift-monitoring 1
   spec:
     groups:
     - name: example-rules
       rules:
       - alert: ExampleAlert 2
         for: 1m 3
         expr: vector(1) 4
         labels:
           severity: warning 5
         annotations:
           message: This is an example alert. 6

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The name of the alerting rule you want to create.

   3

   The duration for which the condition should be true before an alert is fired.

   4

   The PromQL query expression that defines the new rule.

   5

   The severity that alerting rule assigns to the alert.

   6

   The message associated with the alert.

   Important

   You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-alerting-rule.yaml

Procedure

1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

   apiVersion: monitoring.openshift.io/v1
   kind: AlertRelabelConfig
   metadata:
     name: watchdog
     namespace: openshift-monitoring 1
   spec:
     configs:
     - sourceLabels: [alertname,severity] 2
       regex: "Watchdog;none" 3
       targetLabel: severity 4
       replacement: critical 5
       action: Replace 6

   1

   Ensure that the namespace is openshift-monitoring.

   2

   The source labels for the values you want to modify.

   3

   The regular expression against which the value of sourceLabels is matched.

   4

   The target label of the value you want to modify.

   5

   The new value to replace the target label.

   6

   The relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

   Important

   You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-modified-alerting-rule.yaml

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert 1
         for: 1m 2
         expr: version{job="prometheus-example-app"} == 0 3
         labels:
           severity: warning 4
         annotations:
           message: This is an example alert. 5

   1

   The name of the alerting rule you want to create.

   2

   The duration for which the condition should be true before an alert is fired.

   3

   The PromQL query expression that defines the new rule.

   4

   The severity that alerting rule assigns to the alert.

   5

   The message associated with the alert.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

Procedure

1. To list alerting rules in <project>:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run the following:

   $ oc -n <project> get prometheusrule <rule> -o yaml

Procedure

1. In the Administrator perspective, navigate to Observe → Alerting → Alerting rules.

2. Select the Platform and User sources in the Filter drop-down menu.

   Note

   The Platform source is selected by default.

Procedure

1. In the Administrator perspective, go to Administration → Cluster Settings → Configuration → Alertmanager.

   Note

   Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OpenShift Container Platform web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

2. Click Create Receiver in the Receivers section of the page.
3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

4. Edit the receiver configuration:

   • For PagerDuty receivers:

     1. Choose an integration type and add a PagerDuty integration key.
     2. Add the URL of your PagerDuty installation.
     3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

   • For webhook receivers:

     1. Add the endpoint to send HTTP POST requests to.
     2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

   • For email receivers:

     1. Add the email address to send notifications to.
     2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.
     3. Select whether TLS is required.
     4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

   • For Slack receivers:

     1. Add the URL of the Slack webhook.
     2. Add the Slack channel or user name to send notifications to.
     3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

   1. Add routing label names and values in the Routing labels section of the form.
   2. Click Add label to add further routing labels.
6. Click Create to create the receiver.

Procedure

1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

2. Add an AlertmanagerConfig YAML definition to the file. For example:

   apiVersion: monitoring.coreos.com/v1beta1
   kind: AlertmanagerConfig
   metadata:
     name: example-routing
     namespace: ns1
   spec:
     route:
       receiver: default
       groupBy: [job]
     receivers:
     - name: default
       webhookConfigs:
       - url: https://example.org/post

   Note

   For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration defined in the AlertmanagerConfig object for namespace ns1 only applies to PrometheusRules resources in the same namespace.

3. Save the file.

4. Apply the resource to the cluster:

   $ oc apply -f example-app-alert-routing.yaml

   The configuration is automatically applied to the Alertmanager pods.

Procedure

1. Open the Alertmanager YAML configuration file:

   • To open the Alertmanager configuration from the CLI:

     1. Print the currently active Alertmanager configuration from the alertmanager-main secret into alertmanager.yaml file:

        $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

     2. Open the alertmanager.yaml file.

   • To open the Alertmanager configuration from the OpenShift Container Platform web console:

     1. Go to the Administration → Cluster Settings → Configuration → Alertmanager → YAML page of the web console.

2. Edit the Alertmanager configuration by updating parameters in the YAML:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s 1
     group_interval: 5m 2
     repeat_interval: 12h 3
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=<your_service>" 4
       routes:
       - matchers:
         - <your_matching_rules> 5
         receiver: <receiver> 6
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
     <receiver_configuration> 7

   1

   Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.

   2

   Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.

   3

   Specify the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.

   4

   Specify the name of the service that fires the alerts.

   5

   Specify labels to match your alerts.

   6

   Specify the name of the receiver to use for the alerts.

   7

   Specify the receiver configuration.

   Important
   • Use the matchers key name to indicate the matchers that an alert has to fulfill to match the node. Do not use the match or match_re key names, which are both deprecated and planned for removal in a future release.

   • If you define inhibition rules, use the following key names:

     • target_matchers: to indicate the target matchers
     • source_matchers: to indicate the source matchers

     Do not use the target_match, target_match_re, source_match, or source_match_re key names, which are deprecated and planned for removal in a future release.

   The following Alertmanager configuration example configures PagerDuty as an alert receiver:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - matchers:
       - "alertname=Watchdog"
       repeat_interval: 2m
       receiver: watchdog
     - matchers:
       - "service=example-app"
       routes:
       - matchers:
         - "severity=critical"
         receiver: team-frontend-page
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page
     pagerduty_configs:
     - service_key: "<your_key>"

   With this configuration, alerts of critical severity that are fired by the example-app service are sent through the team-frontend-page receiver. Typically, these types of alerts would be paged to an individual or a critical response team.

3. Apply the new configuration in the file:

   • To apply the changes from the CLI, run the following command:

     $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

   • To apply the changes from the OpenShift Container Platform web console, click Save.

Procedure

1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

   $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

2. Edit the configuration in alertmanager.yaml:

   route:
     receiver: Default
     group_by:
     - name: Default
     routes:
     - matchers:
       - "service = prometheus-example-monitor" 1
       receiver: <receiver> 2
   receivers:
   - name: Default
   - name: <receiver>
     <receiver_configuration> 3

   1

   Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.

   2

   Specify the name of the receiver to use for the alerts group.

   3

   Specify the receiver configuration.

3. Apply the new configuration in the file:

   $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-