Monitoring

Procedure

1. Check whether the

   cluster-monitoring-config

   ConfigMap

   object exists:

   $ oc -n openshift-monitoring get configmap cluster-monitoring-config

2. If the

   ConfigMap

   object does not exist:

   1. Create the following YAML manifest. In this example the file is called

      cluster-monitoring-config.yaml

      :

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the

      ConfigMap

      object:

      $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Check whether the

   user-workload-monitoring-config

   ConfigMap

   object exists:

   $ oc -n openshift-user-workload-monitoring get configmap user-workload-monitoring-config

2. If the

   user-workload-monitoring-config

   ConfigMap

   object does not exist:

   1. Create the following YAML manifest. In this example the file is called

      user-workload-monitoring-config.yaml

      :

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |

   2. Apply the configuration to create the

      ConfigMap

      object:

      $ oc apply -f user-workload-monitoring-config.yaml

      Note

      Configurations applied to the

      user-workload-monitoring-config

      ConfigMap

      object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

Procedure

1. If you have not done so yet, add a label to the nodes on which you want to run the monitoring components:

   $ oc label nodes <node-name> <node-label>

2. Edit the

   ConfigMap

   object:

   • To move a component that monitors core OpenShift Container Platform projects:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Specify the node labels for the

        nodeSelector

        constraint for the component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              nodeSelector:
                <node-label-1> 

        2

        
                <node-label-2> 

        3

        
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a

        Pending

        state after configuring the

        nodeSelector

        constraint, check the pod events for errors relating to taints and tolerations.

   • To move a component that monitors user-defined projects:

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Specify the node labels for the

        nodeSelector

        constraint for the component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              nodeSelector:
                <node-label-1> 

        2

        
                <node-label-2> 

        3

        
                <...>

        1

        Substitute <component> with the appropriate monitoring stack component name.

        2

        Substitute <node-label-1> with the label you added to the node.

        3

        Optional: Specify additional labels. If you specify additional labels, the pods for the component are only scheduled on the nodes that contain all of the specified labels.

        Note

        If monitoring components remain in a

        Pending

        state after configuring the

        nodeSelector

        constraint, check the pod events for errors relating to taints and tolerations.

3. Save the file to apply the changes. The components specified in the new configuration are moved to the new nodes automatically.

   Note

   Configurations applied to the

   user-workload-monitoring-config

   ConfigMap

   object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the

   cluster-monitoring-config

   ConfigMap

   object in the

   openshift-monitoring

   namespace:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add an

   enabled: true

   key-value pair as shown in the following sample:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       k8sPrometheusAdapter:
         dedicatedServiceMonitors:
           enabled: true 

   1

   1

   Set the value of the enabled field to true to deploy a dedicated service monitor that exposes the kubelet /metrics/resource endpoint.

3. Save the file to apply the changes automatically.

   Warning

   When you save changes to a

   cluster-monitoring-config

   config map, the pods and other resources in the

   openshift-monitoring

   project might be redeployed. The running monitoring processes in that project might also restart.

Procedure

1. Edit the

   ConfigMap

   object:

   • To configure a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add your PVC configuration for the component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify

        volumeClaimTemplate

        .

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called

        local-storage

        .

        The following example configures a PVC that claims local persistent storage for Alertmanager:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

   • To configure a PVC for a component that monitors user-defined projects:

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add your PVC configuration for the component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class>
                  resources:
                    requests:
                      storage: <amount_of_storage>

        See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify

        volumeClaimTemplate

        .

        The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

        In the above example, the storage class created by the Local Storage Operator is called

        local-storage

        .

        The following example configures a PVC that claims local persistent storage for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 10Gi

        Note

        Storage requirements for the

        thanosRuler

        component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

   Note

   Configurations applied to the

   user-workload-monitoring-config

   ConfigMap

   object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Edit the

   ConfigMap

   object:

   • To resize a PVC for a component that monitors core OpenShift Container Platform projects:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add a new storage size for the PVC configuration for the component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 

        2

        
                  resources:
                    requests:
                      storage: <amount_of_storage> 

        3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures a PVC that sets the local persistent storage to 100 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example configures a PVC that sets the local persistent storage for Alertmanager to 40 gigabytes:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            alertmanagerMain:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 40Gi

   • To resize a PVC for a component that monitors user-defined projects:

     Note

     You can resize the volumes for the Thanos Ruler and Prometheus instances that monitor user-defined projects.

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Update the PVC configuration for the monitoring component under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>: 

        1

        
              volumeClaimTemplate:
                spec:
                  storageClassName: <storage_class> 

        2

        
                  resources:
                    requests:
                      storage: <amount_of_storage> 

        3

        1

        Specify the core monitoring component.

        2

        Specify the storage class.

        3

        Specify the new size for the storage volume.

        The following example configures the PVC size to 100 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 100Gi

        The following example sets the PVC size to 20 gigabytes for Thanos Ruler:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              volumeClaimTemplate:
                spec:
                  storageClassName: local-storage
                  resources:
                    requests:
                      storage: 20Gi

        Note

        Storage requirements for the

        thanosRuler

        component depend on the number of rules that are evaluated and how many samples each rule generates.

2. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

   Warning

   When you save changes to a monitoring config map, the pods and other resources in the related project might be redeployed. The monitoring processes running in that project might also be restarted.

3. Manually patch every PVC with the updated storage request. The following example resizes the storage size for the Prometheus component in the

   openshift-monitoring

   namespace to 100Gi:

   $ for p in $(oc -n openshift-monitoring get pvc -l app.kubernetes.io/name=prometheus -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \
     oc -n openshift-monitoring patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"100Gi"}}}}'; \
     done

4. Delete the underlying StatefulSet with the

   --cascade=orphan

   parameter:

   $ oc delete statefulset -l app.kubernetes.io/name=prometheus --cascade=orphan

Procedure

1. Edit the

   ConfigMap

   object:

   • To modify the retention time and size for the Prometheus instance that monitors core OpenShift Container Platform projects:

     1. Edit the

        cluster-monitoring-config

        ConfigMap

        object in the

        openshift-monitoring

        project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

     2. Add the retention time and size configuration under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: <time_specification> 

        1

        
              retentionSize: <size_specification> 

        2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors core OpenShift Container Platform components:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              retention: 24h
              retentionSize: 10GB

   • To modify the retention time and size for the Prometheus instance that monitors user-defined projects:

     1. Edit the

        user-workload-monitoring-config

        ConfigMap

        object in the

        openshift-user-workload-monitoring

        project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

     2. Add the retention time and size configuration under

        data/config.yaml

        :

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: <time_specification> 

        1

        
              retentionSize: <size_specification> 

        2

        1

        The retention time: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s.

        2

        The retention size: a number directly followed by B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), or EB (exabytes).

        The following example sets the retention time to 24 hours and the retention size to 10 gigabytes for the Prometheus instance that monitors user-defined projects:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            prometheus:
              retention: 24h
              retentionSize: 10GB

2. Save the file to apply the changes. The pods affected by the new configuration restart automatically.

Procedure

1. Edit the

   user-workload-monitoring-config

   ConfigMap

   object in the

   openshift-user-workload-monitoring

   project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the retention time configuration under

   data/config.yaml

   :

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: <time_specification> 

   1

   1

   Specify the retention time in the following format: a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). You can also combine time values for specific times, such as 1h30m15s. The default is 24h.

   The following example sets the retention time to 10 days for Thanos Ruler data:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       thanosRuler:
         retention: 10d

3. Save the file to apply the changes. The pods affected by the new configuration automatically restart.

Procedure

1. Edit the

   cluster-monitoring-config

   ConfigMap

   object in the

   openshift-monitoring

   project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Note

   If you configure cluster ID labels for metrics for the Prometheus instance that monitors user-defined projects, edit the

   user-workload-monitoring-config

   config map in the

   openshift-user-workload-monitoring

   namespace. Note that the Prometheus component is called

   prometheus

   in this config map and not

   prometheusK8s

   , which is the name used in the

   cluster-monitoring-config

   config map.

2. In the

   writeRelabelConfigs:

   section under

   data/config.yaml/prometheusK8s/remoteWrite

   , add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 

   1

   
             - <relabel_config> 

   2

   1

   Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.

   2

   Substitute the label configuration for the metrics sent to the remote write endpoint.

   The following sample shows how to forward a metric with the cluster ID label

   cluster_id

   in default platform monitoring:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           writeRelabelConfigs:
           - sourceLabels:
             - __tmp_openshift_cluster_id__ 

   1

   
             targetLabel: cluster_id 

   2

   
             action: replace 

   3

   1

   The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.

   2

   Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.

   3

   The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes to the

   ConfigMap

   object. The pods affected by the updated configuration automatically restart.
   Warning

   Saving changes to a monitoring

   ConfigMap

   object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.

Procedure

1. Edit the

   user-workload-monitoring-config

   ConfigMap

   object in the

   openshift-user-workload-monitoring

   project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add the

   enforcedSampleLimit

   configuration to

   data/config.yaml

   to limit the number of samples that can be accepted per target scrape in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedSampleLimit: 50000 

   1

   1

   A value is required if this parameter is specified. This enforcedSampleLimit example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000.

3. Add the

   enforcedLabelLimit

   ,

   enforcedLabelNameLengthLimit

   , and

   enforcedLabelValueLengthLimit

   configurations to

   data/config.yaml

   to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         enforcedLabelLimit: 500 

   1

   
         enforcedLabelNameLengthLimit: 50 

   2

   
         enforcedLabelValueLengthLimit: 600 

   3

   1

   Specifies the maximum number of labels per scrape. The default value is 0, which specifies no limit.

   2

   Specifies the maximum length in characters of a label name. The default value is 0, which specifies no limit.

   3

   Specifies the maximum length in characters of a label value. The default value is 0, which specifies no limit.

4. Save the file to apply the changes. The limits are applied automatically.

   Note

   Configurations applied to the

   user-workload-monitoring-config

   ConfigMap

   object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

   Warning

   When changes are saved to the

   user-workload-monitoring-config

   ConfigMap

   object, the pods and other resources in the

   openshift-user-workload-monitoring

   project might be redeployed. The running monitoring processes in that project might also be restarted.

Procedure

1. Create a YAML file with alerts that inform you when the targets are down and when the enforced sample limit is approaching. The file in this example is called

   monitoring-stack-alerts.yaml

   :

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       prometheus: k8s
       role: alert-rules
     name: monitoring-stack-alerts 

   1

   
     namespace: ns1 

   2

   
   spec:
     groups:
     - name: general.rules
       rules:
       - alert: TargetDown 

   3

   
         annotations:
           message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service
             }} targets in {{ $labels.namespace }} namespace are down.' 

   4

   
         expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
           namespace, service)) > 10
         for: 10m 

   5

   
         labels:
           severity: warning 

   6

   
       - alert: ApproachingEnforcedSamplesLimit 

   7

   
         annotations:
           message: '{{ $labels.container }} container of the {{ $labels.pod }} pod in the {{ $labels.namespace }} namespace consumes {{ $value | humanizePercentage }} of the samples limit budget.' 

   8

   
         expr: scrape_samples_scraped/50000 > 0.8 

   9

   
         for: 10m 

   10

   
         labels:
           severity: warning 

   11

   1

   Defines the name of the alerting rule.

   2

   Specifies the user-defined project where the alerting rule will be deployed.

   3

   The TargetDown alert will fire if the target cannot be scraped or is not available for the for duration.

   4

   The message that will be output when the TargetDown alert fires.

   5

   The conditions for the TargetDown alert must be true for this duration before the alert is fired.

   6

   Defines the severity for the TargetDown alert.

   7

   The ApproachingEnforcedSamplesLimit alert will fire when the defined scrape sample threshold is reached or exceeded for the specified for duration.

   8

   The message that will be output when the ApproachingEnforcedSamplesLimit alert fires.

   9

   The threshold for the ApproachingEnforcedSamplesLimit alert. In this example the alert will fire when the number of samples per target scrape has exceeded 80% of the enforced sample limit of 50000. The for duration must also have passed before the alert will fire. The <number> in the expression scrape_samples_scraped/<number> > <threshold> must match the enforcedSampleLimit value defined in the user-workload-monitoring-config ConfigMap object.

   10

   The conditions for the ApproachingEnforcedSamplesLimit alert must be true for this duration before the alert is fired.

   11

   Defines the severity for the ApproachingEnforcedSamplesLimit alert.

2. Apply the configuration to the user-defined project:

   $ oc apply -f monitoring-stack-alerts.yaml