Chapter 2. Monitoring your own services

Procedure

1. Start editing the cluster-monitoring-config ConfigMap object:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Set the techPreviewUserWorkload setting to true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       techPreviewUserWorkload:
         enabled: true

3. Save the file to apply the changes. Monitoring your own services is enabled automatically.

4. Optional: You can check that the prometheus-user-workload pods were created:

   $ oc -n openshift-user-workload-monitoring get pod

   Example output

   NAME                                   READY   STATUS        RESTARTS   AGE
   prometheus-operator-6f7b748d5b-t7nbg   2/2     Running       0          3h
   prometheus-user-workload-0             5/5     Running       1          3h
   prometheus-user-workload-1             5/5     Running       1          3h
   thanos-ruler-user-workload-0           3/3     Running       0          3h
   thanos-ruler-user-workload-1           3/3     Running       0          3h

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Fill the file with the configuration for deploying the service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.3.0
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   This configuration deploys a service named prometheus-example-app in the ns1 project. This service exposes the custom version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   It will take some time to deploy the service.

4. You can check that the service is running:

   $ oc -n ns1 get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

Procedure

1. In the web console, navigate to User Management Role Bindings Create Binding.
2. In Binding Type, select the "Namespace Role Binding" type.
3. In Name, enter a name for the binding.
4. In Namespace, select the namespace where you want to grant the access. For example, select ns1.

5. In Role Name, enter monitoring-rules-view, monitoring-rules-edit, or monitoring-edit.

   • monitoring-rules-view allows reading PrometheusRule custom resources within the namespace.
   • monitoring-rules-edit allows creating, modifying, and deleting PrometheusRule custom resources matching the permitted namespace.
   • monitoring-edit gives the same permissions as monitoring-rules-edit. Additionally, it allows creating scraping targets for services or pods. It also allows creating, modifying, and deleting ServiceMonitor and PodMonitor resources.
   Important

   Whichever role you choose, you must bind it against a specific namespace as a cluster administrator.

   For example, enter monitoring-edit.

6. In Subject, select User.
7. In Subject Name, enter the name of the user. For example, enter johnsmith.
8. Confirm the role binding. If you followed the example, then user johnsmith has been assigned the permissions for setting up metrics collection and creating alerting rules in the ns1 namespace.

Procedure

1. Create a YAML file for the ServiceMonitor resource configuration. In this example, the file is called example-app-service-monitor.yaml.

2. Fill the file with the configuration for creating the ServiceMonitor resource:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-example-monitor
     name: prometheus-example-monitor
     namespace: ns1
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-example-app

   This configuration makes OpenShift Monitoring scrape the metrics exposed by the sample service deployed in "Deploying a sample service", which includes the single version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   It will take some time to deploy the ServiceMonitor resource.

4. You can check that the ServiceMonitor resource is running:

   $ oc -n ns1 get servicemonitor

   Example output

   NAME                         AGE
   prometheus-example-monitor   81m

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Fill the file with the configuration for the alerting rules:

   Note

   When you create an alerting rule, a namespace label is enforced on it if a rule with the same name exists in another namespace.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

   This configuration creates an alerting rule named example-alert, which fires an alert when the version metric exposed by the sample service becomes 0.

   Important

   For every namespace, you can use metrics of that namespace and cluster metrics, but not metrics of another namespace.

   For example, an alerting rule for ns1 can have metrics from ns1 and cluster metrics, such as the CPU and memory metrics. However, the rule cannot include metrics from ns2.

   Additionally, you cannot create alerting rules for the openshift-* core OpenShift namespaces. OpenShift Container Platform Monitoring by default provides a set of alerting rules for these namespaces.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   It will take some time to create the alerting rules.

Procedure

1. To list alerting rules in a project, run:

   $ oc -n <project> get prometheusrule

2. To list the configuration of an alerting rule, run:

   $ oc -n <project> get prometheusrule <rule> -oyaml

Procedure

1. List routes for the openshift-user-workload-monitoring namespace:

   $ oc -n openshift-user-workload-monitoring get routes

   The output shows the URL for the Thanos Ruler UI:

   NAME           HOST/PORT
   ...
   thanos-ruler   thanos-ruler-openshift-user-workload-monitoring.apps.example.devcluster.openshift.com

2. Navigate to the listed URL. Here you can see user alerting rules from all namespaces.

Procedure

1. Go to the OpenShift Container Platform web console, switch to the Developer Perspective, then click Advanced Metrics. Select the project you want to see the metrics for.

   Note

   Developers can only use the Developer Perspective and not the Administrator Perspective. They can only query metrics from a single project. They cannot access the third-party UIs provided with OpenShift Container Platform Monitoring.

2. Use the PromQL interface to run queries for your services.