Chapter 2. Monitoring your own services

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Set the techPreviewUserWorkload setting to true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       techPreviewUserWorkload:
         enabled: true

3. Save the file to apply the changes. Monitoring your own services is enabled automatically.

4. Optional: You can check that the prometheus-user-workload pods were created:

   $ oc -n openshift-user-workload-monitoring get pod
   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-85bbb7b64d-7jwjd   1/1     Running   0          3m24s
   prometheus-user-workload-0             5/5     Running   1          3m13s
   prometheus-user-workload-1             5/5     Running   1          3m13s

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Fill the file with the configuration for deploying the service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: quay.io/brancz/prometheus-example-app:v0.2.0
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   This configuration deploys a service named prometheus-example-app in the ns1 project. This service exposes the custom version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   It will take some time to deploy the service.

4. You can check that the service is running:

   $ oc -n ns1 get pod
   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

Procedure

1. Create a YAML file for the new role. In this example, it is called custom-metrics-role.yaml.

2. Fill the file with the configuration for the monitor-crd-edit role:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: monitor-crd-edit
   rules:
   - apiGroups: ["monitoring.coreos.com"]
     resources: ["prometheusrules", "servicemonitors", "podmonitors"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

   This role enables a user to set up metrics collection for services.

3. Apply the configuration file to the cluster:

   $ oc apply -f custom-metrics-role.yaml

   Now the role is created.

Procedure

1. In the Web console, navigate to User Management Role Bindings Create Binding.
2. In Binding Type, select the "Namespace Role Binding" type.
3. In Name, enter a name for the binding.
4. In Namespace, select the namespace where you want to grant the access.
5. In Role Name, enter monitor-crd-edit.
6. In Subject, select User.
7. In Subject Name, enter name of the user, for example johnsmith.
8. Confirm the role binding. Now the user has been assigned the monitor-crd-edit role, which allows him to set up metrics collection for a service in the namespace.

Procedure

1. Create a YAML file for the ServiceMonitor configuration. In this example, the file is called example-app-service-monitor.yaml.

2. Fill the file with the configuration for creating the ServiceMonitor:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-example-monitor
     name: prometheus-example-monitor
     namespace: ns1
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-example-app

   This configuration makes OpenShift Monitoring scrape the metrics exposed by the sample service deployed in "Deploying a sample service", which includes the single version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   It will take some time to deploy the ServiceMonitor.

4. You can check that the ServiceMonitor is running:

   $ oc -n ns1 get servicemonitor
   NAME                         AGE
   prometheus-example-monitor   81m

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Fill the file with the configuration for the alerting rules:

   Note

   The expression can only reference metrics exposed by your own services. Currently it is not possible to correlate existing cluster metrics.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

   This configuration creates an alerting rule named example-alert, which fires an alert when the version metric exposed by the sample service becomes 0.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   It will take some time to create the alerting rules.

Procedure

1. Access the Prometheus web interface:

   • To access the metrics as a cluster administrator, go to the OpenShift Container Platform web console, switch to the Administrator Perspective, and click Monitoring Metrics.

     Note

     Cluster administrators, when using the Administrator Perspective, have access to all cluster metrics and to custom service metrics from all projects.

     Note

     Only cluster administrators have access to the Alertmanager and Prometheus UIs.

   • To access the metrics as a developer or a user with permissions, go to the OpenShift Container Platform web console, switch to the Developer Perspective, then click Advanced Metrics. Select the project you want to see the metrics for.

     Note

     Developers can only use the Developer Perspective. They can only query metrics from a single project.

2. Use the PromQL interface to run queries for your services.