Monitoring

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

   Copy to Clipboard Toggle word wrap

2. Set the techPreviewUserWorkload setting to true under data/config.yaml:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       techPreviewUserWorkload:
         enabled: true

   Copy to Clipboard Toggle word wrap
3. Save the file to apply the changes. Monitoring your own services is enabled automatically.

4. Optional: You can check that the prometheus-user-workload pods were created:

   $ oc -n openshift-user-workload-monitoring get pod
   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-85bbb7b64d-7jwjd   1/1     Running   0          3m24s
   prometheus-user-workload-0             5/5     Running   1          3m13s
   prometheus-user-workload-1             5/5     Running   1          3m13s

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Fill the file with the configuration for deploying the service:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: quay.io/brancz/prometheus-example-app:v0.2.0
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   Copy to Clipboard Toggle word wrap

   This configuration deploys a service named prometheus-example-app in the ns1 project. This service exposes the custom version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   Copy to Clipboard Toggle word wrap

   It will take some time to deploy the service.

4. You can check that the service is running:

   $ oc -n ns1 get pod
   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for the new role. In this example, it is called custom-metrics-role.yaml.

2. Fill the file with the configuration for the monitor-crd-edit role:

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: monitor-crd-edit
   rules:
   - apiGroups: ["monitoring.coreos.com"]
     resources: ["prometheusrules", "servicemonitors", "podmonitors"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

   Copy to Clipboard Toggle word wrap

   This role enables a user to set up metrics collection for services.

3. Apply the configuration file to the cluster:

   $ oc apply -f custom-metrics-role.yaml

   Copy to Clipboard Toggle word wrap

   Now the role is created.

Procedure

1. In the Web console, navigate to User Management → Role Bindings → Create Binding.
2. In Binding Type, select the "Namespace Role Binding" type.
3. In Name, enter a name for the binding.
4. In Namespace, select the namespace where you want to grant the access.
5. In Role Name, enter monitor-crd-edit.
6. In Subject, select User.
7. In Subject Name, enter name of the user, for example johnsmith.
8. Confirm the role binding. Now the user has been assigned the monitor-crd-edit role, which allows him to set up metrics collection for a service in the namespace.

Procedure

1. Create a YAML file for the ServiceMonitor configuration. In this example, the file is called example-app-service-monitor.yaml.

2. Fill the file with the configuration for creating the ServiceMonitor:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: prometheus-example-monitor
     name: prometheus-example-monitor
     namespace: ns1
   spec:
     endpoints:
     - interval: 30s
       port: web
       scheme: http
     selector:
       matchLabels:
         app: prometheus-example-app

   Copy to Clipboard Toggle word wrap

   This configuration makes OpenShift Monitoring scrape the metrics exposed by the sample service deployed in "Deploying a sample service", which includes the single version metric.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   Copy to Clipboard Toggle word wrap

   It will take some time to deploy the ServiceMonitor.

4. You can check that the ServiceMonitor is running:

   $ oc -n ns1 get servicemonitor
   NAME                         AGE
   prometheus-example-monitor   81m

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

2. Fill the file with the configuration for the alerting rules:

   Note

   The expression can only reference metrics exposed by your own services. Currently it is not possible to correlate existing cluster metrics.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: example-alert
     namespace: ns1
   spec:
     groups:
     - name: example
       rules:
       - alert: VersionAlert
         expr: version{job="prometheus-example-app"} == 0

   Copy to Clipboard Toggle word wrap

   This configuration creates an alerting rule named example-alert, which fires an alert when the version metric exposed by the sample service becomes 0.

3. Apply the configuration file to the cluster:

   $ oc apply -f example-app-alerting-rule.yaml

   Copy to Clipboard Toggle word wrap

   It will take some time to create the alerting rules.

Procedure

1. Access the Prometheus web interface:

   • To access the metrics as a cluster administrator, go to the OpenShift Container Platform web console, switch to the Administrator Perspective, and click Monitoring → Metrics.

     Note

     Cluster administrators, when using the Administrator Perspective, have access to all cluster metrics and to custom service metrics from all projects.

     Note

     Only cluster administrators have access to the Alertmanager and Prometheus UIs.

   • To access the metrics as a developer or a user with permissions, go to the OpenShift Container Platform web console, switch to the Developer Perspective, then click Advanced → Metrics. Select the project you want to see the metrics for.

     Note

     Developers can only use the Developer Perspective. They can only query metrics from a single project.

2. Use the PromQL interface to run queries for your services.

Procedure

1. Create a YAML file for your configuration. In this example, it is called deploy.yaml.

2. Add configuration for creating the service account, necessary roles, and role bindings for prometheus-adapter:

   kind: ServiceAccount
   apiVersion: v1
   metadata:
     name: custom-metrics-apiserver
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: custom-metrics-server-resources
   rules:
   - apiGroups:
     - custom.metrics.k8s.io
     resources: ["*"]
     verbs: ["*"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: custom-metrics-resource-reader
   rules:
   - apiGroups:
     - ""
     resources:
     - namespaces
     - pods
     - services
     verbs:
     - get
     - list
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: custom-metrics:system:auth-delegator
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:auth-delegator
   subjects:
   - kind: ServiceAccount
     name: custom-metrics-apiserver
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: custom-metrics-auth-reader
     namespace: kube-system
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: extension-apiserver-authentication-reader
   subjects:
   - kind: ServiceAccount
     name: custom-metrics-apiserver
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: custom-metrics-resource-reader
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: custom-metrics-resource-reader
   subjects:
   - kind: ServiceAccount
     name: custom-metrics-apiserver
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: hpa-controller-custom-metrics
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: custom-metrics-server-resources
   subjects:
   - kind: ServiceAccount
     name: horizontal-pod-autoscaler
     namespace: kube-system
   ---

   Copy to Clipboard Toggle word wrap

3. Add configuration for the custom metrics for prometheus-adapter:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: adapter-config
     namespace: default
   data:
     config.yaml: |
       rules:
       - seriesQuery: 'http_requests_total{namespace!="",pod!=""}' 

   1

   
         resources:
           overrides:
             namespace: {resource: "namespace"}
             pod: {resource: "pod"}
             service: {resource: "service"}
         name:
           matches: "^(.*)_total"
           as: "${1}_per_second" 

   2

   
         metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
   ---

   Copy to Clipboard Toggle word wrap

   1

   Specifies the chosen metric to be the number of HTTP requests.

   2

   Specifies the frequency for the metric.

4. Add configuration for registering prometheus-adapter as an API service:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.alpha.openshift.io/serving-cert-secret-name: prometheus-adapter-tls
     labels:
       name: prometheus-adapter
     name: prometheus-adapter
     namespace: default
   spec:
     ports:
     - name: https
       port: 443
       targetPort: 6443
     selector:
       app: prometheus-adapter
     type: ClusterIP
   ---
   apiVersion: apiregistration.k8s.io/v1beta1
   kind: APIService
   metadata:
     name: v1beta1.custom.metrics.k8s.io
   spec:
     service:
       name: prometheus-adapter
       namespace: default
     group: custom.metrics.k8s.io
     version: v1beta1
     insecureSkipTLSVerify: true
     groupPriorityMinimum: 100
     versionPriority: 100
   ---

   Copy to Clipboard Toggle word wrap

5. Show the Prometheus Adapter image to use:

   $ kubectl get -n openshift-monitoring deploy/prometheus-adapter -o jsonpath="{..image}"
   quay.io/openshift-release-dev/ocp-v4.3-art-dev@sha256:76db3c86554ad7f581ba33844d6a6ebc891236f7db64f2d290c3135ba81c264c

   Copy to Clipboard Toggle word wrap

6. Add configuration for deploying prometheus-adapter:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-adapter
     name: prometheus-adapter
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-adapter
     template:
       metadata:
         labels:
           app: prometheus-adapter
         name: prometheus-adapter
       spec:
         serviceAccountName: custom-metrics-apiserver
         containers:
         - name: prometheus-adapter
           image: openshift-release-dev/ocp-v4.3-art-dev 

   1

   
           args:
           - --secure-port=6443
           - --tls-cert-file=/var/run/serving-cert/tls.crt
           - --tls-private-key-file=/var/run/serving-cert/tls.key
           - --logtostderr=true
           - --prometheus-url=http://prometheus-operated.default.svc:9090/
           - --metrics-relist-interval=1m
           - --v=4
           - --config=/etc/adapter/config.yaml
           ports:
           - containerPort: 6443
           volumeMounts:
           - mountPath: /var/run/serving-cert
             name: volume-serving-cert
             readOnly: true
           - mountPath: /etc/adapter/
             name: config
             readOnly: true
           - mountPath: /tmp
             name: tmp-vol
         volumes:
         - name: volume-serving-cert
           secret:
             secretName: prometheus-adapter-tls
         - name: config
           configMap:
             name: adapter-config
         - name: tmp-vol
           emptyDir: {}

   Copy to Clipboard Toggle word wrap

   1

   image: openshift-release-dev/ocp-v4.3-art-dev specifies the Prometheus Adapter image found in the previous step.

7. Apply the configuration file to the cluster:

   $ oc apply -f deploy.yaml

   Copy to Clipboard Toggle word wrap
8. Now the application’s metrics are exposed and can be used to configure horizontal pod autoscaling.