Chapter 8. Setting up metrics and dashboards for AMQ Streams

Procedure

1. Check that monitoring for user-defined projects is enabled:

   oc get pods -n openshift-user-workload-monitoring

   If enabled, pods for the monitoring components are returned. For example:

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-5cc59f9bc6-kgcq8   1/1     Running   0          25s
   prometheus-user-workload-0             5/5     Running   1          14s
   prometheus-user-workload-1             5/5     Running   1          14s
   thanos-ruler-user-workload-0           3/3     Running   0          14s
   thanos-ruler-user-workload-1           3/3     Running   0          14s

   If no pods are returned, monitoring for user-defined projects is disabled. See the Prerequisites in Section 8.5, “Viewing Kafka metrics and dashboards in OpenShift”.

2. Multiple PodMonitor resources are defined in examples/metrics/prometheus-install/strimzi-pod-monitor.yaml.

   For each PodMonitor resource, edit the spec.namespaceSelector.matchNames property:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: cluster-operator-metrics
     labels:
       app: strimzi
   spec:
     selector:
       matchLabels:
         strimzi.io/kind: cluster-operator
     namespaceSelector:
       matchNames:
         - <project-name> 1
     podMetricsEndpoints:
     - path: /metrics
       port: http
   # ...

   1

   The project where the pods to scrape the metrics from are running, for example, Kafka.

3. Deploy the strimzi-pod-monitor.yaml file to the project where your Kafka cluster is running:

   oc apply -f strimzi-pod-monitor.yaml -n MY-PROJECT

4. Deploy the example Prometheus rules to the same project:

   oc apply -f prometheus-rules.yaml -n MY-PROJECT

Procedure

1. Create a ServiceAccount for Grafana. Here the resource is named grafana-serviceaccount.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: grafana-serviceaccount
     labels:
       app: strimzi

2. Deploy the ServiceAccount to the project containing your Kafka cluster:

   oc apply -f GRAFANA-SERVICEACCOUNT -n MY-PROJECT

3. Create a ClusterRoleBinding resource that assigns the cluster-monitoring-view role to the Grafana ServiceAccount. Here the resource is named grafana-cluster-monitoring-binding.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: grafana-cluster-monitoring-binding
     labels:
       app: strimzi
   subjects:
     - kind: ServiceAccount
       name: grafana-serviceaccount
       namespace: <my-project> 1
   roleRef:
     kind: ClusterRole
     name: cluster-monitoring-view
     apiGroup: rbac.authorization.k8s.io

   1

   Name of your project.

4. Deploy the ClusterRoleBinding to the project containing your Kafka cluster:

   oc apply -f <grafana-cluster-monitoring-binding> -n <my-project>

Procedure

1. Get the access token of the Grafana ServiceAccount:

   oc serviceaccounts get-token grafana-serviceaccount -n MY-PROJECT

   Copy the access token to use in the next step.

2. Create a datasource.yaml file containing the Thanos Querier configuration for Grafana.

   Paste the access token into the httpHeaderValue1 property as indicated.

   apiVersion: 1
   
   datasources:
   - name: Prometheus
     type: prometheus
     url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
     access: proxy
     basicAuth: false
     withCredentials: false
     isDefault: true
     jsonData:
       timeInterval: 5s
       tlsSkipVerify: true
       httpHeaderName1: "Authorization"
     secureJsonData:
       httpHeaderValue1: "Bearer ${GRAFANA-ACCESS-TOKEN}" 1
     editable: true

   1

   GRAFANA-ACCESS-TOKEN: The value of the access token for the Grafana ServiceAccount.

3. Create a config map named grafana-config from the datasource.yaml file:

   oc create configmap grafana-config --from-file=datasource.yaml -n MY-PROJECT

4. Create a Grafana application consisting of a Deployment and a Service.

   The grafana-config config map is mounted as a volume for the datasource configuration.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     replicas: 1
     selector:
       matchLabels:
         name: grafana
     template:
       metadata:
         labels:
           name: grafana
       spec:
         serviceAccountName: grafana-serviceaccount
         containers:
         - name: grafana
           image: grafana/grafana:9.3.1
           ports:
           - name: grafana
             containerPort: 3000
             protocol: TCP
           volumeMounts:
           - name: grafana-data
             mountPath: /var/lib/grafana
           - name: grafana-logs
             mountPath: /var/log/grafana
           - name: grafana-config
             mountPath: /etc/grafana/provisioning/datasources/datasource.yaml
             readOnly: true
             subPath: datasource.yaml
           readinessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 5
             periodSeconds: 10
           livenessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 15
             periodSeconds: 20
         volumes:
         - name: grafana-data
           emptyDir: {}
         - name: grafana-logs
           emptyDir: {}
         - name: grafana-config
           configMap:
             name: grafana-config
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     ports:
     - name: grafana
       port: 3000
       targetPort: 3000
       protocol: TCP
     selector:
       name: grafana
     type: ClusterIP

5. Deploy the Grafana application to the project containing your Kafka cluster:

   oc apply -f <grafana-application> -n <my-project>

Procedure

1. Get the details of the Route to the Grafana Service. For example:

   oc get routes
   
   NAME               HOST/PORT                         PATH  SERVICES
   MY-GRAFANA-ROUTE   MY-GRAFANA-ROUTE-amq-streams.net        grafana

2. In a web browser, access the Grafana login screen using the URL for the Route host and port.

3. Enter your user name and password, and then click Log In.

   The default Grafana user name and password are both admin. After logging in for the first time, you can change the password.

4. In Configuration > Data Sources, check that the Prometheus data source was created. The data source was created in Section 8.5.5, “Deploying Grafana with a Prometheus datasource”.
5. Click the + icon and then click Import.
6. In examples/metrics/grafana-dashboards, copy the JSON of the dashboard to import.
7. Paste the JSON into the text box, and then click Load.
8. Repeat steps 5-7 for the other example Grafana dashboards.