21.5. Viewing Kafka metrics and dashboards in OpenShift

Procedure

1. Check that monitoring for user-defined projects is enabled:

   oc get pods -n openshift-user-workload-monitoring

   If enabled, pods for the monitoring components are returned. For example:

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-5cc59f9bc6-kgcq8   1/1     Running   0          25s
   prometheus-user-workload-0             5/5     Running   1          14s
   prometheus-user-workload-1             5/5     Running   1          14s
   thanos-ruler-user-workload-0           3/3     Running   0          14s
   thanos-ruler-user-workload-1           3/3     Running   0          14s

   If no pods are returned, monitoring for user-defined projects is disabled. See the Prerequisites in 第 21.5 节 “Viewing Kafka metrics and dashboards in OpenShift”.

2. Multiple PodMonitor resources are defined in examples/metrics/prometheus-install/strimzi-pod-monitor.yaml.

   For each PodMonitor resource, edit the spec.namespaceSelector.matchNames property:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: cluster-operator-metrics
     labels:
       app: strimzi
   spec:
     selector:
       matchLabels:
         strimzi.io/kind: cluster-operator
     namespaceSelector:
       matchNames:
         - <project-name> 

   1

   
     podMetricsEndpoints:
     - path: /metrics
       port: http
   # ...

   1

   The project where the pods to scrape the metrics from are running, for example, Kafka.

3. Deploy the strimzi-pod-monitor.yaml file to the project where your Kafka cluster is running:

   oc apply -f strimzi-pod-monitor.yaml -n MY-PROJECT

4. Deploy the example Prometheus rules to the same project:

   oc apply -f prometheus-rules.yaml -n MY-PROJECT

Procedure

1. Create a ServiceAccount for Grafana in the project containing your Kafka cluster:

    oc create sa grafana-service-account -n my-project

   In this example, a service account named grafana-service-account is created in the my-project namespace.

2. Create a ClusterRoleBinding resource that assigns the cluster-monitoring-view role to the Grafana ServiceAccount. Here the resource is named grafana-cluster-monitoring-binding.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: grafana-cluster-monitoring-binding
     labels:
       app: strimzi
   subjects:
     - kind: ServiceAccount
       name: grafana-service-account
       namespace: my-project
   roleRef:
     kind: ClusterRole
     name: cluster-monitoring-view
     apiGroup: rbac.authorization.k8s.io

3. Deploy the ClusterRoleBinding to the same project:

   oc apply -f grafana-cluster-monitoring-binding.yaml -n my-project

4. Create a token secret for the service account:

   apiVersion: v1
   kind: Secret
   metadata:
     name: secret-sa
     annotations:
       kubernetes.io/service-account.name: "grafana-service-account" 

   1

   
   type: kubernetes.io/service-account-token 

   2

   1

   Specifies the service account.

   2

   Specifies a service account token secret.

5. Create the Secret object and access token:

   oc create -f <secret_configuration>.yaml

   You need the access token when deploying Grafana.

Procedure

1. Get the access token of the Grafana ServiceAccount:

   oc describe sa/grafana-service-account | grep Tokens:
   oc describe secret grafana-service-account-token-mmlp9 | grep token:

   In this example, the service account is named grafana-service-account. Copy the access token to use in the next step.

2. Create a datasource.yaml file containing the Thanos Querier configuration for Grafana.

   Paste the access token into the httpHeaderValue1 property as indicated.

   apiVersion: 1
   
   datasources:
   - name: Prometheus
     type: prometheus
     url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
     access: proxy
     basicAuth: false
     withCredentials: false
     isDefault: true
     jsonData:
       timeInterval: 5s
       tlsSkipVerify: true
       httpHeaderName1: "Authorization"
     secureJsonData:
       httpHeaderValue1: "Bearer ${GRAFANA-ACCESS-TOKEN}" 

   1

   
     editable: true

   1

   GRAFANA-ACCESS-TOKEN: The value of the access token for the Grafana ServiceAccount.

3. Create a config map named grafana-config from the datasource.yaml file:

   oc create configmap grafana-config --from-file=datasource.yaml -n MY-PROJECT

4. Create a Grafana application consisting of a Deployment and a Service.

   The grafana-config config map is mounted as a volume for the datasource configuration.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     replicas: 1
     selector:
       matchLabels:
         name: grafana
     template:
       metadata:
         labels:
           name: grafana
       spec:
         serviceAccountName: grafana-service-account
         containers:
         - name: grafana
           image: grafana/grafana:10.4.2
           ports:
           - name: grafana
             containerPort: 3000
             protocol: TCP
           volumeMounts:
           - name: grafana-data
             mountPath: /var/lib/grafana
           - name: grafana-logs
             mountPath: /var/log/grafana
           - name: grafana-config
             mountPath: /etc/grafana/provisioning/datasources/datasource.yaml
             readOnly: true
             subPath: datasource.yaml
           readinessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 5
             periodSeconds: 10
           livenessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 15
             periodSeconds: 20
         volumes:
         - name: grafana-data
           emptyDir: {}
         - name: grafana-logs
           emptyDir: {}
         - name: grafana-config
           configMap:
             name: grafana-config
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     ports:
     - name: grafana
       port: 3000
       targetPort: 3000
       protocol: TCP
     selector:
       name: grafana
     type: ClusterIP

5. Deploy the Grafana application to the project containing your Kafka cluster:

   oc apply -f <grafana-application> -n <my-project>

Procedure

1. Get the details of the Route to the Grafana Service. For example:

   oc get routes
   
   NAME               HOST/PORT                         PATH  SERVICES
   MY-GRAFANA-ROUTE   MY-GRAFANA-ROUTE-amq-streams.net        grafana

2. In a web browser, access the Grafana login screen using the URL for the Route host and port.

3. Enter your user name and password, and then click Log In.

   The default Grafana user name and password are both admin. After logging in for the first time, you can change the password.

4. In Configuration > Data Sources, check that the Prometheus data source was created. The data source was created in 第 21.5.4 节 “Deploying Grafana with a Prometheus datasource”.
5. Click the + icon and then click Import.
6. In examples/metrics/grafana-dashboards, copy the JSON of the dashboard to import.
7. Paste the JSON into the text box, and then click Load.
8. Repeat steps 5-7 for the other example Grafana dashboards.