Chapter 14. Monitoring Data Grid services

1. In the OpenShift Web Console, select the </> Developer perspective and then select Monitoring.
2. Open the Dashboard tab for the namespace where your Data Grid cluster runs.

3. Open the Metrics tab and confirm that you can query Data Grid metrics such as:

   vendor_cache_manager_default_cluster_size

Procedure

1. Create a ServiceAccount that lets Grafana read Data Grid metrics from Prometheus.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: infinispan-monitoring

   1. Apply the ServiceAccount.

      oc apply -f service-account.yaml

   2. Grant cluster-monitoring-view permissions to the ServiceAccount.

      oc adm policy add-cluster-role-to-user cluster-monitoring-view -z infinispan-monitoring

2. Create a Grafana data source.

   1. Retrieve the token for the ServiceAccount.

      oc serviceaccounts get-token infinispan-monitoring

   2. Define a GrafanaDataSource that includes the token in the spec.datasources.secureJsonData.httpHeaderValue1 field, as in the following example:

      apiVersion: integreatly.org/v1alpha1
      kind: GrafanaDataSource
      metadata:
        name: grafanadatasource
      spec:
        name: datasource.yaml
        datasources:
          - access: proxy
            editable: true
            isDefault: true
            jsonData:
              httpHeaderName1: Authorization
              timeInterval: 5s
              tlsSkipVerify: true
            name: Prometheus
            secureJsonData:
              httpHeaderValue1: >-
                Bearer
                eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O...
            type: prometheus
            url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'

3. Apply the GrafanaDataSource.

   oc apply -f grafana-datasource.yaml

Procedure

1. Create a ConfigMap named infinispan-operator-config in the Data Grid Operator namespace.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: infinispan-operator-config
   data:
     grafana.dashboard.namespace: infinispan
     grafana.dashboard.name: infinispan
     grafana.dashboard.monitoring.key: middleware

2. Specify the namespace of your Data Grid cluster with the data.grafana.dashboard.namespace property.

   Note

   Deleting the value for this property removes the dashboard. Changing the value moves the dashboard to that namespace.

3. Specify a name for the dashboard with the data.grafana.dashboard.name property.
4. If necessary, specify a monitoring key with the data.grafana.dashboard.monitoring.key property.

5. Create infinispan-operator-config or update the configuration.

   oc apply -f infinispan-operator-config.yaml

6. Open the Grafana UI, which is available at:

   oc get routes grafana-route -o jsonpath=https://"{.spec.host}"

1. Each Data Grid server pod exposes an authenticated JMX endpoint on port 9999 utilizing the "admin" security-realm, which includes the Operator user credentials.
2. The <cluster-name>-admin Service exposes port 9999.

Procedure

1. Enable JMX in your Infinispan CR.

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: infinispan
   spec:
     jmx:
       enabled: true

2. Retrieve the Operator user credentials to authenticate client JMX connections.

   oc get secret infinispan-generated-operator-secret -o jsonpath="{.data.identities\.yaml}" | base64 --decode

Procedure

1. Create a Cryostat CR in the same namespace as your Infinispan CR.

   apiVersion: operator.cryostat.io/v1beta1
   kind: Cryostat
   metadata:
     name: cryostat-sample
   spec:
     minimal: false
     enableCertManager: true

   Note

   The Cryostat Operator requires cert-manager for traffic encryption. If the cert-manager is enabled but not installed, the deployment fails. For details, see the Installing Cryostat guide.

2. Wait for the Cryostat CR to be ready.

   oc wait -n <namespace> --for=condition=MainDeploymentAvailable cryostat/cryostat-sample

3. Open the Cryostat status.applicationUrl.

   oc -n <namespace> get cryostat cryostat-sample

4. Retrieve the Operator user credentials to authenticate client JMX connections in the Cryostat UI.

   oc get secret infinispan-generated-operator-secret -o jsonpath="{.data.identities\.yaml}" | base64 --decode

5. In the Cryostat UI, navigate to the Security menu.
6. In the Store Credentials window, click the Add button. The Store Credentials window opens.

7. In the Match Expression filed, enter match expression details in the following format:

   target.labels['infinispan_cr'] == '<cluster_name>'