Chapter 14. Monitoring Data Grid services

1. In the OpenShift Web Console, select the </> Developer perspective and then select Monitoring.
2. Open the Dashboard tab for the namespace where your Data Grid cluster runs.

3. Open the Metrics tab and confirm that you can query Data Grid metrics such as:

   vendor_cache_manager_default_cluster_size

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ServiceAccount that lets Grafana read Data Grid metrics from Prometheus.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: infinispan-monitoring

   Copy to Clipboard Toggle word wrap

   1. Apply the ServiceAccount.

      oc apply -f service-account.yaml

      Copy to Clipboard Toggle word wrap

   2. Grant cluster-monitoring-view permissions to the ServiceAccount.

      oc adm policy add-cluster-role-to-user cluster-monitoring-view -z infinispan-monitoring

      Copy to Clipboard Toggle word wrap

2. Create a Grafana data source.

   1. Retrieve the token for the ServiceAccount.

      oc serviceaccounts get-token infinispan-monitoring

      Copy to Clipboard Toggle word wrap

   2. Define a GrafanaDataSource that includes the token in the spec.datasources.secureJsonData.httpHeaderValue1 field, as in the following example:

      apiVersion: integreatly.org/v1alpha1
      kind: GrafanaDataSource
      metadata:
        name: grafanadatasource
      spec:
        name: datasource.yaml
        datasources:
          - access: proxy
            editable: true
            isDefault: true
            jsonData:
              httpHeaderName1: Authorization
              timeInterval: 5s
              tlsSkipVerify: true
            name: Prometheus
            secureJsonData:
              httpHeaderValue1: >-
                Bearer
                eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O...
            type: prometheus
            url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'

      Copy to Clipboard Toggle word wrap

3. Apply the GrafanaDataSource.

   oc apply -f grafana-datasource.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ConfigMap named infinispan-operator-config in the Data Grid Operator namespace.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: infinispan-operator-config
   data:
     grafana.dashboard.namespace: infinispan
     grafana.dashboard.name: infinispan
     grafana.dashboard.monitoring.key: middleware

   Copy to Clipboard Toggle word wrap

2. Specify the namespace of your Data Grid cluster with the data.grafana.dashboard.namespace property.

   Note

   Deleting the value for this property removes the dashboard. Changing the value moves the dashboard to that namespace.

3. Specify a name for the dashboard with the data.grafana.dashboard.name property.
4. If necessary, specify a monitoring key with the data.grafana.dashboard.monitoring.key property.

5. Create infinispan-operator-config or update the configuration.

   oc apply -f infinispan-operator-config.yaml

   Copy to Clipboard Toggle word wrap

6. Open the Grafana UI, which is available at:

   oc get routes grafana-route -o jsonpath=https://"{.spec.host}"

   Copy to Clipboard Toggle word wrap

1. Each Data Grid server pod exposes an authenticated JMX endpoint on port 9999 utilizing the "admin" security-realm, which includes the Operator user credentials.
2. The <cluster-name>-admin Service exposes port 9999.

Procedure

1. Enable JMX in your Infinispan CR.

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: infinispan
   spec:
     jmx:
       enabled: true

   Copy to Clipboard Toggle word wrap

2. Retrieve the Operator user credentials to authenticate client JMX connections.

   oc get secret infinispan-generated-operator-secret -o jsonpath="{.data.identities\.yaml}" | base64 --decode

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Cryostat CR in the same namespace as your Infinispan CR.

   apiVersion: operator.cryostat.io/v1beta1
   kind: Cryostat
   metadata:
     name: cryostat-sample
   spec:
     minimal: false
     enableCertManager: true

   Copy to Clipboard Toggle word wrap
   Note

   The Cryostat Operator requires cert-manager for traffic encryption. If the cert-manager is enabled but not installed, the deployment fails. For details, see the Installing Cryostat guide.

2. Wait for the Cryostat CR to be ready.

   oc wait -n <namespace> --for=condition=MainDeploymentAvailable cryostat/cryostat-sample

   Copy to Clipboard Toggle word wrap

3. Open the Cryostat status.applicationUrl.

   oc -n <namespace> get cryostat cryostat-sample

   Copy to Clipboard Toggle word wrap

4. Retrieve the Operator user credentials to authenticate client JMX connections in the Cryostat UI.

   oc get secret infinispan-generated-operator-secret -o jsonpath="{.data.identities\.yaml}" | base64 --decode

   Copy to Clipboard Toggle word wrap
5. In the Cryostat UI, navigate to the Security menu.
6. In the Store Credentials window, click the Add button. The Store Credentials window opens.

7. In the Match Expression filed, enter match expression details in the following format:

   target.labels['infinispan_cr'] == '<cluster_name>'

   Copy to Clipboard Toggle word wrap