Chapter 13. Prometheus metrics monitoring in Red Hat Decision Manager

Procedure

1. In your Decision Server instance, set the org.kie.prometheus.server.ext.disabled system property to false to enable the Prometheus extension. You can define this property when you start Decision Server or in the standalone.xml or standalone-full.xml file of Red Hat Decision Manager distribution.

2. If you are running Red Hat Decision Manager on Spring Boot, add the following dependencies in the pom.xml file of your Maven project and configure the required key in the application.properties system property:

   Spring Boot pom.xml dependencies for Prometheus

   <dependency>
     <groupId>org.kie.server</groupId>
     <artifactId>kie-server-services-prometheus</artifactId>
     <version>${rhdm.version}</version>
   </dependency>
   
   <dependency>
     <groupId>org.kie.server</groupId>
     <artifactId>kie-server-rest-prometheus</artifactId>
     <version>${rhdm.version}</version>
   </dependency>

   Copy to Clipboard Toggle word wrap

   Spring Boot application.properties key for Red Hat Decision Manager and Prometheus

   kieserver.drools.enabled=true
   kieserver.dmn.enabled=true
   kieserver.prometheus.enabled=true

   Copy to Clipboard Toggle word wrap

3. In the prometheus.yaml file of your Prometheus distribution, add the following settings in the scrape_configs section to configure Prometheus to scrape metrics from Decision Server:

   Scrape configurations in prometheus.yaml file

   scrape_configs:
     - job_name: 'kie-server'
       metrics_path: /SERVER_PATH/services/rest/metrics
       basic_auth:
         username: USER_NAME
         password: PASSWORD
       static_configs:
         - targets: ["HOST:PORT"]

   Copy to Clipboard Toggle word wrap

   Scrape configurations in prometheus.yaml file for Spring Boot (if applicable)

   scrape_configs:
     - job_name: 'kie'
       metrics_path: /rest/metrics
       static_configs:
         - targets: ["HOST:PORT"]

   Copy to Clipboard Toggle word wrap

   Replace the values according to your Decision Server location and settings.

4. Start the Decision Server instance.

   Example start command for Red Hat Decision Manager on Red Hat JBoss EAP

   $ cd ~/EAP_HOME/bin
   $ ./standalone.sh --c standalone-full.xml

   Copy to Clipboard Toggle word wrap

   After you start the configured Decision Server instance, Prometheus begins collecting metrics and Decision Server publishes the metrics to the REST API endpoint http://HOST:PORT/SERVER/services/rest/metrics (or on Spring Boot, to http://HOST:PORT/rest/metrics).

5. In a REST client or curl utility, send a REST API request with the following components to verify that Decision Server is publishing the metrics:

   For REST client:

   • Authentication: Enter the user name and password of the Decision Server user with the kie-server role.

   • HTTP Headers: Set the following header:

     • Accept: application/json
   • HTTP method: Set to GET.
   • URL: Enter the Decision Server REST API base URL and metrics endpoint, such as http://localhost:8080/kie-server/services/rest/metrics (or on Spring Boot, http://localhost:8080/rest/metrics).

   For curl utility:

   • -u: Enter the user name and password of the Decision Server user with the kie-server role.

   • -H: Set the following header:

     • accept: application/json
   • -X: Set to GET.
   • URL: Enter the Decision Server REST API base URL and metrics endpoint, such as http://localhost:8080/kie-server/services/rest/metrics (or on Spring Boot, http://localhost:8080/rest/metrics).

   curl -u 'baAdmin:password@1' -H "accept: application/json" -X GET "http://localhost:8080/kie-server/services/rest/metrics"

   Copy to Clipboard Toggle word wrap

   Example server response

   # HELP kie_server_container_started_total Kie Server Started Containers
   # TYPE kie_server_container_started_total counter
   kie_server_container_started_total{container_id="task-assignment-kjar-1.0",} 1.0
   # HELP solvers_running Number of solvers currently running
   # TYPE solvers_running gauge
   solvers_running 0.0
   # HELP dmn_evaluate_decision_nanosecond DMN Evaluation Time
   # TYPE dmn_evaluate_decision_nanosecond histogram
   # HELP solver_duration_seconds Time in seconds it took solver to solve the constraint problem
   # TYPE solver_duration_seconds summary
   solver_duration_seconds_count{solver_id="100tasks-5employees.xml",} 1.0
   solver_duration_seconds_sum{solver_id="100tasks-5employees.xml",} 179.828255925
   solver_duration_seconds_count{solver_id="24tasks-8employees.xml",} 1.0
   solver_duration_seconds_sum{solver_id="24tasks-8employees.xml",} 179.995759653
   # HELP drl_match_fired_nanosecond Drools Firing Time
   # TYPE drl_match_fired_nanosecond histogram
   # HELP dmn_evaluate_failed_count DMN Evaluation Failed
   # TYPE dmn_evaluate_failed_count counter
   # HELP kie_server_start_time Kie Server Start Time
   # TYPE kie_server_start_time gauge
   kie_server_start_time{name="myapp-kieserver",server_id="myapp-kieserver",location="http://myapp-kieserver-demo-monitoring.127.0.0.1.nip.io:80/services/rest/server",version="7.4.0.redhat-20190428",} 1.557221271502E12
   # HELP kie_server_container_running_total Kie Server Running Containers
   # TYPE kie_server_container_running_total gauge
   kie_server_container_running_total{container_id="task-assignment-kjar-1.0",} 1.0
   # HELP solver_score_calculation_speed Number of moves per second for a particular solver solving the constraint problem
   # TYPE solver_score_calculation_speed summary
   solver_score_calculation_speed_count{solver_id="100tasks-5employees.xml",} 1.0
   solver_score_calculation_speed_sum{solver_id="100tasks-5employees.xml",} 6997.0
   solver_score_calculation_speed_count{solver_id="24tasks-8employees.xml",} 1.0
   solver_score_calculation_speed_sum{solver_id="24tasks-8employees.xml",} 19772.0

   Copy to Clipboard Toggle word wrap

   If the metrics are not available in Decision Server, review and verify the Decision Server and Prometheus configurations described in this section.

   You can also interact with your collected metrics in the Prometheus expression browser at http://HOST:PORT/graph, or integrate your Prometheus data source with a data-graphing tool such as Grafana:

   Figure 13.1. Prometheus expression browser with Decision Server metrics

   

   View larger image

   

   Figure 13.2. Prometheus expression browser with Decision Server target

   

   View larger image

   

   Figure 13.3. Grafana dashboard with Decision Server metrics for DMN models

   

   View larger image

   

   Figure 13.4. Grafana dashboard with Decision Server metrics for solvers

   

   View larger image

   

Procedure

1. In the DeploymentConfig object of your Decision Server deployment on OpenShift, set the PROMETHEUS_SERVER_EXT_DISABLED environment variable to false to enable the Prometheus extension. You can set this variable in the OpenShift web console or use the oc command in a command terminal:

   oc set env dc/<dc_name> PROMETHEUS_SERVER_EXT_DISABLED=false -n <namespace>

   Copy to Clipboard Toggle word wrap

   If you have not yet deployed your Decision Server on OpenShift, then in the OpenShift template that you plan to use for your OpenShift deployment (for example, rhdm75-prod-immutable-kieserver.yaml), you can set the PROMETHEUS_SERVER_EXT_DISABLED template parameter to false to enable the Prometheus extension.

   If you are using the OpenShift Operator to deploy Decision Server on OpenShift, then in your Decision Server configuration, set the PROMETHEUS_SERVER_EXT_DISABLED environment variable to false to enable the Prometheus extension:

   apiVersion: app.kiegroup.org/v1
   kind: KieApp
   metadata:
     name: enable-prometheus
   spec:
     environment: rhpam-trial
     objects:
       servers:
       - env:
         - name: PROMETHEUS_SERVER_EXT_DISABLED
           value: "false"

   Copy to Clipboard Toggle word wrap

2. Create a service-metrics.yaml file to add a service that exposes the metrics from Decision Server to Prometheus:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       description: RHDM Prometheus metrics exposed
     labels:
       app: myapp-kieserver
       application: myapp-kieserver
       template: myapp-kieserver
       metrics: rhdm
     name: rhdm-app-metrics
   spec:
     ports:
       - name: web
         port: 8080
         protocol: TCP
         targetPort: 8080
     selector:
       deploymentConfig: myapp-kieserver
     sessionAffinity: None
     type: ClusterIP

   Copy to Clipboard Toggle word wrap

3. In a command terminal, use the oc command to apply the service-metrics.yaml file to your OpenShift deployment:

   oc apply -f service-metrics.yaml

   Copy to Clipboard Toggle word wrap
4. Create an OpenShift secret, such as metrics-secret, to access the Prometheus metrics on Decision Server. The secret must contain the "username" and "password" elements with Decision Server user credentials. For information about OpenShift secrets, see the Secrets chapter in the OpenShift Developer Guide.

5. Create a service-monitor.yaml file that defines the ServiceMonitor object. A service monitor enables Prometheus to connect to the Decision Server metrics service.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: rhdm-service-monitor
     labels:
       team: frontend
   spec:
     selector:
       matchLabels:
         metrics: rhdm
     endpoints:
       - port: web
         path: /services/rest/metrics
         basicAuth:
           password:
             name: metrics-secret
             key: password
           username:
             name: metrics-secret
             key: username

   Copy to Clipboard Toggle word wrap

6. In a command terminal, use the oc command to apply the service-monitor.yaml file to your OpenShift deployment:

   oc apply -f service-monitor.yaml

   Copy to Clipboard Toggle word wrap

   After you complete these configurations, Prometheus begins collecting metrics and Decision Server publishes the metrics to the REST API endpoint http://HOST:PORT/kie-server/services/rest/metrics.

   You can interact with your collected metrics in the Prometheus expression browser at http://HOST:PORT/graph, or integrate your Prometheus data source with a data-graphing tool such as Grafana.

   The host and port for the Prometheus expression browser location http://HOST:PORT/graph was defined in the route where you exposed the Prometheus web console when you installed the Prometheus Operator. For information about OpenShift routes, see the Routes chapter in the OpenShift Architecture documentation.

   Figure 13.5. Prometheus expression browser with Decision Server metrics

   

   View larger image

   

   Figure 13.6. Prometheus expression browser with Decision Server target

   

   View larger image

   

   Figure 13.7. Grafana dashboard with Decision Server metrics for DMN models

   

   View larger image

   

   Figure 13.8. Grafana dashboard with Decision Server metrics for solvers

   

   View larger image

   

Procedure

1. Create an empty Maven project and define the following packaging type and dependencies in the pom.xml file for the project:

   Example pom.xml file in the sample project

   <packaging>jar</packaging>
   
   <properties>
     <version.org.kie>7.26.0.Final-redhat-00005</version.org.kie>
   </properties>
   
   <dependencies>
     <dependency>
       <groupId>org.kie</groupId>
       <artifactId>kie-api</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie.server</groupId>
       <artifactId>kie-server-api</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie.server</groupId>
       <artifactId>kie-server-services-common</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie.server</groupId>
       <artifactId>kie-server-services-drools</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie.server</groupId>
       <artifactId>kie-server-services-prometheus</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie</groupId>
       <artifactId>kie-dmn-api</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.kie</groupId>
       <artifactId>kie-dmn-core</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.jbpm</groupId>
       <artifactId>jbpm-services-api</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.jbpm</groupId>
       <artifactId>jbpm-executor</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>org.optaplanner</groupId>
       <artifactId>optaplanner-core</artifactId>
       <version>${version.org.kie}</version>
     </dependency>
     <dependency>
       <groupId>io.prometheus</groupId>
       <artifactId>simpleclient</artifactId>
       <version>0.5.0</version>
     </dependency>
   </dependencies>

   Copy to Clipboard Toggle word wrap

2. Implement the relevant listener from the org.kie.server.services.prometheus.PrometheusMetricsProvider interface as part of the custom listener class that defines your custom Prometheus metrics, as shown in the following example:

   Sample implementation of the DMNRuntimeEventListener listener in a custom listener class

   package org.kie.server.ext.prometheus;
   
   import io.prometheus.client.Gauge;
   import org.kie.dmn.api.core.ast.DecisionNode;
   import org.kie.dmn.api.core.event.AfterEvaluateBKMEvent;
   import org.kie.dmn.api.core.event.AfterEvaluateContextEntryEvent;
   import org.kie.dmn.api.core.event.AfterEvaluateDecisionEvent;
   import org.kie.dmn.api.core.event.AfterEvaluateDecisionServiceEvent;
   import org.kie.dmn.api.core.event.AfterEvaluateDecisionTableEvent;
   import org.kie.dmn.api.core.event.BeforeEvaluateBKMEvent;
   import org.kie.dmn.api.core.event.BeforeEvaluateContextEntryEvent;
   import org.kie.dmn.api.core.event.BeforeEvaluateDecisionEvent;
   import org.kie.dmn.api.core.event.BeforeEvaluateDecisionServiceEvent;
   import org.kie.dmn.api.core.event.BeforeEvaluateDecisionTableEvent;
   import org.kie.dmn.api.core.event.DMNRuntimeEventListener;
   import org.kie.server.api.model.ReleaseId;
   import org.kie.server.services.api.KieContainerInstance;
   
   public class ExampleCustomPrometheusMetricListener implements DMNRuntimeEventListener {
   
       private final KieContainerInstance kieContainer;
   
       private final Gauge randomGauge = Gauge.build()
               .name("random_gauge_nanosecond")
               .help("Random gauge as an example of custom KIE Prometheus metric")
               .labelNames("container_id", "group_id", "artifact_id", "version", "decision_namespace", "decision_name")
               .register();
   
       public ExampleCustomPrometheusMetricListener(KieContainerInstance containerInstance) {
           kieContainer = containerInstance;
       }
   
       public void beforeEvaluateDecision(BeforeEvaluateDecisionEvent e) {
       }
   
       public void afterEvaluateDecision(AfterEvaluateDecisionEvent e) {
           DecisionNode decisionNode = e.getDecision();
           ReleaseId releaseId = kieContainer.getResource().getReleaseId();
           randomGauge.labels(kieContainer.getContainerId(), releaseId.getGroupId(),
                              releaseId.getArtifactId(), releaseId.getVersion(),
                              decisionNode.getModelName(), decisionNode.getModelNamespace())
                   .set((int) (Math.random() * 100));
       }
   
       public void beforeEvaluateBKM(BeforeEvaluateBKMEvent event) {
       }
   
       public void afterEvaluateBKM(AfterEvaluateBKMEvent event) {
       }
   
       public void beforeEvaluateContextEntry(BeforeEvaluateContextEntryEvent event) {
       }
   
       public void afterEvaluateContextEntry(AfterEvaluateContextEntryEvent event) {
       }
   
       public void beforeEvaluateDecisionTable(BeforeEvaluateDecisionTableEvent event) {
       }
   
       public void afterEvaluateDecisionTable(AfterEvaluateDecisionTableEvent event) {
       }
   
       public void beforeEvaluateDecisionService(BeforeEvaluateDecisionServiceEvent event) {
       }
   
       public void afterEvaluateDecisionService(AfterEvaluateDecisionServiceEvent event) {
       }
   }

   Copy to Clipboard Toggle word wrap

   The PrometheusMetricsProvider interface contains the required listeners for collecting Prometheus metrics. The interface is incorporated by the kie-server-services-prometheus dependency that you declared in your project pom.xml file.

   In this example, the ExampleCustomPrometheusMetricListener class implements the DMNRuntimeEventListener listener (from the PrometheusMetricsProvider interface) and defines the custom DMN metrics to be collected and stored by Prometheus.

3. Implement the PrometheusMetricsProvider interface as part of a custom metrics provider class that associates your custom listener with the PrometheusMetricsProvider interface, as shown in the following example:

   Sample implementation of the PrometheusMetricsProvider interface in a custom metrics provider class

   package org.kie.server.ext.prometheus;
   
   import org.jbpm.executor.AsynchronousJobListener;
   import org.jbpm.services.api.DeploymentEventListener;
   import org.kie.api.event.rule.AgendaEventListener;
   import org.kie.api.event.rule.DefaultAgendaEventListener;
   import org.kie.dmn.api.core.event.DMNRuntimeEventListener;
   import org.kie.server.services.api.KieContainerInstance;
   import org.kie.server.services.prometheus.PrometheusMetricsProvider;
   import org.optaplanner.core.impl.phase.event.PhaseLifecycleListener;
   import org.optaplanner.core.impl.phase.event.PhaseLifecycleListenerAdapter;
   
   public class MyPrometheusMetricsProvider implements PrometheusMetricsProvider {
   
       public DMNRuntimeEventListener createDMNRuntimeEventListener(KieContainerInstance kContainer) {
           return new ExampleCustomPrometheusMetricListener(kContainer);
       }
   
       public AgendaEventListener createAgendaEventListener(String kieSessionId, KieContainerInstance kContainer) {
           return new DefaultAgendaEventListener();
       }
   
       public PhaseLifecycleListener createPhaseLifecycleListener(String solverId) {
           return new PhaseLifecycleListenerAdapter() {
           };
       }
   
       public AsynchronousJobListener createAsynchronousJobListener() {
           return null;
       }
   
       public DeploymentEventListener createDeploymentEventListener() {
           return null;
       }
   }

   Copy to Clipboard Toggle word wrap

   In this example, the MyPrometheusMetricsProvider class implements the PrometheusMetricsProvider interface and includes your custom ExampleCustomPrometheusMetricListener listener class.

4. To make the new metrics provider discoverable for Decision Server, create a META-INF/services/org.kie.server.services.prometheus.PrometheusMetricsProvider file in your Maven project and add the fully qualified class name of the PrometheusMetricsProvider implementation class within the file. For this example, the file contains the single line org.kie.server.ext.prometheus.MyPrometheusMetricsProvider.
5. Build your project and copy the resulting JAR file into the ~/kie-server.war/WEB-INF/lib directory of your project. For example, on Red Hat JBoss EAP, the path to this directory is EAP_HOME/standalone/deployments/kie-server.war/WEB-INF/lib.

6. Start the Decision Server and deploy the built project to the running Decision Server. You can deploy the project using the Business Central interface or the Decision Server REST API (a PUT request to http://SERVER:PORT/kie-server/services/rest/server/containers/{containerId}).

   After your project is deployed on a running Decision Server, Prometheus begins collecting metrics and Decision Server publishes the metrics to the REST API endpoint http://HOST:PORT/SERVER/services/rest/metrics (or on Spring Boot, to http://HOST:PORT/rest/metrics).