Chapter 7. Monitoring your brokers
You can use Fuse Console for OpenShift instead of AMQ Management Console to view runtime data and perform basic management operations. You can also use the Prometheus plugin for AMQ Broker to collect runtime metrics, which you can use monitor the health and performance of brokers.
7.1. Viewing brokers in Fuse Console Copy linkLink copied to clipboard!
You can configure an Operator-based broker deployment to use Fuse Console for OpenShift instead of AMQ Management Console. After you configure your broker deployment, Fuse Console discovers the brokers and displays them on a dedicated Artemis tab.
You can view the same broker runtime data in Fuse Console for OpenShift as in AMQ Management Console. You can also perform the same basic management operations, such as creating addresses and queues.
The following procedure describes how to configure the Custom Resource (CR) instance for a broker deployment to enable Fuse Console for OpenShift to discover and display brokers in the deployment.
Prerequisites
- Fuse Console for OpenShift must be deployed to an OCP cluster, or to a specific namespace on that cluster. If you have deployed the console to a specific namespace, your broker deployment must be in the same namespace, to enable the console to discover the brokers. Otherwise, it is sufficient for Fuse Console and the brokers to be deployed on the same OCP cluster.
- You must have already created a broker deployment. For example, to learn how to use a Custom Resource (CR) instance to create a basic Operator-based deployment, see Section 3.3.1, “Deploying a single broker instance”.
Procedure
Open the CR instance that you used for your broker deployment. For example, the CR for a basic deployment might resemble the following:
apiVersion: broker.amq.io/v1beta1 kind: ActiveMQArtemis metadata: name: ex-aao spec: deploymentPlan: size: 4 image: registry.redhat.io/amq7/amq-broker-rhel8:7.13 ...In the
deploymentPlansection, add thejolokiaAgentEnabledandmanagementRBACEnabledproperties and specify values, as shown below.apiVersion: broker.amq.io/v1beta1 kind: ActiveMQArtemis metadata: name: ex-aao spec: deploymentPlan: size: 4 image: registry.redhat.io/amq7/amq-broker-rhel8:7.13 ... jolokiaAgentEnabled: true managementRBACEnabled: false- jolokiaAgentEnabled
-
Specifies whether Fuse Console can discover and display runtime data for the brokers in the deployment. To use Fuse Console, set the value to
true. - managementRBACEnabled
Specifies whether role-based access control (RBAC) is enabled for the brokers in the deployment. You must set the value to
falseto use Fuse Console because Fuse Console uses its own role-based access control.ImportantIf you set the value of
managementRBACEnabledtofalseto enable use of Fuse Console, management MBeans for the brokers no longer require authorization. You should not use the AMQ management console whilemanagementRBACEnabledis set tofalsebecause this potentially exposes all management operations on the brokers to unauthorized use.
- Save the CR instance.
Switch to the project in which you previously created your broker deployment.
$ oc project <project_name>At the command line, apply the change.
$ oc apply -f <path/to/custom_resource_instance>.yaml- In Fuse Console, click the Online tab to view Fuse applications. To view running brokers, in the left navigation menu, click Artemis.
7.2. Monitoring broker runtime metrics using Prometheus Copy linkLink copied to clipboard!
You can use the Prometheus metrics plugin for AMQ Broker on OpenShift Container Platform to collect and store broker runtime metrics. You might also use a graphical tool such as Grafana to configure more advanced visualizations and dashboards of the data that the Prometheus plugin collects.
The Prometheus metrics plugin enables you to collect and export broker metrics in Prometheus format. However, Red Hat does not provide support for installation or configuration of Prometheus itself, nor of visualization tools such as Grafana. If you require support with installing, configuring, or running Prometheus or Grafana, visit the product websites for resources such as community support and documentation.
7.2.1. Metrics overview Copy linkLink copied to clipboard!
You can use the Prometheus plugin for AMQ Broker to collect runtime metrics, which you can use monitor the health and performance of brokers. The AMQ Broker Prometheus plugin exports the runtime metrics in Prometheus format, enabling you to use Prometheus itself to visualize and query the data.
You can also use a graphical tool, such as Grafana, to configure more advanced visualizations and dashboards for the metrics that the Prometheus plugin collects.
The metrics that the plugin exports to Prometheus format are described below.
For higher-level broker metrics that are not listed below, you can calculate these by aggregating lower-level metrics. For example, to calculate total message count, you can aggregate the artemis_message_count metrics from all queues in your broker deployment.
For a standalone deployment of AMQ Broker on RHEL, metrics for the Java Virtual Machine (JVM) hosting the broker are also exported to Prometheus format. This does not apply to a deployment of AMQ Broker on OpenShift Container Platform.
7.2.1.1. Broker metrics Copy linkLink copied to clipboard!
artemis_address_memory_usage- Number of bytes used by all addresses on this broker for in-memory messages.
artemis_address_memory_usage_percentage-
Memory used by all the addresses on this broker as a percentage of the
global-max-sizeparameter. artemis_connection_count- Number of clients connected to this broker.
artemis_session_count- Number of sessions on this broker.
artemis_total_session_count- Total number of sessions created on this broker since it was started.
artemis_total_connection_count- Number of clients that have connected to this broker since it was started.
artemis_disk_store_usage- The fraction of the total disk store used.
artemis_replica_sync- Is the initial replication synchronization process between primary and backup brokers complete.
artemis_active- Is the broker active in a primary-backup group configuration.
artemis_authentication_count- Number of successful and failed authentication attempts.
artemis_authorization_count- Number of successful and failed authorization attempts.
7.2.1.2. Address metrics Copy linkLink copied to clipboard!
artemis_routed_message_count- Number of messages routed to one or more queue bindings.
artemis_unrouted_message_count- Number of messages not routed to any queue bindings.
artemis_address_size- The size in bytes used by all queues bound to this address.
artemis_number_of_pages- The number of page files used to store messages for this address.
artemis_limit_percent-
The percentage of a configured memory limit that is used by this address. If a memory limit is set for the address by using the
max-size-bytesattribute, the percentage reported is based on that value. If themax-size-bytesattribute is not set, the percentage reported is based on the value of theglobal-max-sizeattribute. If neither attribute is set, a value of 0 is reported.
7.2.1.3. Queue metrics Copy linkLink copied to clipboard!
artemis_consumer_count- Number of clients consuming messages from a given queue.
artemis_delivering_durable_message_count- Number of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_durable_persistent_size- Persistent size of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_message_count- Number of messages that a given queue is currently delivering to consumers.
artemis_delivering_persistent_size- Persistent size of messages that a given queue is currently delivering to consumers.
artemis_durable_message_count- Number of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_durable_persistent_size- Persistent size of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_acknowledged- Number of messages acknowledged from a given queue since the queue was created.
artemis_messages_added- Number of messages added to a given queue since the queue was created.
artemis_message_count- Number of messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_killed- Number of messages removed from a given queue since the queue was created. The broker kills a message when the message exceeds the configured maximum number of delivery attempts.
artemis_messages_expired- Number of messages expired from a given queue since the queue was created.
artemis_persistent_size- Persistent size of all messages (both durable and non-durable) currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_scheduled_durable_message_count- Number of durable, scheduled messages in a given queue.
artemis_scheduled_durable_persistent_size- Persistent size of durable, scheduled messages in a given queue.
artemis_scheduled_message_count- Number of scheduled messages in a given queue.
artemis_scheduled_persistent_size- Persistent size of scheduled messages in a given queue.
7.2.2. Enabling the Prometheus plugin using a CR Copy linkLink copied to clipboard!
When you install AMQ Broker, a Prometheus metrics plugin is included in your installation. You must enable the plugin to collect runtime metrics for the broker and exports these to Prometheus format.
See Section 7.2.3, “Enabling the Prometheus plugin for a running broker deployment using an environment variable” for an alternative procedure with running brokers.
Procedure
-
Edit the
ActiveMQArtemisCR instance that you use for your broker deployment. In the
deploymentPlansection, add theenableMetricsPluginattribute and set the value totrue, as shown below.apiVersion: broker.amq.io/v1beta1 kind: ActiveMQArtemis metadata: name: amq-with-metrics spec: deploymentPlan: enableMetricsPlugin: true ...Save the CR instance.
The metrics plugin starts to gather broker runtime metrics in Prometheus format.
NoteAfter you enable the Prometheus plugin, you must create a
ServiceMonitorcustom resource (CR) for the Prometheus Operator. Prometheus uses the CR to find the endpoint where the broker metrics are exposed. For example:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: amq-with-metrics-monitor spec: selector: matchLabels: application: amq-with-metrics-app endpoints: - port: console-jolokiaNoteFor the
applicationattribute in theServiceMonitorCR, specify the value of themetadata.nameattribute that is in theActiveMQArtemisCR and append a suffix of-app.
Additional resources
7.2.3. Enabling the Prometheus plugin for a running broker deployment using an environment variable Copy linkLink copied to clipboard!
You can enable the Prometheus metrics plugin by setting the AMQ_ENABLE_METRICS_PLUGIN environment variable to true in the custom resource (CR) for your broker.
See Section 7.2.2, “Enabling the Prometheus plugin using a CR” for an alternative procedure.
Prerequisites
- You can enable the Prometheus plugin for a broker Pod created with the AMQ Broker Operator. However, your deployed broker must use the broker container image for AMQ Broker 7.7 or later.
Procedure
- Log in to the OpenShift Container Platform web console with administrator privileges for the project that contains your broker deployment.
- In the web console, click menu:Home[Projects]. Choose the project that contains your broker deployment.
- To see the StatefulSets or DeploymentConfigs in your project, click menu:Workloads[StatefulSets] or menu:Workloads[DeploymentConfigs].
- Click the StatefulSet or DeploymentConfig that corresponds to your broker deployment.
- To access the environment variables for your broker deployment, click the Environment tab.
Add a new environment variable,
AMQ_ENABLE_METRICS_PLUGIN. Set the value of the variable totrue.When you set the
AMQ_ENABLE_METRICS_PLUGINenvironment variable, OpenShift restarts each broker Pod in the StatefulSet or DeploymentConfig. When there are multiple Pods in the deployment, OpenShift restarts each Pod in turn. When each broker Pod restarts, the Prometheus plugin for that broker starts to gather broker runtime metrics.
7.2.4. Accessing Prometheus metrics for a running broker Pod Copy linkLink copied to clipboard!
You can view Prometheus metrics directly in a web browser by appending /metrics to the URL of the route that was created when you exposed AMQ Management Console.
Prerequisites
- You must have already enabled the Prometheus plugin for your broker pod. See Section 7.2.3, “Enabling the Prometheus plugin for a running broker deployment using an environment variable”.
Procedure
For the broker pod whose metrics you want to access, you need to identify the route you previously created to connect the Pod to the AMQ Broker management console. The route name forms part of the URL needed to access the metrics.
- Click menu:Networking[Routes].
For your chosen broker pod, identify the route created to connect the pod to the AMQ Broker management console. Under Hostname, note the complete URL that is shown. For example:
http://rte-console-access-pod1.openshiftdomain
To access Prometheus metrics, in a web browser, enter the previously noted route name appended with
“/metrics”. For example:http://rte-console-access-pod1.openshiftdomain/metricsNoteIf your console configuration does not use SSL, specify
httpin the URL. In this case, DNS resolution of the host name directs traffic to port 80 of the OpenShift router. If your console configuration uses SSL, specifyhttpsin the URL. In this case, your browser defaults to port 443 of the OpenShift router. This enables a successful connection to the console if the OpenShift router also uses port 443 for SSL traffic, which the router does by default.
7.3. Monitoring broker runtime data using JMX Copy linkLink copied to clipboard!
View an example of how to monitor a broker by using the Jolokia REST interface to JMX.
Prerequisites
- Completion of Deploying a basic broker is recommended.
Procedure
Get the list of running pods:
$ oc get pods NAME READY STATUS RESTARTS AGE ex-aao-ss-1 1/1 Running 0 14dRun the
oc logscommand:$ oc logs -f ex-aao-ss-1 ... Running Broker in /home/jboss/amq-broker ... 2021-09-17 09:35:10,813 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 2021-09-17 09:35:10,882 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging) 2021-09-17 09:35:10,971 INFO [org.apache.activemq.artemis.core.server] AMQ221013: Using NIO Journal 2021-09-17 09:35:11,114 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 2,566,914,048 2021-09-17 09:35:11,369 WARNING [org.jgroups.stack.Configurator] JGRP000014: BasicTCP.use_send_queues has been deprecated: will be removed in 4.0 2021-09-17 09:35:11,385 WARNING [org.jgroups.stack.Configurator] JGRP000014: Discovery.timeout has been deprecated: GMS.join_timeout should be used instead 2021-09-17 09:35:11,480 INFO [org.jgroups.protocols.openshift.DNS_PING] serviceName [ex-aao-ping-svc] set; clustering enabled 2021-09-17 09:35:24,540 INFO [org.openshift.ping.common.Utils] 3 attempt(s) with a 1000ms sleep to execute [GetServicePort] failed. Last failure was [javax.naming.CommunicationException: DNS error] ... 2021-09-17 09:35:25,044 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock 2021-09-17 09:35:25,045 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock 2021-09-17 09:35:25,206 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST] 2021-09-17 09:35:25,240 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ 2021-09-17 09:35:25,360 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST] 2021-09-17 09:35:25,362 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue 2021-09-17 09:35:25,656 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at ex-aao-ss-1.ex-aao-hdls-svc.broker.svc.cluster.local:61616 for protocols [CORE] 2021-09-17 09:35:25,660 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live 2021-09-17 09:35:25,660 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.16.0.redhat-00022 [amq-broker, nodeID=8d886031-179a-11ec-9e02-0a580ad9008b] 2021-09-17 09:35:26,470 INFO [org.apache.amq.hawtio.branding.PluginContextListener] Initialized amq-broker-redhat-branding plugin 2021-09-17 09:35:26,656 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin ...Run your query to monitor your broker for
MaxConsumers:$ curl -k -u admin:admin http://console-broker.amq-demo.apps.example.com/console/jolokia/read/org.apache.activemq.artemis:broker=%22amq-broker%22,component=addresses,address=%22TESTQUEUE%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22TESTQUEUE%22/MaxConsumers {"request":{"mbean":"org.apache.activemq.artemis:address=\"TESTQUEUE\",broker=\"amq-broker\",component=addresses,queue=\"TESTQUEUE\",routing-type=\"anycast\",subcomponent=queues","attribute":"MaxConsumers","type":"read"},"value":-1,"timestamp":1528297825,"status":200}