Chapter 5. Monitoring Data Grid Servers
5.1. Working with Data Grid Server Logs
Data Grid uses Apache Log4j 2 to provide configurable logging mechanisms that capture details about the environment and record cache operations for troubleshooting purposes and root cause analysis.
5.1.1. Data Grid Log Files
Data Grid writes log messages to the following directory:$RHDG_HOME/${infinispan.server.root}/log
server.log
-
Messages in human readable format, including boot logs that relate to the server startup.
Data Grid creates this file by default when you launch servers. server.log.json
-
Messages in JSON format that let you parse and analyze Data Grid logs.
Data Grid creates this file when you enable theJSON-FILE
appender.
5.1.2. Configuring Data Grid Log Properties
You configure Data Grid logs with log4j2.xml
, which is described in the Log4j 2 manual.
Procedure
-
Open
$RHDG_HOME/${infinispan.server.root}/conf/log4j2.xml
with any text editor. - Change logging configuration as appropriate.
-
Save and close
log4j2.xml
.
5.1.2.1. Log Levels
Log levels indicate the nature and severity of messages.
Log level | Description |
---|---|
| Fine-grained debug messages, capturing the flow of individual requests through the application. |
| Messages for general debugging, not related to an individual request. |
| Messages about the overall progress of applications, including lifecycle events. |
| Events that can lead to error or degrade performance. |
| Error conditions that might prevent operations or activites from being successful but do not prevent applications from running. |
| Events that could cause critical service failure and application shutdown. |
In addition to the levels of individual messages presented above, the configuration allows two more values: ALL
to include all messages, and OFF
to exclude all messages.
5.1.2.2. Data Grid Log Categories
Data Grid provides categories for INFO
, WARN
, ERROR
, FATAL
level messages that organize logs by functional area.
org.infinispan.CLUSTER
- Messages specific to Data Grid clustering that include state transfer operations, rebalancing events, partitioning, and so on.
org.infinispan.CONFIG
- Messages specific to Data Grid configuration.
org.infinispan.CONTAINER
- Messages specific to the data container that include expiration and eviction operations, cache listener notifications, transactions, and so on.
org.infinispan.PERSISTENCE
- Messages specific to cache loaders and stores.
org.infinispan.SECURITY
- Messages specific to Data Grid security.
org.infinispan.SERVER
- Messages specific to Data Grid servers.
org.infinispan.XSITE
- Messages specific to cross-site replication operations.
5.1.2.3. Log Appenders
Log appenders define how Data Grid records log messages.
- CONSOLE
-
Write log messages to the host standard out (
stdout
) or standard error (stderr
) stream.
Uses theorg.apache.logging.log4j.core.appender.ConsoleAppender
class by default. - FILE
-
Write log messages to a file.
Uses theorg.apache.logging.log4j.core.appender.RollingFileAppender
class by default. - JSON-FILE
-
Write log messages to a file in JSON format.
Uses theorg.apache.logging.log4j.core.appender.RollingFileAppender
class by default.
5.1.2.4. Log Patterns
The CONSOLE
and FILE
appenders use a PatternLayout
to format the log messages according to a pattern.
An example is the default pattern in the FILE appender:%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c{1}] %m%throwable%n
-
%d{yyyy-MM-dd HH:mm:ss,SSS}
adds the current time and date. -
%-5p
specifies the log level, aligned to the right. -
%t
adds the name of the current thread. -
%c{1}
adds the short name of the logging category. -
%m
adds the log message. -
%throwable
adds the exception stack trace. -
%n
adds a new line.
Patterns are fully described in the PatternLayout
documentation .
5.1.2.5. Enabling and Configuring the JSON Log Handler
Data Grid provides a JSON log handler to write messages in JSON format.
Prerequisites
Ensure that Data Grid is not running. You cannot dynamically enable log handlers.
Procedure
When you start Data Grid, it writes each log message as a JSON map in the following file:$RHDG_HOME/${infinispan.server.root}/log/server.log.json
5.1.3. Access Logs
Hot Rod and REST endpoints can record all inbound client requests as log entries with the following categories:
-
org.infinispan.HOTROD_ACCESS_LOG
logging category for the Hot Rod endpoint. -
org.infinispan.REST_ACCESS_LOG
logging category for the REST endpoint.
5.1.3.1. Enabling Access Logs
Access logs for Hot Rod and REST endpoints are disabled by default. To enable either logging category, set the level to TRACE
in the Data Grid logging configuration, as in the following example:
<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="INFO"> <AppenderRef ref="HR-ACCESS-FILE"/> </Logger>
5.1.3.2. Access Log Properties
The default format for access logs is as follows:
`%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] "%X{method} %m %X{protocol}" %X{status} %X{requestSize} %X{responseSize} %X{duration}%n`
The preceding format creates log entries such as the following:
127.0.0.1 - [DD/MM/YYYY:HH:MM:SS +0000] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10
Logging properties use the %X{name}
notation and let you modify the format of access logs. The following are the default logging properties:
Property | Description |
---|---|
|
Either the |
| Principal name, if using authentication. |
|
Method used. |
|
Protocol used. |
|
An HTTP status code for the REST endpoint. |
| Size, in bytes, of the request. |
| Size, in bytes, of the response. |
| Number of milliseconds that the server took to handle the request. |
Use the header name prefixed with h:
to log headers that were included in requests; for example, %X{h:User-Agent}
.
5.2. Configuring Statistics, Metrics, and JMX
Enable statistics that Data Grid exports to a MicroProfile Metrics endpoint or via JMX MBeans. You can also register JMX MBeans to perform management operations.
5.2.1. Enabling Data Grid Statistics
Data Grid lets you enable statistics for Cache Managers and specific cache instances.
Data Grid servers provide default infinispan.xml
configuration files that enable statistics for Cache Managers. If you use the default Data Grid server configuration, you only need to enable statistics when you configure caches.
Procedure
- Enable statistics declaratively or programmatically.
Declaratively
<cache-container statistics="true"> 1 <local-cache name="mycache" statistics="true"/> 2 </cache-container>
Programmatically
GlobalConfiguration globalConfig = new GlobalConfigurationBuilder() .cacheContainer().statistics(true) 1 .build(); ... Configuration config = new ConfigurationBuilder() .statistics().enable() 2 .build();
Enabling statistics for Cache Managers does not globally enable statistics for caches. This configuration only collects statistics for the Cache Manager.
5.2.2. Enabling Data Grid Metrics
Data Grid is compatible with the Eclipse MicroProfile Metrics API and can generate gauge and histogram metrics.
-
Data Grid metrics are provided at the
vendor
scope. Metrics related to the JVM are provided in thebase
scope for Data Grid server. - Gauges provide values such as the average number of nanoseconds for write operations or JVM uptime. Gauges are enabled by default. If you enable statistics, Data Grid automatically generates gauges.
- Histograms provide details about operation execution times such as read, write, and remove times. Data Grid does not enable histograms by default because they require additional computation.
Procedure
- Configure metrics declaratively or programmatically.
Declaratively
<cache-container statistics="true"> 1 <metrics gauges="true" histograms="true" /> 2 </cache-container>
Programmatically
GlobalConfiguration globalConfig = new GlobalConfigurationBuilder() .statistics().enable() 1 .metrics().gauges(true).histograms(true) 2 .build();
5.2.3. Collecting Data Grid Metrics
Collect Data Grid metrics with monitoring tools such as Prometheus.
Prerequisites
-
Enable statistics. If you do not enable statistics, Data Grid provides
0
and-1
values for metrics. - Optionally enable histograms. By default Data Grid generates gauges but not histograms.
Procedure
Get metrics in Prometheus (OpenMetrics) format:
$ curl -v http://localhost:11222/metrics
Get metrics in MicroProfile JSON format:
$ curl --header "Accept: application/json" http://localhost:11222/metrics
Next steps
Configure monitoring applications to collect Data Grid metrics. For example, add the following to prometheus.yml
:
static_configs: - targets: ['localhost:11222']
5.2.4. Enabling and Configuring JMX
You can enable JMX with Data Grid servers to collect statistics and perform administrative operations.
You can enable JMX without enabling statistics if you want to use management operations only. In this case, Data Grid provides 0
values for all statistics.
Procedure
- Enable JMX declaratively or programmatically.
Declaratively
<cache-container>
<jmx enabled="true" /> 1
</cache-container>
Programmatically
GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
.jmx().enable() 1
.build();
Reference
5.2.4.1. Naming Multiple Cache Managers
In cases where multiple Data Grid Cache Managers run on the same JVM, you should uniquely identify each Cache Manager to prevent conflicts.
Procedure
- Uniquely identify each cache manager in your environment.
For example, the following examples specify "Hibernate2LC" as the cache manager name, which results in a JMX MBean named org.infinispan:type=CacheManager,name="Hibernate2LC"
.
Declaratively
<cache-container name="Hibernate2LC"> <jmx enabled="true" /> ... </cache-container>
Programmatically
GlobalConfiguration globalConfig = new GlobalConfigurationBuilder() .cacheManagerName("Hibernate2LC") .jmx().enable() .build();
5.2.4.2. Registering MBeans In Custom MBean Servers
Data Grid includes an MBeanServerLookup
interface that you can use to register MBeans in custom MBeanServer instances.
Procedure
-
Create an implementation of
MBeanServerLookup
so that thegetMBeanServer()
method returns the custom MBeanServer instance. - Configure Data Grid with the fully qualified name of your class, as in the following example:
Declaratively
<cache-container> <jmx enabled="true" mbean-server-lookup="com.acme.MyMBeanServerLookup" /> </cache-container>
Programmatically
GlobalConfiguration globalConfig = new GlobalConfigurationBuilder() .jmx().enable().mBeanServerLookup(new com.acme.MyMBeanServerLookup()) .build();
5.2.4.3. Data Grid MBeans
Data Grid exposes JMX MBeans that represent manageable resources.
org.infinispan:type=Cache
- Attributes and operations available for cache instances.
org.infinispan:type=CacheManager
- Attributes and operations available for cache managers, including Data Grid cache and cluster health statistics.
For a complete list of available JMX MBeans along with descriptions and available operations and attributes, see the Data Grid JMX Components documentation.
Reference
5.3. Retrieving Server Health Statistics
Monitor the health of your Data Grid clusters in the following ways:
-
Programmatically with
embeddedCacheManager.getHealth()
method calls. - JMX MBeans
- Data Grid REST Server
5.3.1. Accessing the Health API via JMX
Retrieve Data Grid cluster health statistics via JMX.
Procedure
Connect to Data Grid server using any JMX capable tool such as JConsole and navigate to the following object:
org.infinispan:type=CacheManager,name="default",component=CacheContainerHealth
- Select available MBeans to retrieve cluster health statistics.
5.3.2. Accessing the Health API via REST
Get Data Grid cluster health via the REST API.
Procedure
Invoke a
GET
request to retrieve cluster health.GET /rest/v2/cache-managers/{cacheManagerName}/health
Data Grid responds with a JSON
document such as the following:
{ "cluster_health":{ "cluster_name":"ISPN", "health_status":"HEALTHY", "number_of_nodes":2, "node_names":[ "NodeA-36229", "NodeB-28703" ] }, "cache_health":[ { "status":"HEALTHY", "cache_name":"___protobuf_metadata" }, { "status":"HEALTHY", "cache_name":"cache2" }, { "status":"HEALTHY", "cache_name":"mycache" }, { "status":"HEALTHY", "cache_name":"cache1" } ] }
Get cache manager status as follows:
GET /rest/v2/cache-managers/{cacheManagerName}/health/status
Reference
See the REST v2 (version 2) API documentation for more information.