Chapter 16. Troubleshooting Data Grid Server deployments
Gather diagnostic information about Data Grid Server deployments and perform troubleshooting steps to resolve issues.
16.1. Getting diagnostic reports from Data Grid Server
Data Grid Server provides aggregated reports in tar.gz archives that contain diagnostic information about server instances and host systems. The report provides details about CPU, memory, open files, network sockets and routing, threads, in addition to configuration and log files.
Procedure
- Create a CLI connection to Data Grid Server.
Use the
server reportcommand to download atar.gzarchive:server report Downloaded report 'infinispan-<hostname>-<timestamp>-report.tar.gz'
The command responds with the name of the report, as in the following example:
Downloaded report 'infinispan-<hostname>-<timestamp>-report.tar.gz'
-
Move the
tar.gzfile to a suitable location on your filesystem. -
Extract the
tar.gzfile with any archiving tool.
16.2. Changing Data Grid Server logging configuration at runtime
Modify the logging configuration for Data Grid Server at runtime to temporarily adjust logging to troubleshoot issues and perform root cause analysis.
Modifying the logging configuration through the CLI is a runtime-only operation, which means that changes:
-
Are not saved to the
log4j2.xmlfile. Restarting server nodes or the entire cluster resets the logging configuration to the default properties in thelog4j2.xmlfile. - Apply only to the nodes in the cluster when you invoke the CLI. Nodes that join the cluster after you change the logging configuration use the default properties.
Procedure
- Create a CLI connection to Data Grid Server.
Use the
loggingto make the required adjustments.List all appenders defined on the server:
logging list-appenders
The command provides a JSON response such as the following:
{ "STDOUT" : { "name" : "STDOUT" }, "JSON-FILE" : { "name" : "JSON-FILE" }, "HR-ACCESS-FILE" : { "name" : "HR-ACCESS-FILE" }, "FILE" : { "name" : "FILE" }, "REST-ACCESS-FILE" : { "name" : "REST-ACCESS-FILE" } }List all logger configurations defined on the server:
logging list-loggers
The command provides a JSON response such as the following:
[ { "name" : "", "level" : "INFO", "appenders" : [ "STDOUT", "FILE" ] }, { "name" : "org.infinispan.HOTROD_ACCESS_LOG", "level" : "INFO", "appenders" : [ "HR-ACCESS-FILE" ] }, { "name" : "com.arjuna", "level" : "WARN", "appenders" : [ ] }, { "name" : "org.infinispan.REST_ACCESS_LOG", "level" : "INFO", "appenders" : [ "REST-ACCESS-FILE" ] } ]Add and modify logger configurations with the
setsubcommandFor example, the following command sets the logging level for the
org.infinispanpackage toDEBUG:logging set --level=DEBUG org.infinispan
Remove existing logger configurations with the
removesubcommand.For example, the following command removes the
org.infinispanlogger configuration, which means the root configuration is used instead:logging remove org.infinispan
16.3. Gathering resource statistics from the CLI
You can inspect server-collected statistics for some Data Grid Server resources with the stats command.
Use the stats command either from the context of a resource that provides statistics (containers, caches) or with a path to such a resource:
stats
{
"statistics_enabled" : true,
"number_of_entries" : 0,
"hit_ratio" : 0.0,
"read_write_ratio" : 0.0,
"time_since_start" : 0,
"time_since_reset" : 49,
"current_number_of_entries" : 0,
"current_number_of_entries_in_memory" : 0,
"total_number_of_entries" : 0,
"off_heap_memory_used" : 0,
"data_memory_used" : 0,
"stores" : 0,
"retrievals" : 0,
"hits" : 0,
"misses" : 0,
"remove_hits" : 0,
"remove_misses" : 0,
"evictions" : 0,
"average_read_time" : 0,
"average_read_time_nanos" : 0,
"average_write_time" : 0,
"average_write_time_nanos" : 0,
"average_remove_time" : 0,
"average_remove_time_nanos" : 0,
"required_minimum_number_of_nodes" : -1
}stats /containers/default/caches/mycache
{
"time_since_start" : -1,
"time_since_reset" : -1,
"current_number_of_entries" : -1,
"current_number_of_entries_in_memory" : -1,
"total_number_of_entries" : -1,
"off_heap_memory_used" : -1,
"data_memory_used" : -1,
"stores" : -1,
"retrievals" : -1,
"hits" : -1,
"misses" : -1,
"remove_hits" : -1,
"remove_misses" : -1,
"evictions" : -1,
"average_read_time" : -1,
"average_read_time_nanos" : -1,
"average_write_time" : -1,
"average_write_time_nanos" : -1,
"average_remove_time" : -1,
"average_remove_time_nanos" : -1,
"required_minimum_number_of_nodes" : -1
}16.4. Accessing cluster health via REST
Get Data Grid cluster health via the REST API.
Procedure
Invoke a
GETrequest to retrieve cluster health.GET /rest/v2/cache-managers/{cacheManagerName}/health
Data Grid responds with a JSON document such as the following:
{
"cluster_health":{
"cluster_name":"ISPN",
"health_status":"HEALTHY",
"number_of_nodes":2,
"node_names":[
"NodeA-36229",
"NodeB-28703"
]
},
"cache_health":[
{
"status":"HEALTHY",
"cache_name":"___protobuf_metadata"
},
{
"status":"HEALTHY",
"cache_name":"cache2"
},
{
"status":"HEALTHY",
"cache_name":"mycache"
},
{
"status":"HEALTHY",
"cache_name":"cache1"
}
]
}Get cache manager status as follows:
GET /rest/v2/cache-managers/{cacheManagerName}/health/statusReference
See the REST v2 (version 2) API documentation for more information.
16.5. Accessing cluster health via JMX
Retrieve Data Grid cluster health statistics via JMX.
Procedure
Connect to Data Grid server using any JMX capable tool such as JConsole and navigate to the following object:
org.infinispan:type=CacheManager,name="default",component=CacheContainerHealth
- Select available MBeans to retrieve cluster health statistics.
