Chapter 17. Troubleshooting Data Grid Server deployments

Data Grid Server provides aggregated reports in tar.gz archives that contain diagnostic information about server instances and host systems. The report provides details about CPU, memory, open files, network sockets and routing, threads, in addition to configuration and log files.

Modify the logging configuration for Data Grid Server at runtime to temporarily adjust logging to troubleshoot issues and perform root cause analysis.

Modifying the logging configuration through the CLI is a runtime-only operation, which means that changes:

You can inspect server-collected statistics for some Data Grid Server resources with the stats command.

Use the stats command either from the context of a resource that provides statistics (containers, caches) or with a path to such a resource:

stats

{
  "statistics_enabled" : true,
  "number_of_entries" : 0,
  "hit_ratio" : 0.0,
  "read_write_ratio" : 0.0,
  "time_since_start" : 0,
  "time_since_reset" : 49,
  "current_number_of_entries" : 0,
  "current_number_of_entries_in_memory" : 0,
  "off_heap_memory_used" : 0,
  "data_memory_used" : 0,
  "stores" : 0,
  "retrievals" : 0,
  "hits" : 0,
  "misses" : 0,
  "remove_hits" : 0,
  "remove_misses" : 0,
  "evictions" : 0,
  "average_read_time" : 0,
  "average_read_time_nanos" : 0,
  "average_write_time" : 0,
  "average_write_time_nanos" : 0,
  "average_remove_time" : 0,
  "average_remove_time_nanos" : 0,
  "required_minimum_number_of_nodes" : -1
}

stats /containers/default/caches/mycache

{
  "time_since_start" : -1,
  "time_since_reset" : -1,
  "current_number_of_entries" : -1,
  "current_number_of_entries_in_memory" : -1,
  "off_heap_memory_used" : -1,
  "data_memory_used" : -1,
  "stores" : -1,
  "retrievals" : -1,
  "hits" : -1,
  "misses" : -1,
  "remove_hits" : -1,
  "remove_misses" : -1,
  "evictions" : -1,
  "average_read_time" : -1,
  "average_read_time_nanos" : -1,
  "average_write_time" : -1,
  "average_write_time_nanos" : -1,
  "average_remove_time" : -1,
  "average_remove_time_nanos" : -1,
  "required_minimum_number_of_nodes" : -1
}

You can define Java Virtual Machine (JVM) settings for Data Grid either by editing the server.conf configuration file, or by setting the JAVA_OPTS and JAVA_OPTIONS environment variables. Set the JAVA_OPTIONS variable if you just want to append JVM settings to those that are automatically set by the server.conf file. Set the JAVA_OPTS variable if you want to completely override the JVM settings.

Editing the configuration file

You can edit the required values in the server.conf configuration file. For example, to set the options to pass to the JVM, edit the following lines:

if [ "$JAVA_OPTS" = "" ]; then
   JAVA_OPTS="..."
else
   echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi

You can uncomment the existing example settings as well. For example, to configure Java Platform Debugger Architecture (JPDA) settings for remote socket debugging, update the file as follows:

# Sample JPDA settings for remote socket debugging
JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n"

Additionally, you can add more settings to JAVA_OPTS like this:

JAVA_OPTS="$JAVA_OPTS <key_1>=<value_1>, ..., <key_N>=<value_N> "

Setting an environment variable

You can override the settings in server.conf configuration file by setting the JAVA_OPTS environment variable. For example:

export JAVA_OPTS="-Xmx1024M"

set JAVA_OPTS="-Xmx1024M"

export GC_LOG="false"

set "GC_LOG=false"

export HEAP_DUMP="false"

set "HEAP_DUMP=false"

Infinispan supports virtual threads, which can significantly improve application responsiveness and scalability under high concurrency. By default, they are enabled if you are running on JDK 21 or higher.

Get Data Grid cluster health via the REST API.

Data Grid responds with a JSON document such as the following:

{
    "cluster_health":{
        "cluster_name":"ISPN",
        "health_status":"HEALTHY",
        "number_of_nodes":2,
        "node_names":[
            "NodeA-36229",
            "NodeB-28703"
        ]
    },
    "cache_health":[
        {
            "status":"HEALTHY",
            "cache_name":"___protobuf_metadata"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache2"
        },
        {
            "status":"HEALTHY",
            "cache_name":"mycache"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache1"
        }
    ]

}

GET /rest/v2/container/health/status

Retrieve Data Grid cluster health statistics via JMX.

Data Grid provides a built-in benchmark utility accessible through the CLI to validate server configuration changes. This benchmark will run operations for writes and reads, submitting requests as fast as possible. Since this might be a contrived scenario that does not represent a real production environment, the numbers should be read carefully. Nevertheless, this approach is simple enough to verify for configuration changes.

Check the CLI command documentation for more information.