Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 16. Troubleshooting Data Grid Server deployments

Gather diagnostic information about Data Grid Server deployments and perform troubleshooting steps to resolve issues.

Data Grid Server provides aggregated reports in tar.gz archives that contain diagnostic information about server instances and host systems. The report provides details about CPU, memory, open files, network sockets and routing, threads, in addition to configuration and log files.

Procedure

  1. Create a CLI connection to Data Grid Server.

  2. Use the server report command to download a tar.gz archive: 

    server report
Downloaded report 'infinispan-<hostname>-<timestamp>-report.tar.gz'
    Copy to Clipboard Toggle word wrap

    The command responds with the name of the report, as in the following example: 

    Downloaded report 'infinispan-<hostname>-<timestamp>-report.tar.gz'
    Copy to Clipboard Toggle word wrap
  3. Move the tar.gz file to a suitable location on your filesystem.
  4. Extract the tar.gz file with any archiving tool.

Modify the logging configuration for Data Grid Server at runtime to temporarily adjust logging to troubleshoot issues and perform root cause analysis.

Modifying the logging configuration through the CLI is a runtime-only operation, which means that changes:

  • Are not saved to the log4j2.xml file. Restarting server nodes or the entire cluster resets the logging configuration to the default properties in the log4j2.xml file.
  • Apply only to the nodes in the cluster when you invoke the CLI. Nodes that join the cluster after you change the logging configuration use the default properties.

Procedure

  1. Create a CLI connection to Data Grid Server.

  2. Use the logging to make the required adjustments.

    • List all appenders defined on the server: 

      logging list-appenders
      Copy to Clipboard Toggle word wrap

      The command provides a JSON response such as the following: 

      {
  "STDOUT" : {
    "name" : "STDOUT"
  },
  "JSON-FILE" : {
    "name" : "JSON-FILE"
  },
  "HR-ACCESS-FILE" : {
    "name" : "HR-ACCESS-FILE"
  },
  "FILE" : {
    "name" : "FILE"
  },
  "REST-ACCESS-FILE" : {
    "name" : "REST-ACCESS-FILE"
  }
}
      Copy to Clipboard Toggle word wrap

    • List all logger configurations defined on the server: 

      logging list-loggers
      Copy to Clipboard Toggle word wrap

      The command provides a JSON response such as the following: 

      [ {
  "name" : "",
  "level" : "INFO",
  "appenders" : [ "STDOUT", "FILE" ]
}, {
  "name" : "org.infinispan.HOTROD_ACCESS_LOG",
  "level" : "INFO",
  "appenders" : [ "HR-ACCESS-FILE" ]
}, {
  "name" : "com.arjuna",
  "level" : "WARN",
  "appenders" : [ ]
}, {
  "name" : "org.infinispan.REST_ACCESS_LOG",
  "level" : "INFO",
  "appenders" : [ "REST-ACCESS-FILE" ]
} ]
      Copy to Clipboard Toggle word wrap

    • Add and modify logger configurations with the set subcommand

      For example, the following command sets the logging level for the org.infinispan package to DEBUG: 

      logging set --level=DEBUG org.infinispan
      Copy to Clipboard Toggle word wrap

    • Remove existing logger configurations with the remove subcommand.

      For example, the following command removes the org.infinispan logger configuration, which means the root configuration is used instead: 

      logging remove org.infinispan
      Copy to Clipboard Toggle word wrap

16.3. Gathering resource statistics from the CLI
Copy link

You can inspect server-collected statistics for some Data Grid Server resources with the stats command.

Use the stats command either from the context of a resource that provides statistics (containers, caches) or with a path to such a resource: 

stats
Copy to Clipboard Toggle word wrap 
{
  "statistics_enabled" : true,
  "number_of_entries" : 0,
  "hit_ratio" : 0.0,
  "read_write_ratio" : 0.0,
  "time_since_start" : 0,
  "time_since_reset" : 49,
  "current_number_of_entries" : 0,
  "current_number_of_entries_in_memory" : 0,
  "total_number_of_entries" : 0,
  "off_heap_memory_used" : 0,
  "data_memory_used" : 0,
  "stores" : 0,
  "retrievals" : 0,
  "hits" : 0,
  "misses" : 0,
  "remove_hits" : 0,
  "remove_misses" : 0,
  "evictions" : 0,
  "average_read_time" : 0,
  "average_read_time_nanos" : 0,
  "average_write_time" : 0,
  "average_write_time_nanos" : 0,
  "average_remove_time" : 0,
  "average_remove_time_nanos" : 0,
  "required_minimum_number_of_nodes" : -1
}
Copy to Clipboard Toggle word wrap 
stats /containers/default/caches/mycache
Copy to Clipboard Toggle word wrap 
{
  "time_since_start" : -1,
  "time_since_reset" : -1,
  "current_number_of_entries" : -1,
  "current_number_of_entries_in_memory" : -1,
  "total_number_of_entries" : -1,
  "off_heap_memory_used" : -1,
  "data_memory_used" : -1,
  "stores" : -1,
  "retrievals" : -1,
  "hits" : -1,
  "misses" : -1,
  "remove_hits" : -1,
  "remove_misses" : -1,
  "evictions" : -1,
  "average_read_time" : -1,
  "average_read_time_nanos" : -1,
  "average_write_time" : -1,
  "average_write_time_nanos" : -1,
  "average_remove_time" : -1,
  "average_remove_time_nanos" : -1,
  "required_minimum_number_of_nodes" : -1
}
Copy to Clipboard Toggle word wrap

16.4. Accessing cluster health via REST
Copy link

Get Data Grid cluster health via the REST API.

Procedure

  • Invoke a GET request to retrieve cluster health. 

    GET /rest/v2/cache-managers/{cacheManagerName}/health
    Copy to Clipboard Toggle word wrap

Data Grid responds with a JSON document such as the following: 

{
    "cluster_health":{
        "cluster_name":"ISPN",
        "health_status":"HEALTHY",
        "number_of_nodes":2,
        "node_names":[
            "NodeA-36229",
            "NodeB-28703"
        ]
    },
    "cache_health":[
        {
            "status":"HEALTHY",
            "cache_name":"___protobuf_metadata"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache2"
        },
        {
            "status":"HEALTHY",
            "cache_name":"mycache"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache1"
        }
    ]
}
Copy to Clipboard Toggle word wrap
Tip

Get cache manager status as follows: 

GET /rest/v2/cache-managers/{cacheManagerName}/health/status
Copy to Clipboard Toggle word wrap

Reference

See the REST v2 (version 2) API documentation for more information.

16.5. Accessing cluster health via JMX
Copy link

Retrieve Data Grid cluster health statistics via JMX.

Procedure

  1. Connect to Data Grid server using any JMX capable tool such as JConsole and navigate to the following object: 

    org.infinispan:type=CacheManager,name="default",component=CacheContainerHealth
    Copy to Clipboard Toggle word wrap
  2. Select available MBeans to retrieve cluster health statistics.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat