Chapter 10. Enabling and configuring Data Grid OpenTelemetry tracing
Data Grid generates tracing spans compatible with the OpenTelemetry standard, allowing you to export, visualize, and analyze tracing data related to the most important cache operations.
10.1. Configuring Data Grid tracing
Configure OpenTelemetry tracing, to enable monitoring and tracing of cache operations.
Procedure
- Open your Data Grid configuration for editing.
-
Add the
tracingelement or object to the cache container. -
Define the endpoint URL of the OpenTelemetry collector with the
collector_endpointattribute or field. It is mandatory to enable tracing.4318is typically thehttp/protobufOTLP standard port. -
Enable or disable tracing globally with the
enableattribute or field. -
Enable or disable security event tracing with the
securityattribute or field. -
Optionally change the tracing exporter protocol changing the
exporter_protocolattribute or field. By default, it isotlp(OpenTelemetry protocol). -
Optionally change the tracing service name associated with the generated tracing span changing the
service_nameattribute or field. By default, it isinfinispan-server. - Save and close your client configuration.
Next steps
To apply any global tracing configuration changes, stop the server and repeat the procedure.
Tracing configuration
Data Grid applies the tracing configuration globally to all caches.
XML
<infinispan>
<cache-container statistics="true">
<tracing collector-endpoint="http://localhost:4318"
enabled="true"
exporter-protocol="OTLP"
service-name="infinispan-server"
security="false" />
</cache-container>
</infinispan>
JSON
{
"infinispan" : {
"cache-container" : {
"statistics" : true,
"tracing" : {
"collector-endpoint" : "http://localhost:4318",
"enabled" : true,
"exporter-protocol" : "OTLP",
"service-name" : "infinispan-server",
"security" : false
}
}
}
}
YAML
infinispan:
cacheContainer:
statistics: true
tracing:
collector-endpoint: "http://localhost:4318"
enabled: true
exporter-protocol: "OTLP"
service-name: "infinispan-server"
security: false
10.1.1. Further Tracing Options
To configure further tracing options it is possible to pass system properties or to set environment variables to the Data Grid server at startup to configure directly the OpenTelemetry SDK Autoconfigure that is used by the Data Grid server to configure the OpenTelemetry tracing.
Procedure
Pass the system properties to Data Grid Server at startup.
Use
-D<property-name>=<property-value>arguments like in the following example:bin/server.sh -Dotel.exporter.otlp.timeout=10000
Additional resources
Tracing data format
The Data Grid Server, by default, exports tracing data using the OTLP http/protobuf protocol.
tracing.properties
otel.exporter.otlp.protocol = http/protobuf
To use a different protocol, you must copy JAR files or dependencies to the $ISPN_HOME/server/lib directory of your Data Grid Server installation.
10.2. Configure tracing at cache level
Once the tracing is configured at server level, it will be automatically enabled by default for all caches. A cache configuration level of tracing allows on the other hand to enable or disable it at cache level and at runtime.
Tracing categories
Several categories are potentially traced:
- Container. That are all the main cache operations, such as replace, put, clear, getForReplace, remove operations and size. With the exception of all the getting operation.
- Cluster. Operations that are replicated to another node in the same cluster.
- X-Site. Operations that are replicated to another external site.
- Persistence. All the operations involving persistence via a cache store and/or cache loader.
Each of category can be enabled/disabled at start time or runtime listing them in the categories list attribute. By default only the container category is enabled.
There is also the Security category, to trace security audit events. This category is configured globally, not only at cache level, since their events can have different scopes (cache, container or server), not only cache scope.
Enable/disable tracing for a given cache
XML
<replicated-cache>
<tracing enabled="true" categories="container cluster x-site persistence" />
</replicated-cache>
JSON
{
"distributed-cache": {
"tracing": {
"enabled" : true,
"categories": [
"container",
"cluster",
"x-site",
"persistence"
]
}
}
}
YAML
distributedCache:
tracing:
enabled: true
categories:
- "container"
- "cluster"
- "x-site"
- "persistence"
Enable/disable tracing at runtime
The cache-level tracing attribute enable is a mutable attribute, it means it can be changed at runtime without the need to restart the Infinispan cluster.
To change a mutable attribute both HotRod and REST APIs can be used.
HotRod
remoteCacheManager.administration() .updateConfigurationAttribute(CACHE_A, "tracing.enabled", "false");
REST
restClient.cache(CACHE_A)
.updateConfigurationAttribute("tracing.enabled", "false");
Additional resources
