Chapter 10. Observability in JBoss EAP
If you’re a developer or system administrator, observability is a set of practices and technologies you can use to determine, based on certain signals from your application, the location and source of a problem in your application. The most common signals are metrics, events, and tracing. JBoss EAP uses OpenTelemetry for observability.
10.1. OpenTelemetry in JBoss EAP Copy linkLink copied to clipboard!
OpenTelemetry is a set of tools, application programming interfaces (APIs), and software development kits (SDKs) you can use to instrument, generate, collect, and export telemetry data for your applications. Telemetry data includes metrics, logs, and traces. Analyzing an application’s telemetry data helps you to improve your application’s performance. JBoss EAP provides OpenTelemetry capability through the opentelemetry subsystem.
Red Hat JBoss Enterprise Application Platform 7.4 provides only OpenTelemetry tracing capabilities.
OpenTelemetry is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview .
10.2. OpenTelemetry configuration in JBoss EAP Copy linkLink copied to clipboard!
You configure a number of aspects of OpenTelemetry in JBoss EAP using the opentelemetry subsystem. These include exporter, span processor, and sampler.
- exporter
- To analyze and visualize traces and metrics, you export them to a collector such as Jaeger. You can configure JBoss EAP to use either Jaeger or any collector that supports the OpenTelemetry protocol (OTLP).
- span processor
- You can configure the span processor to export spans either as they are produced or in batches. You can also configure the number of traces to export.
- sampler
- You can configure the number of traces to record by configuring the sampler.
Example configuration
The following XML is an example of the full OpenTelemetry configuration, including default values. JBoss EAP does not persist the default values when you make changes, so your configuration might look different.
<subsystem xmlns="urn:wildfly:opentelemetry:1.0"
service-name="example">
<exporter
type="jaeger"
endpoint="http://localhost:14250"/>
<span-processor
type="batch"
batch-delay="4500"
max-queue-size="128"
max-export-batch-size="512"
export-timeout="45"/>
<sampler
type="on"/>
</subsystem>
You cannot use an OpenShift route object to connect with a Jaeger endpoint. Instead, use http://<ip_address>:<port> or http://<service_name>:<port>.
10.3. OpenTelemetry tracing in JBoss EAP Copy linkLink copied to clipboard!
JBoss EAP provides OpenTelemetry tracing to help you track the progress of user requests as they pass through different parts of your application. By analyzing traces, you can improve your application’s performance and debug availability issues.
OpenTelemetry tracing consists of the following components:
- Trace
- A collection of operations that a request goes through in an application.
- Span
- A single operation within a trace. It provides request, error, and duration (RED) metrics and contains a span context.
- Span context
- A set of unique identifiers that represents a request that the containing span is a part of.
JBoss EAP automatically traces REST calls to your Jakarta RESTful Web Services applications and container-managed Jakarta RESTful Web Services client invocations. JBoss EAP traces REST calls implicitly as follows:
For each incoming request:
- JBoss EAP extracts the span context from the request.
- JBoss EAP starts a new span, then closes it when the request is completed.
For each outgoing request:
- JBoss EAP injects span context into the request.
- JBoss EAP starts a new span, then closes it when the request is completed.
In addition to implicit tracing, you can create custom spans by injecting a Tracer instance into your application for granular tracing.
If you see duplicate traces exported for REST calls, disable the microprofile-opentracing-smallrye subsystem. For information about disabling the microprofile-opentracing-smallrye, see Removing the microprofile-opentracing-smallrye subsystem.
10.4. Enabling OpenTelemetry tracing in JBoss EAP Copy linkLink copied to clipboard!
To use OpenTelemetry tracing in JBoss EAP you must first enable the opentelemetry subsystem.
Prerequisites
- You have installed JBoss EAP XP.
Procedure
Add the OpenTelemetry extension using the management CLI.
/extension=org.wildfly.extension.opentelemetry:addEnable the
opentelemetrysubsystem using the management CLI./subsystem=opentelemetry:addReload JBoss EAP.
reload
10.5. Configuring the opentelemetry subsystem Copy linkLink copied to clipboard!
You can configure the opentelemetry subsystem to set different aspects of tracing. Configure these based on the collector you use for observing the traces.
Prerequisites
-
You have enabled the
opentelemetrysubsystem. For more information, see Enabling OpenTelemetry tracing in JBoss EAP.
Procedure
Set the exporter type for the traces.
Syntax
/subsystem=opentelemetry:write-attribute(name=exporter-type, value=<exporter_type>)Example
/subsystem=opentelemetry:write-attribute(name=exporter-type, value=jaeger)Set the endpoint at which to export the traces.
Syntax
/subsystem=opentelemetry:write-attribute(name=endpoint, value=<URL:port>)Example
/subsystem=opentelemetry:write-attribute(name=endpoint, value=http:localhost:14250)Set the service name under which the traces are exported.
Syntax
/subsystem=opentelemetry:write-attribute(name=service-name, value=<service_name>)Example
/subsystem=opentelemetry:write-attribute(name=service-name, value=exampleOpenTelemetryService)
10.6. Using Jaeger to observe the OpenTelemetry traces for an application Copy linkLink copied to clipboard!
JBoss EAP automatically and implicitly traces REST calls to Jakarta RESTful Web Services applications. You do not need to add any configuration to your Jakarta RESTful Web Services application or configure the opentelemetry subsystem. The following procedure demonstrates how to observe traces for the helloworld-rs quickstart in the Jaeger console.
Prerequisites
- You have installed Docker. For more information, see Get Docker.
-
You have downloaded the
helloworld-rsquickstart. The quickstart is available at helloworld-rs. -
You have configured the the
opentelemetrysubsystem. For more information, see Configuring theopentelemetrysubsystem.
Procedure
Start the Jaeger console using its Docker image.
$ docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -p 5775:5775/udp \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 14268:14268 \ -p 14250:14250 \ -p 9411:9411 \ jaegertracing/all-in-one:1.29Use Maven to deploy the
helloworld-rsquickstart from its root directory.$ mvn clean install wildfly:deploy-
In a web browser, access the quickstart at
http://localhost:8080/helloworld-rs/, then click any link. -
In a web browser, open the Jaeger console at
http://localhost:16686/search.hello-world.rsis listed under Service. -
Select
hello-world.rsand click Find Traces. The details of the trace forhello-world.rsare listed.
10.7. OpenTelemetry tracing application development Copy linkLink copied to clipboard!
Although JBoss EAP automatically and implicitly traces REST calls to Jakarta RESTful Web Services applications, you can create custom spans from your application for granular tracing. A span is a single operation within a trace. You can create a span when, for example, a resource is defined, a method is called, and so on, in your application. You create custom traces in your application by injecting a Tracer instance.
10.7.1. Configuring a Maven project for OpenTelemetry tracing Copy linkLink copied to clipboard!
For creating an OpenTelemetry tracing application, create a Maven project with the required dependencies and directory structure.
Prerequisites
- You have installed Maven. For more information, see Downloading Apache Maven.
- You have configured your Maven repository for the latest release. For information about installing the latest Maven repository patch, see Maven and the JBoss EAP microprofile maven repository.
Procedure
In the CLI, use the
mvncommand to set up a Maven project. This command creates the directory structure for the project and thepom.xmlconfiguration file.Syntax
$ mvn archetype:generate \ -DgroupId=<group-to-which-your-application-belongs> \ -DartifactId=<name-of-your-application> \ -DarchetypeGroupId=org.apache.maven.archetypes \ -DarchetypeArtifactId=maven-archetype-webapp \ -DinteractiveMode=falseExample
$ mvn archetype:generate \ -DgroupId=com.example.opentelemetry \ -DartifactId=simple-tracing-example \ -DarchetypeGroupId=org.apache.maven.archetypes \ -DarchetypeArtifactId=maven-archetype-webapp \ -DinteractiveMode=falseNavigate to the application root directory.
Syntax
$ cd <name-of-your-application>Example
$ cd simple-tracing-exampleUpdate the generated
pom.xmlfile.Set the following properties:
<properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <failOnMissingWebXml>false</failOnMissingWebXml> <version.server.bom>4.0.0.GA</version.server.bom> <version.wildfly-jar.maven.plugin>6.1.1.Final</version.wildfly-jar.maven.plugin> </properties>Set the following dependencies:
<dependencies> <dependency> <groupId>jakarta.enterprise</groupId> <artifactId>jakarta.enterprise.cdi-api</artifactId> <version>2.0.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.jboss.spec.javax.ws.rs</groupId> <artifactId>jboss-jaxrs-api_2.1_spec</artifactId> <version>2.0.2.Final</version> <scope>provided</scope> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.5.0</version> <scope>provided</scope> </dependency> </dependencies>Set the following build configuration to use
mvn widlfy:deployto deploy the application:<build> <!-- Set the name of the archive --> <finalName>${project.artifactId}</finalName> <plugins> <!-- Allows to use mvn wildfly:deploy --> <plugin> <groupId>org.wildfly.plugins</groupId> <artifactId>wildfly-maven-plugin</artifactId> </plugin> </plugins> </build>
Verification
In the application root directory, enter the following command:
$ mvn installYou get an output similar to the following:
[INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.440 s [INFO] Finished at: 2021-12-27T14:45:12+05:30 [INFO] ------------------------------------------------------------------------
You can now create an OpenTelemetry tracing application.
10.7.2. Creating applications that create custom spans Copy linkLink copied to clipboard!
The following procedure demonstrates how to create an application that can create two custom spans like these:
-
prepare-hello- When the methodgetHello()in the application is called. -
process-hello- When the valuehellois assigned to a newStringobjecthello.
This procedure also demonstrates how to view these spans in a Jaeger console. <application_root> in the procedure denotes the directory that contains the pom.xml file, which contains the Maven configuration for your application.
Prerequisites
- You have installed Docker. For more information, see Get Docker.
- You have created a Maven project. For more information, see Configuring Maven project for OpenTelemetry tracing.
-
You have configured the the
opentelemetrysubsystem. For more information, see Configuring theopentelemetrysubsystem.
Procedure
In the <application_root>, create a directory to store the Java files.
Syntax
$ mkdir -p src/main/java/com/example/opentelemetryExample
$ mkdir -p src/main/java/com/example/opentelemetryNavigate to the new directory.
Syntax
$ cd src/main/java/com/example/opentelemetryExample
$ cd src/main/java/com/example/opentelemetryCreate a
JakartaRestApplication.javafile with the following content. ThisJakartaRestApplicationclass declares the application as a Jakarta RESTful Web Services application.package com.example.opentelemetry; import javax.ws.rs.ApplicationPath; import javax.ws.rs.core.Application; @ApplicationPath("/") public class JakartaRestApplication extends Application { }Create an
ExplicitlyTracedBean.javafile with the following content for the classExplicitlyTracedBean. This class creates custom spans by injecting aTracerclass.package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; @RequestScoped public class ExplicitlyTracedBean { @Inject private Tracer tracer;1 public String getHello() { Span prepareHelloSpan = tracer.spanBuilder("prepare-hello").startSpan();2 prepareHelloSpan.makeCurrent(); String hello = "hello"; Span processHelloSpan = tracer.spanBuilder("process-hello").startSpan();3 processHelloSpan.makeCurrent(); hello = hello.toUpperCase(); processHelloSpan.end(); prepareHelloSpan.end(); return hello; } }Create a
TracedResource.javafile with the following content forTracedResourceclass. This file injects theExplicitlyTracedBeanclass and declares two endpoints:tracedandcdi-trace.package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; @Path("/hello") @RequestScoped public class TracedResource { @Inject private ExplicitlyTracedBean tracedBean; @GET @Path("/traced") @Produces(MediaType.TEXT_PLAIN) public String hello() { return "hello"; } @GET @Path("/cdi-trace") @Produces(MediaType.TEXT_PLAIN) public String cdiHello() { return tracedBean.getHello(); } }Navigate to the application root directory.
Syntax
$ cd <path_to_application_root>/<application_root>Example
$ cd ~/applications/simple-tracing-exampleCompile and deploy the application with the following command:
$ mvn clean package wildfly:deployStart the Jaeger console.
$ docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -p 5775:5775/udp \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 14268:14268 \ -p 14250:14250 \ -p 9411:9411 \ jaegertracing/all-in-one:1.29-
In a browser, navigate to
\localhost:8080/simple-tracing-example/hello/cdi-trace. -
In a browser, open the Jaeger console at
http://localhost:16686/search. - In the Jaeger console, select JBoss EAP XP and click Find Traces.
- Click 3 Spans.
The Jaeger console displays the following traces:
|GET /hello/cdi-trace1 - | prepare-hello2 - | process-hello3 - 1
- This is the span for the automatic implicit trace.
- 2
- The custom span
prepare-helloindicates that the methodgetHello()was called. It is the child of span for the automatic implicit trace. - 3
- The custom span
process-helloindicates that the valuehellowas assigned to a newStringobjecthello. It is the child of theprepare-hellospan.
Whenever you access the application endpoint at http://localhost:16686/search, a new trace is created with all the child spans.