Este conteúdo não está disponível no idioma selecionado.

Chapter 10. Distributed tracing

Distributed tracing allows you to track the progress of transactions between applications in a distributed system. In a microservices architecture, tracing tracks the progress of transactions between services. Trace data is useful for monitoring application performance and investigating issues with target systems and end-user applications.

In AMQ Streams, tracing facilitates the end-to-end tracking of messages: from source systems to Kafka, and then from Kafka to target systems and applications. It complements the metrics that are available to view in Grafana dashboards, as well as the component loggers.

How AMQ Streams supports tracing

Support for tracing is built in to the following components:

Kafka Connect (including Kafka Connect with Source2Image support)
MirrorMaker
MirrorMaker 2.0
AMQ Streams Kafka Bridge

You enable and configure tracing for these components using template configuration properties in their custom resources.

To enable tracing in Kafka producers, consumers, and Kafka Streams API applications, you instrument application code using the OpenTracing Apache Kafka Client Instrumentation library (included with AMQ Streams). When instrumented, clients generate trace data; for example, when producing messages or writing offsets to the log.

Traces are sampled according to a sampling strategy and then visualized in the Jaeger user interface.

Note

Tracing is not supported for Kafka brokers.

Setting up tracing for applications and systems beyond AMQ Streams is outside the scope of this chapter. To learn more about this subject, search for "inject and extract" in the OpenTracing documentation.

Outline of procedures

To set up tracing for AMQ Streams, follow these procedures in order:

Set up tracing for clients:
- Initialize a Jaeger tracer for Kafka clients
Instrument clients with tracers:
- Instrument producers and consumers for tracing
- Instrument Kafka Streams applications for tracing
Set up tracing for MirrorMaker, Kafka Connect, and the Kafka Bridge

Prerequisites

The Jaeger backend components are deployed to your OpenShift cluster. For deployment instructions, see the Jaeger deployment documentation.

10.1. Overview of OpenTracing and Jaeger

AMQ Streams uses the OpenTracing and Jaeger projects.

OpenTracing is an API specification that is independent from the tracing or monitoring system.

The OpenTracing APIs are used to instrument application code
Instrumented applications generate traces for individual transactions across the distributed system
Traces are composed of spans that define specific units of work over time

Jaeger is a tracing system for microservices-based distributed systems.

Jaeger implements the OpenTracing APIs and provides client libraries for instrumentation
The Jaeger user interface allows you to query, filter, and analyze trace data

The Jaeger user interface showing a simple query

Additional resources

10.2. Setting up tracing for Kafka clients

Initialize a Jaeger tracer to instrument your client applications for distributed tracing.

10.2.1. Initializing a Jaeger tracer for Kafka clients

Configure and initialize a Jaeger tracer using a set of tracing environment variables.

Procedure

In each client application:

Add Maven dependencies for Jaeger to the pom.xml file for the client application:

<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-client</artifactId>
    <version>1.1.0.redhat-00002</version>
</dependency>

Define the configuration of the Jaeger tracer using the tracing environment variables.
Create the Jaeger tracer from the environment variables that you defined in step two:
```
Tracer tracer = Configuration.fromEnv().getTracer();
```
Note
For alternative ways to initialize a Jaeger tracer, see the Java OpenTracing library documentation.
Register the Jaeger tracer as a global tracer:
```
GlobalTracer.register(tracer);
```

A Jaeger tracer is now initialized for the client application to use.

10.2.2. Environment variables for tracing

Use these environment variables when configuring a Jaeger tracer for Kafka clients.

Note

The tracing environment variables are part of the Jaeger project and are subject to change. For the latest environment variables, see the Jaeger documentation.

Property	Required	Description
`JAEGER_SERVICE_NAME`	Yes	The name of the Jaeger tracer service.
`JAEGER_AGENT_HOST`	No	The hostname for communicating with the `jaeger-agent` through the User Datagram Protocol (UDP).
`JAEGER_AGENT_PORT`	No	The port used for communicating with the `jaeger-agent` through UDP.
`JAEGER_ENDPOINT`	No	The `traces` endpoint. Only define this variable if the client application will bypass the `jaeger-agent` and connect directly to the `jaeger-collector`.
`JAEGER_AUTH_TOKEN`	No	The authentication token to send to the endpoint as a bearer token.
`JAEGER_USER`	No	The username to send to the endpoint if using basic authentication.
`JAEGER_PASSWORD`	No	The password to send to the endpoint if using basic authentication.
`JAEGER_PROPAGATION`	No	A comma-separated list of formats to use for propagating the trace context. Defaults to the standard Jaeger format. Valid values are `jaeger`, `b3`, and `w3c`.
`JAEGER_REPORTER_LOG_SPANS`	No	Indicates whether the reporter should also log the spans.
`JAEGER_REPORTER_MAX_QUEUE_SIZE`	No	The reporter’s maximum queue size.
`JAEGER_REPORTER_FLUSH_INTERVAL`	No	The reporter’s flush interval, in ms. Defines how frequently the Jaeger reporter flushes span batches.
`JAEGER_SAMPLER_TYPE`	No	The sampling strategy to use for client traces: Constant Probabilistic Rate Limiting Remote (the default) To sample all traces, use the Constant sampling strategy with a parameter of 1. For more information, see the Jaeger documentation.
`JAEGER_SAMPLER_PARAM`	No	The sampler parameter (number).
`JAEGER_SAMPLER_MANAGER_HOST_PORT`	No	The hostname and port to use if a Remote sampling strategy is selected.
`JAEGER_TAGS`	No	A comma-separated list of tracer-level tags that are added to all reported spans. The value can also refer to an environment variable using the format `${envVarName:default}`. `:default` is optional and identifies a value to use if the environment variable cannot be found.

Additional resources

Section 10.2.1, “Initializing a Jaeger tracer for Kafka clients”

10.3. Instrumenting Kafka clients with tracers

Instrument Kafka producer and consumer clients, and Kafka Streams API applications for distributed tracing.

10.3.1. Instrumenting producers and consumers for tracing

Use a Decorator pattern or Interceptors to instrument your Java producer and consumer application code for tracing.

Procedure

In the application code of each producer and consumer application:

Add the Maven dependency for OpenTracing to the producer or consumer’s pom.xml file.

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-kafka-client</artifactId>
    <version>0.1.15.redhat-00001</version>
</dependency>

Instrument your client application code using either a Decorator pattern or Interceptors.

To use a Decorator pattern:

// Create an instance of the KafkaProducer:
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

// Create an instance of the TracingKafkaProducer:
TracingKafkaProducer<Integer, String> tracingProducer = new TracingKafkaProducer<>(producer,
        tracer);

// Send:
tracingProducer.send(...);

// Create an instance of the KafkaConsumer:
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);

// Create an instance of the TracingKafkaConsumer:
TracingKafkaConsumer<Integer, String> tracingConsumer = new TracingKafkaConsumer<>(consumer,
        tracer);

// Subscribe:
tracingConsumer.subscribe(Collections.singletonList("messages"));

// Get messages:
ConsumerRecords<Integer, String> records = tracingConsumer.poll(1000);

// Retrieve SpanContext from polled record (consumer side):
ConsumerRecord<Integer, String> record = ...
SpanContext spanContext = TracingKafkaUtils.extractSpanContext(record.headers(), tracer);

To use Interceptors:

// Register the tracer with GlobalTracer:
GlobalTracer.register(tracer);

// Add the TracingProducerInterceptor to the sender properties:
senderProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
          TracingProducerInterceptor.class.getName());

// Create an instance of the KafkaProducer:
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

// Send:
producer.send(...);

// Add the TracingConsumerInterceptor to the consumer properties:
consumerProps.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
          TracingConsumerInterceptor.class.getName());

// Create an instance of the KafkaConsumer:
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);

// Subscribe:
consumer.subscribe(Collections.singletonList("messages"));

// Get messages:
ConsumerRecords<Integer, String> records = consumer.poll(1000);

// Retrieve the SpanContext from a polled message (consumer side):
ConsumerRecord<Integer, String> record = ...
SpanContext spanContext = TracingKafkaUtils.extractSpanContext(record.headers(), tracer);

10.3.1.1. Custom span names in a Decorator pattern

A span is a logical unit of work in Jaeger, with an operation name, start time, and duration.

To use a Decorator pattern to instrument your producer and consumer applications, define custom span names by passing a BiFunction object as an additional argument when creating the TracingKafkaProducer and TracingKafkaConsumer objects. The OpenTracing Apache Kafka Client Instrumentation library includes several built-in span names.

Example: Using custom span names to instrument client application code in a Decorator pattern

// Create a BiFunction for the KafkaProducer that operates on (String operationName, ProducerRecord consumerRecord) and returns a String to be used as the name:

BiFunction<String, ProducerRecord, String> producerSpanNameProvider =
    (operationName, producerRecord) -> "CUSTOM_PRODUCER_NAME";

// Create an instance of the KafkaProducer:
KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps);

// Create an instance of the TracingKafkaProducer
TracingKafkaProducer<Integer, String> tracingProducer = new TracingKafkaProducer<>(producer,
        tracer,
        producerSpanNameProvider);

// Spans created by the tracingProducer will now have "CUSTOM_PRODUCER_NAME" as the span name.

// Create a BiFunction for the KafkaConsumer that operates on (String operationName, ConsumerRecord consumerRecord) and returns a String to be used as the name:

BiFunction<String, ConsumerRecord, String> consumerSpanNameProvider =
    (operationName, consumerRecord) -> operationName.toUpperCase();

// Create an instance of the KafkaConsumer:
KafkaConsumer<Integer, String> consumer = new KafkaConsumer<>(consumerProps);

// Create an instance of the TracingKafkaConsumer, passing in the consumerSpanNameProvider BiFunction:

TracingKafkaConsumer<Integer, String> tracingConsumer = new TracingKafkaConsumer<>(consumer,
        tracer,
        consumerSpanNameProvider);

// Spans created by the tracingConsumer will have the operation name as the span name, in upper-case.
// "receive" -> "RECEIVE"

10.3.1.2. Built-in span names

When defining custom span names, you can use the following BiFunctions in the ClientSpanNameProvider class. If no spanNameProvider is specified, CONSUMER_OPERATION_NAME and PRODUCER_OPERATION_NAME are used.

BiFunction	Description
`CONSUMER_OPERATION_NAME, PRODUCER_OPERATION_NAME`	Returns the `operationName` as the span name: "receive" for consumers and "send" for producers.
`CONSUMER_PREFIXED_OPERATION_NAME(String prefix), PRODUCER_PREFIXED_OPERATION_NAME(String prefix)`	Returns a String concatenation of `prefix` and `operationName`.
`CONSUMER_TOPIC, PRODUCER_TOPIC`	Returns the name of the topic that the message was sent to or retrieved from in the format `(record.topic())`.
`PREFIXED_CONSUMER_TOPIC(String prefix), PREFIXED_PRODUCER_TOPIC(String prefix)`	Returns a String concatenation of `prefix` and the topic name in the format `(record.topic())`.
`CONSUMER_OPERATION_NAME_TOPIC, PRODUCER_OPERATION_NAME_TOPIC`	Returns the operation name and the topic name: `"operationName - record.topic()"`.
`CONSUMER_PREFIXED_OPERATION_NAME_TOPIC(String prefix), PRODUCER_PREFIXED_OPERATION_NAME_TOPIC(String prefix)`	Returns a String concatenation of `prefix` and `"operationName - record.topic()"`.

10.3.2. Instrumenting Kafka Streams applications for tracing

This section describes how to instrument Kafka Streams API applications for distributed tracing.

Procedure

In each Kafka Streams API application:

Add the opentracing-kafka-streams dependency to the pom.xml file for your Kafka Streams API application:

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-kafka-streams</artifactId>
    <version>0.1.15.redhat-00001</version>
</dependency>

Create an instance of the TracingKafkaClientSupplier supplier interface:

KafkaClientSupplier supplier = new TracingKafkaClientSupplier(tracer);

Provide the supplier interface to KafkaStreams:

KafkaStreams streams = new KafkaStreams(builder.build(), new StreamsConfig(config), supplier);
streams.start();

10.4. Setting up tracing for MirrorMaker, Kafka Connect, and the Kafka Bridge

Distributed tracing is supported for MirrorMaker, MirrorMaker 2.0, Kafka Connect (including Kafka Connect with Source2Image support), and the AMQ Streams Kafka Bridge.

Tracing in MirrorMaker and MirrorMaker 2.0

For MirrorMaker and MirrorMaker 2.0, messages are traced from the source cluster to the target cluster. The trace data records messages entering and leaving the MirrorMaker or MirrorMaker 2.0 component.

Tracing in Kafka Connect

Only messages produced and consumed by Kafka Connect itself are traced. To trace messages sent between Kafka Connect and external systems, you must configure tracing in the connectors for those systems. For more information, see Section 2.2.1, “Configuring Kafka Connect”.

Tracing in the Kafka Bridge

Messages produced and consumed by the Kafka Bridge are traced. Incoming HTTP requests from client applications to send and receive messages through the Kafka Bridge are also traced. To have end-to-end tracing, you must configure tracing in your HTTP clients.

10.4.1. Enabling tracing in MirrorMaker, Kafka Connect, and Kafka Bridge resources

Update the configuration of KafkaMirrorMaker, KafkaMirrorMaker2, KafkaConnect, KafkaConnectS2I, and KafkaBridge custom resources to specify and configure a Jaeger tracer service for each resource. Updating a tracing-enabled resource in your OpenShift cluster triggers two events:

Interceptor classes are updated in the integrated consumers and producers in MirrorMaker, MirrorMaker 2.0, Kafka Connect, or the AMQ Streams Kafka Bridge.
For MirrorMaker, MirrorMaker 2.0, and Kafka Connect, the tracing agent initializes a Jaeger tracer based on the tracing configuration defined in the resource.
For the Kafka Bridge, a Jaeger tracer based on the tracing configuration defined in the resource is initialized by the Kafka Bridge itself.

Procedure

Perform these steps for each KafkaMirrorMaker, KafkaMirrorMaker2, KafkaConnect, KafkaConnectS2I, and KafkaBridge resource.

In the spec.template property, configure the Jaeger tracer service. For example:

Jaeger tracer configuration for Kafka Connect

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  #...
  template:
    connectContainer: 1
      env:
        - name: JAEGER_SERVICE_NAME
          value: my-jaeger-service
        - name: JAEGER_AGENT_HOST
          value: jaeger-agent-name
        - name: JAEGER_AGENT_PORT
          value: "6831"
  tracing: 2
    type: jaeger
  #...

Jaeger tracer configuration for MirrorMaker

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker
metadata:
  name: my-mirror-maker
spec:
  #...
  template:
    mirrorMakerContainer:
      env:
        - name: JAEGER_SERVICE_NAME
          value: my-jaeger-service
        - name: JAEGER_AGENT_HOST
          value: jaeger-agent-name
        - name: JAEGER_AGENT_PORT
          value: "6831"
  tracing:
    type: jaeger
#...

Jaeger tracer configuration for MirrorMaker 2.0

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mm2-cluster
spec:
  #...
  template:
    connectContainer:
      env:
        - name: JAEGER_SERVICE_NAME
          value: my-jaeger-service
        - name: JAEGER_AGENT_HOST
          value: jaeger-agent-name
        - name: JAEGER_AGENT_PORT
          value: "6831"
  tracing:
    type: jaeger
#...

Jaeger tracer configuration for the Kafka Bridge

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaBridge
metadata:
  name: my-bridge
spec:
  #...
  template:
    bridgeContainer:
      env:
        - name: JAEGER_SERVICE_NAME
          value: my-jaeger-service
        - name: JAEGER_AGENT_HOST
          value: jaeger-agent-name
        - name: JAEGER_AGENT_PORT
          value: "6831"
  tracing:
    type: jaeger
#...

1: Use the tracing environment variables as template configuration properties.
2: Set the spec.tracing.type property to jaeger.

Create or update the resource:
```
oc apply -f your-file
```

Additional resources

Este conteúdo não está disponível no idioma selecionado.

Chapter 10. Distributed tracing

How AMQ Streams supports tracing

Outline of procedures

10.1. Overview of OpenTracing and Jaeger

10.2. Setting up tracing for Kafka clients

10.2.1. Initializing a Jaeger tracer for Kafka clients

10.2.2. Environment variables for tracing

10.3. Instrumenting Kafka clients with tracers

10.3.1. Instrumenting producers and consumers for tracing

10.3.1.1. Custom span names in a Decorator pattern

10.3.1.2. Built-in span names

10.3.2. Instrumenting Kafka Streams applications for tracing

10.4. Setting up tracing for MirrorMaker, Kafka Connect, and the Kafka Bridge

10.4.1. Enabling tracing in MirrorMaker, Kafka Connect, and Kafka Bridge resources

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Tornando o open source mais inclusivo

Sobre a Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links