Chapter 3. Kafka Bridge configuration
Configure a deployment of the Kafka Bridge using configuration properties. Configure Kafka and specify the HTTP connection details needed to be able to interact with Kafka. You can also use configuration properties to enable and use distributed tracing with the Kafka Bridge. Distributed tracing allows you to track the progress of transactions between applications in a distributed system.
Use the KafkaBridge resource to configure properties when you are running the Kafka Bridge on OpenShift.
3.1. Configuring Kafka Bridge properties Copy linkLink copied to clipboard!
This procedure describes how to configure the Kafka and HTTP connection properties used by the Kafka Bridge.
You configure the Kafka Bridge, as any other Kafka client, using appropriate prefixes for Kafka-related properties.
-
kafka.for general configuration that applies to producers and consumers, such as server connection and security. -
kafka.consumer.for consumer-specific configuration passed only to the consumer. -
kafka.producer.for producer-specific configuration passed only to the producer.
As well as enabling HTTP access to a Kafka cluster, HTTP properties provide the capability to enable and define access control for the Kafka Bridge through Cross-Origin Resource Sharing (CORS). CORS is a HTTP mechanism that allows browser access to selected resources from more than one origin. To configure CORS, you define a list of allowed resource origins and HTTP methods to access them. Additional HTTP headers in requests describe the CORS origins that are permitted access to the Kafka cluster.
Prerequisites
Procedure
Edit the
application.propertiesfile provided with the Kafka Bridge installation archive.Use the properties file to specify Kafka and HTTP-related properties.
Configure standard Kafka-related properties, including properties specific to the Kafka consumers and producers.
Use:
-
kafka.bootstrap.serversto define the host/port connections to the Kafka cluster -
kafka.producer.acksto provide acknowledgments to the HTTP client kafka.consumer.auto.offset.resetto determine how to manage reset of the offset in KafkaFor more information on configuration of Kafka properties, see the Apache Kafka website
-
Configure HTTP-related properties to enable HTTP access to the Kafka cluster.
For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- Save the configuration file.
3.2. Configuring distributed tracing Copy linkLink copied to clipboard!
Enable distributed tracing to trace messages consumed and produced by the Kafka Bridge, and HTTP requests from client applications.
Properties to enable tracing are present in the application.properties file. To enable distributed tracing, do the following:
-
Set the
bridge.tracingproperty value to enable the tracing you want to use. Possible values arejaegerandopentelemetry. - Set environment variables for tracing.
With the default configuration, OpenTelemetry tracing uses OTLP as the exporter protocol. By configuring the OTLP endpoint, you can still use a Jaeger backend instance to get traces.
Jaeger has supported the OTLP protocol since version 1.35. Older Jaeger versions cannot get traces using the OTLP protocol.
OpenTelemetry and OpenTracing are API specifications for collecting tracing data as spans of metrics data. Spans represent a specific operation. A trace is a collection of one or more spans.
Traces are generated when the Kafka Bridge does the following:
- Sends messages from Kafka to consumer HTTP clients
- Receives messages from producer HTTP clients to send to Kafka
Jaeger implements the required APIs and presents visualizations of the trace data in its user interface for analysis.
To have end-to-end tracing, you must configure tracing in your HTTP clients.
The OpenTracing project is now archived, so AMQ Streams has deprecated support for OpenTracing. If possible, we will maintain the support for bridge.tracing=jaeger tracing until June 2023 and remove it afterwards. Please migrate to OpenTelemetry as soon as possible.
Prerequisites
Procedure
Edit the
application.propertiesfile provided with the Kafka Bridge installation archive.Use the
bridge.tracingproperty to enable the tracing you want to use.Example configuration to enable OpenTelemetry
#bridge.tracing=jaeger bridge.tracing=opentelemetry
#bridge.tracing=jaeger1 bridge.tracing=opentelemetry2 Copy to Clipboard Copied! Toggle word wrap Toggle overflow With tracing enabled, you initialize tracing when you run the Kafka Bridge script.
- Save the configuration file.
Set the environment variables for tracing.
Environment variables for OpenTelemetry
OTEL_SERVICE_NAME=my-tracing-service OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=my-tracing-service1 OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:43172 Copy to Clipboard Copied! Toggle word wrap Toggle overflow Environment variables for OpenTracing
JAEGER_SERVICE_NAME=my-jaeger-service JAEGER_AGENT_HOST=localhost JAEGER_AGENT_PORT=6831
JAEGER_SERVICE_NAME=my-jaeger-service1 JAEGER_AGENT_HOST=localhost2 JAEGER_AGENT_PORT=68313 Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the Kafka Bridge script with the property enabled for tracing:
Running the Kafka Bridge with OpenTelemetry enabled
./bin/kafka_bridge_run.sh --config-file=<path>/application.properties
./bin/kafka_bridge_run.sh --config-file=<path>/application.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow The internal consumers and producers of the Kafka Bridge are now enabled for tracing.
3.2.1. Specifying tracing systems with OpenTelemetry Copy linkLink copied to clipboard!
Instead of the default OTLP tracing system, you can specify other tracing systems that are supported by OpenTelemetry.
If you want to use another tracing system with OpenTelemetry, do the following:
- Add the library of the tracing system to the Kafka classpath.
Add the name of the tracing system as an additional exporter environment variable.
Additional environment variable when not using OTLP
OTEL_SERVICE_NAME=my-tracing-service OTEL_TRACES_EXPORTER=zipkin OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://localhost:9411/api/v2/spans
OTEL_SERVICE_NAME=my-tracing-service OTEL_TRACES_EXPORTER=zipkin1 OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://localhost:9411/api/v2/spans2 Copy to Clipboard Copied! Toggle word wrap Toggle overflow