Chapter 5. Configuring Streams for Apache Kafka
Use the Kafka configuration properties files to configure Streams for Apache Kafka.
The properties file is in Java format, with each property on a separate line in the following format:
<option> = <value>
Lines starting with #
or !
are treated as comments and are ignored by Streams for Apache Kafka components.
# This is a comment
Values can be split into multiple lines by using \
directly before the newline/carriage return.
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="bob" \ password="bobs-password";
After saving the changes in the properties file, you need to restart the Kafka node. In a multi-node environment, repeat the process on each node in the cluster.
5.1. Using standard Kafka configuration properties
Use standard Kafka configuration properties to configure Kafka components.
The properties provide options to control and tune the configuration of the following Kafka components:
- Brokers
- Topics
- Producer, consumer, and management clients
- Kafka Connect
- Kafka Streams
Broker and client parameters include options to configure authorization, authentication and encryption.
For further information on Kafka configuration properties and how to use the properties to tune your deployment, see the following guides:
5.2. Loading configuration values from environment variables
Use the Environment Variables Configuration Provider plugin to load configuration data from environment variables. You can use the Environment Variables Configuration Provider, for example, to load certificates or JAAS configuration from environment variables.
You can use the provider to load configuration data for all Kafka components, including producers and consumers. Use the provider, for example, to provide the credentials for Kafka Connect connector configuration.
Prerequisites
- Streams for Apache Kafka is installed on each host, and the configuration files are available.
The Environment Variables Configuration Provider JAR file.
The JAR file is available from the Streams for Apache Kafka archive.
Procedure
-
Add the Environment Variables Configuration Provider JAR file to the Kafka
libs
directory. Initialize the Environment Variables Configuration Provider in the configuration properties file of the Kafka component. For example, to initialize the provider for Kafka, add the configuration to the
server.properties
file.Configuration to enable the Environment Variables Configuration Provider
config.providers.env.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider
Add configuration to the properties file to load data from environment variables.
Configuration to load data from an environment variable
option=${env:<MY_ENV_VAR_NAME>}
Use capitalized or upper-case environment variable naming conventions, such as
MY_ENV_VAR_NAME
.- Save the changes.
Restart the Kafka component.
For information on restarting brokers in a multi-node cluster, see Section 3.6, “Performing a graceful rolling restart of Kafka brokers”.
5.3. Configuring Kafka
Kafka uses properties files to store static configuration. The recommended location for the configuration files is /opt/kafka/config/kraft/
. The configuration files have to be readable by the kafka
user.
Streams for Apache Kafka ships example configuration files that highlight various basic and advanced features of the product. They can be found under config/kraft/
in the Streams for Apache Kafka installation directory as follows:
-
(default)
config/kraft/server.properties
for nodes running in combined mode -
config/kraft/broker.properties
for nodes running as brokers -
config/kraft/controller.properties
for nodes running as controllers
This chapter explains the most important configuration options.
5.3.1. Listeners
Listeners are used to connect to Kafka brokers. Each Kafka broker can be configured to use multiple listeners. Each listener requires a different configuration so it can listen on a different port or network interface.
To configure listeners, edit the listeners
property in the Kafka configuration properties file. Add listeners to the listeners
property as a comma-separated list. Configure each property as follows:
<listener_name>://<hostname>:<port>
If <hostname>
is empty, Kafka uses the java.net.InetAddress.getCanonicalHostName()
class as the hostname.
Example configuration for multiple listeners
listeners=internal-1://:9092,internal-2://:9093,replication://:9094
When a Kafka client wants to connect to a Kafka cluster, it first connects to the bootstrap server, which is one of the cluster nodes. The bootstrap server provides the client with a list of all the brokers in the cluster, and the client connects to each one individually. The list of brokers is based on the configured listeners
.
Advertised listeners
Optionally, you can use the advertised.listeners
property to provide the client with a different set of listener addresses than those given in the listeners
property. This is useful if additional network infrastructure, such as a proxy, is between the client and the broker, or an external DNS name is being used instead of an IP address.
The advertised.listeners
property is formatted in the same way as the listeners
property.
Example configuration for advertised listeners
listeners=internal-1://:9092,internal-2://:9093 advertised.listeners=internal-1://my-broker-1.my-domain.com:1234,internal-2://my-broker-1.my-domain.com:1235
The names of the advertised listeners must match those listed in the listeners
property.
Inter-broker listeners
Inter-broker listeners are used for communication between Kafka brokers. Inter-broker communication is required for:
- Coordinating workloads between different brokers
- Replicating messages between partitions stored on different brokers
The inter-broker listener can be assigned to a port of your choice. When multiple listeners are configured, you can define the name of the inter-broker listener in the inter.broker.listener.name
property of your broker configuration.
Here, the inter-broker listener is named as REPLICATION
:
listeners=REPLICATION://0.0.0.0:9091 inter.broker.listener.name=REPLICATION
Controller listeners
Controller configuration is used to connect and communicate with the controller that coordinates the cluster and manages the metadata used to track the status of brokers and partitions.
By default, communication between the controllers and brokers uses a dedicated controller listener. Controllers are responsible for coordinating administrative tasks, such as partition leadership changes, so one or more of these listeners is required.
Specify listeners to use for controllers using the controller.listener.names
property. You can specify a quorum of controller voters using the controller.quorum.voters
property. The quorum enables a leader-follower structure for administrative tasks, with the leader actively managing operations and followers as hot standbys, ensuring metadata consistency in memory and facilitating failover.
listeners=CONTROLLER://0.0.0.0:9090 controller.listener.names=CONTROLLER controller.quorum.voters=1@localhost:9090
The format for the controller voters is <cluster_id>@<hostname>:<port>
.
5.3.2. Commit logs
Apache Kafka stores all records it receives from producers in commit logs. The commit logs contain the actual data, in the form of records, that Kafka needs to deliver. Note that these records differ from application log files, which detail the broker’s activities.
Log directories
You can configure log directories using the log.dirs
property file to store commit logs in one or multiple log directories. It should be set to /var/lib/kafka
directory created during installation:
log.dirs=/var/lib/kafka
For performance reasons, you can configure log.dirs to multiple directories and place each of them on a different physical device to improve disk I/O performance. For example:
log.dirs=/var/lib/kafka1,/var/lib/kafka2,/var/lib/kafka3
5.3.3. Node ID
Node ID is a unique identifier for each node (broker or controller) in the cluster. You can assign an integer greater than or equal to 0 as node ID. The node ID is used to identify the nodes after restarts or crashes and it is therefore important that the ID is stable and does not change over time.
The node ID is configured in the Kafka configuration properties file:
node.id=1