Chapter 3. Getting started

1. Add a new Kafka user and group:

   groupadd kafka
   useradd -g kafka kafka
   passwd kafka

   Copy to Clipboard Toggle word wrap

   This step creates the necessary user and group for managing Kafka.

2. Extract and move the contents of the amq-streams-<version>-kafka-bin.zip file and place it in the /opt/kafka directory:

   unzip amq-streams-<version>-kafka-bin.zip -d /opt
   mv /opt/kafka*redhat* /opt/kafka

   Copy to Clipboard Toggle word wrap

3. Change the ownership of the /opt/kafka directory to the Kafka user:

   chown -R kafka:kafka /opt/kafka

   Copy to Clipboard Toggle word wrap

4. Create directory the /var/lib/zookeeper directory for storing ZooKeeper data and set its ownership to the Kafka user:

   mkdir /var/lib/zookeeper
   chown -R kafka:kafka /var/lib/zookeeper

   Copy to Clipboard Toggle word wrap

5. Create the /var/lib/kafka directory for storing Kafka data and change its ownership to the Kafka user:

   mkdir /var/lib/kafka
   chown -R kafka:kafka /var/lib/kafka

   Copy to Clipboard Toggle word wrap

   You can now run a default configuration of Kafka as a single-node cluster.

6. (Optional) Run other Kafka components, like Kafka Connect, on the same host.

   To run other components, specify the hostname and port to connect to the Kafka broker using the bootstrap.servers property in the component configuration.

   Example bootstrap servers configuration pointing to a single Kafka broker on the same host

   bootstrap.servers=localhost:9092

   Copy to Clipboard Toggle word wrap

   However, we recommend installing and running Kafka components on separate hosts.

7. (Optional) Install Kafka components on separate hosts.

   1. Extract the Kafka installation files into the /opt/kafka/ directory on each host.
   2. Change the ownership of the /opt/kafka/ directory to the Kafka user on each host.

   3. Update the bootstrap.servers property to connect the components to the Kafka brokers running on different hosts.

      Example bootstrap servers configuration pointing to Kafka brokers on different hosts

      bootstrap.servers=kafka0.<host_ip_address>:9092,kafka1.<host_ip_address>:9092,kafka2.<host_ip_address>:9092

      Copy to Clipboard Toggle word wrap

      You can use this configuration for Kafka Connect, MirrorMaker 2, and the Kafka Bridge.

Running the cluster

1. Generate a unique ID for the Kafka cluster.

   You can use the kafka-storage tool to do this:

   ./bin/kafka-storage.sh random-uuid

   Copy to Clipboard Toggle word wrap

   The command returns an ID.

   Note

   A cluster ID is required in KRaft mode.

2. Edit the Kafka configuration file ./config/server.properties. Set the log.dirs option to /var/lib/kafka/:

   log.dirs=/var/lib/kafka/

   Copy to Clipboard Toggle word wrap

3. Start Kafka:

   ./bin/kafka-server-start.sh -daemon ./config/server.properties

   Copy to Clipboard Toggle word wrap

4. Check that Kafka is running:

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

   Returns:

   process ID kafka.Kafka ./config/server.properties

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the Kafka console producer and configure it to send messages to a new topic:

   ./bin/kafka-console-producer.sh --broker-list <bootstrap_address> --topic <topic-name>

   Copy to Clipboard Toggle word wrap

   For example:

   ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

   Copy to Clipboard Toggle word wrap

2. Enter several messages into the console. Press Enter to send each individual message to your new topic:

   >message 1
   >message 2
   >message 3
   >message 4

   Copy to Clipboard Toggle word wrap

   When Kafka creates a new topic automatically, you might receive a warning that the topic does not exist:

   WARN Error while fetching metadata with correlation id 39 :
   {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

   Copy to Clipboard Toggle word wrap

   The warning should not reappear after you send further messages.

3. In a new terminal window, start the Kafka console consumer and configure it to read messages from the beginning of your new topic.

   ./bin/kafka-console-consumer.sh --bootstrap-server <bootstrap_address> --topic <topic-name> --from-beginning

   Copy to Clipboard Toggle word wrap

   For example:

   ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

   Copy to Clipboard Toggle word wrap

   The incoming messages display in the consumer console.

4. Switch to the producer console and send additional messages. Check that they display in the consumer console.
5. Stop the Kafka console producer and then the consumer by pressing Ctrl+C.

Procedure

1. Stop the Kafka broker.

   ./bin/kafka-server-stop.sh

   Copy to Clipboard Toggle word wrap

2. Confirm that the Kafka broker is stopped.

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

3. Stop ZooKeeper.

   ./bin/zookeeper-server-stop.sh

   Copy to Clipboard Toggle word wrap