Chapter 3. Getting started

1. Add a new kafka user and group:

   groupadd kafka
   useradd -g kafka kafka
   passwd kafka

   Copy to Clipboard Toggle word wrap

2. Extract and move the contents of the amq-streams-<version>-bin.zip file into the /opt/kafka directory:

   unzip amq-streams-<version>-bin.zip -d /opt
   mv /opt/kafka*redhat* /opt/kafka

   Copy to Clipboard Toggle word wrap

   This step requires admin privileges.

3. Change the ownership of the /opt/kafka directory to the kafka user:

   chown -R kafka:kafka /opt/kafka

   Copy to Clipboard Toggle word wrap

4. Create directory /var/lib/zookeeper for storing ZooKeeper data and set its ownership to the kafka user:

   mkdir /var/lib/zookeeper
   chown -R kafka:kafka /var/lib/zookeeper

   Copy to Clipboard Toggle word wrap

5. Create directory /var/lib/kafka for storing Kafka data and set its ownership to the kafka user:

   mkdir /var/lib/kafka
   chown -R kafka:kafka /var/lib/kafka

   Copy to Clipboard Toggle word wrap

   You can now run a default configuration of Kafka as a single-node cluster.

   You can also use the installation to run other Kafka components, like Kafka Connect, on the same host.

   To run other components, specify the hostname and port to connect to the Kafka broker using the bootstrap.servers property in the component configuration.

   Example bootstrap servers configuration pointing to a single Kafka broker on the same host

   bootstrap.servers=localhost:9092

   Copy to Clipboard Toggle word wrap

   However, we recommend installing and running Kafka components on separate hosts.

6. (Optional) Install Kafka components on separate hosts.

   1. Extract the installation files to the /opt/kafka directory on each host.
   2. Change the ownership of the /opt/kafka directory to the kafka user.

   3. Add bootstrap.servers configuration that connects the component to the host (or hosts in a multi-node environment) running the Kafka brokers.

      Example bootstrap servers configuration pointing to Kafka brokers on different hosts

      bootstrap.servers=kafka0.<host_ip_address>:9092,kafka1.<host_ip_address>:9092,kafka2.<host_ip_address>:9092

      Copy to Clipboard Toggle word wrap

      You can use this configuration for Kafka Connect, MirrorMaker 2, and the Kafka Bridge.

Running the cluster

1. Edit the ZooKeeper configuration file /opt/kafka/config/zookeeper.properties. Set the dataDir option to /var/lib/zookeeper/:

   dataDir=/var/lib/zookeeper/

   Copy to Clipboard Toggle word wrap

2. Edit the Kafka configuration file /opt/kafka/config/server.properties. Set the log.dirs option to /var/lib/kafka/:

   log.dirs=/var/lib/kafka/

   Copy to Clipboard Toggle word wrap

3. Switch to the kafka user:

   su - kafka

   Copy to Clipboard Toggle word wrap

4. Start ZooKeeper:

   /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

   Copy to Clipboard Toggle word wrap

5. Check that ZooKeeper is running:

   jcmd | grep zookeeper

   Copy to Clipboard Toggle word wrap

   Returns:

   number org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/kafka/config/zookeeper.properties

   Copy to Clipboard Toggle word wrap

6. Start Kafka:

   /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

   Copy to Clipboard Toggle word wrap

7. Check that Kafka is running:

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

   Returns:

   number kafka.Kafka /opt/kafka/config/server.properties

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the Kafka console producer and configure it to send messages to a new topic:

   /opt/kafka/bin/kafka-console-producer.sh --broker-list <bootstrap_address> --topic <topic-name>

   Copy to Clipboard Toggle word wrap

   For example:

   /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

   Copy to Clipboard Toggle word wrap

2. Enter several messages into the console. Press Enter to send each individual message to your new topic:

   >message 1
   >message 2
   >message 3
   >message 4

   Copy to Clipboard Toggle word wrap

   When Kafka creates a new topic automatically, you might receive a warning that the topic does not exist:

   WARN Error while fetching metadata with correlation id 39 :
   {4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

   Copy to Clipboard Toggle word wrap

   The warning should not reappear after you send further messages.

3. In a new terminal window, start the Kafka console consumer and configure it to read messages from the beginning of your new topic.

   /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <bootstrap_address> --topic <topic-name> --from-beginning

   Copy to Clipboard Toggle word wrap

   For example:

   /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

   Copy to Clipboard Toggle word wrap

   The incoming messages display in the consumer console.

4. Switch to the producer console and send additional messages. Check that they display in the consumer console.
5. Stop the Kafka console producer and then the consumer by pressing Ctrl+C.

Procedure

1. Stop the Kafka broker.

   su - kafka
   /opt/kafka/bin/kafka-server-stop.sh

   Copy to Clipboard Toggle word wrap

2. Confirm that the Kafka broker is stopped.

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

3. Stop ZooKeeper.

   su - kafka
   /opt/kafka/bin/zookeeper-server-stop.sh

   Copy to Clipboard Toggle word wrap