Chapter 2. Getting started

Procedure

1. Add new kafka user and group.

   sudo groupadd kafka
   sudo useradd -g kafka kafka
   sudo passwd kafka

   Copy to Clipboard Toggle word wrap

2. Create directory /opt/kafka.

   sudo mkdir /opt/kafka

   Copy to Clipboard Toggle word wrap

3. Create a temporary directory and extract the contents of the AMQ Streams ZIP file.

   mkdir /tmp/kafka
   unzip amq-streams_y.y-x.x.x.zip -d /tmp/kafka

   Copy to Clipboard Toggle word wrap

4. Move the extracted contents into /opt/kafka directory and delete the temporary directory.

   sudo mv /tmp/kafka/kafka_y.y-x.x.x/* /opt/kafka/
   rm -r /tmp/kafka

   Copy to Clipboard Toggle word wrap

5. Change the ownership of the /opt/kafka directory to the kafka user.

   sudo chown -R kafka:kafka /opt/kafka

   Copy to Clipboard Toggle word wrap

6. Create directory /var/lib/zookeeper for storing Zookeeper data and set its ownership to the kafka user.

   sudo mkdir /var/lib/zookeeper
   sudo chown -R kafka:kafka /var/lib/zookeeper

   Copy to Clipboard Toggle word wrap

7. Create directory /var/lib/kafka for storing Kafka data and set its ownership to the kafka user.

   sudo mkdir /var/lib/kafka
   sudo chown -R kafka:kafka /var/lib/kafka

   Copy to Clipboard Toggle word wrap

Running the cluster

1. Edit the Zookeeper configuration file /opt/kafka/config/zookeeper.properties. Set the dataDir option to /var/lib/zookeeper/.

   dataDir=/var/lib/zookeeper/

   Copy to Clipboard Toggle word wrap

2. Start Zookeeper.

   su - kafka
   /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties

   Copy to Clipboard Toggle word wrap

3. Make sure that Apache Zookeeper is running.

   jcmd | grep zookeeper

   Copy to Clipboard Toggle word wrap

4. Edit the Kafka configuration file /opt/kafka/config/server.properties. Set the log.dirs option to /var/lib/kafka/.

   log.dirs=/var/lib/kafka/

   Copy to Clipboard Toggle word wrap

5. Start Kafka.

   su - kafka
   /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties

   Copy to Clipboard Toggle word wrap

6. Make sure that Kafka is running.

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the Kafka console producer.

   bin/kafka-console-producer.sh --broker-list <bootstrap-address> --topic <topic-name>

   Copy to Clipboard Toggle word wrap

   For example:

   bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

   Copy to Clipboard Toggle word wrap
2. Type your message into the console where the producer is running.
3. Press Enter to send.
4. Press Ctrl+C to exit the Kafka console producer.

5. Start the message receiver.

   bin/kafka-console-consumer.sh --bootstrap-server <bootstrap-address> --topic <topic-name> --from-beginning

   Copy to Clipboard Toggle word wrap

   For example:

   bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

   Copy to Clipboard Toggle word wrap
6. Confirm that you see the incoming messages in the consumer console.
7. Press Crtl+C to exit the Kafka console consumer.

Procedure

1. Stop the Kafka broker.

   su - kafka
   /opt/kafka/bin/kafka-server-stop.sh

   Copy to Clipboard Toggle word wrap

2. Confirm that the Kafka broker is stopped.

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap

3. Stop Zookeeper.

   su - kafka
   /opt/kafka/bin/zookeeper-server-stop.sh

   Copy to Clipboard Toggle word wrap

Procedure

1. Open Zookeeper and Kafka broker configuration files in a text editor. The configuration files are located at :

   Zookeeper

   /opt/kafka/config/zookeeper.properties

   Kafka

   /opt/kafka/config/server.properties

2. Edit the configuration options. The configuration files are in the Java properties format. Every configuration option should be on separate line in the following format:

   <option> = <value>

   Copy to Clipboard Toggle word wrap

   Lines starting with # or ! will be treated as comments and will be ignored by AMQ Streams components.

   # This is a comment

   Copy to Clipboard Toggle word wrap

   Values can be split into multiple lines by using \ directly before the newline / carriage return.

   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="bob" \
       password="bobs-password";

   Copy to Clipboard Toggle word wrap
3. Save the changes
4. Restart the Zookeeper or Kafka broker
5. Repeat this procedure on all the nodes of the cluster.