Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Getting started

2.1. AMQ Streams distribution
Copy link

AMQ Streams is distributed as single ZIP file. This ZIP file contains AMQ Streams components:

2.2. Downloading an AMQ Streams Archive
Copy link

An archived distribution of AMQ Streams is available for download from the Red Hat website. You can download a copy of the distribution by following the steps below.

Procedure

  • Download the latest version of the Red Hat AMQ Streams archive from the Customer Portal.

2.3. Installing AMQ Streams
Copy link

Follow this procedure to install the latest version of AMQ Streams on Red Hat Enterprise Linux.

For instructions on upgrading an existing cluster to AMQ Streams 1.4, see AMQ Streams and Kafka upgrades.

Prerequisites

Procedure

  1. Add new kafka user and group. 

    sudo groupadd kafka
sudo useradd -g kafka kafka
sudo passwd kafka
    Copy to Clipboard Toggle word wrap

  2. Create directory /opt/kafka. 

    sudo mkdir /opt/kafka
    Copy to Clipboard Toggle word wrap

  3. Create a temporary directory and extract the contents of the AMQ Streams ZIP file. 

    mkdir /tmp/kafka
unzip amq-streams_y.y-x.x.x.zip -d /tmp/kafka
    Copy to Clipboard Toggle word wrap

  4. Move the extracted contents into /opt/kafka directory and delete the temporary directory. 

    sudo mv /tmp/kafka/kafka_y.y-x.x.x/* /opt/kafka/
rm -r /tmp/kafka
    Copy to Clipboard Toggle word wrap

  5. Change the ownership of the /opt/kafka directory to the kafka user. 

    sudo chown -R kafka:kafka /opt/kafka
    Copy to Clipboard Toggle word wrap

  6. Create directory /var/lib/zookeeper for storing ZooKeeper data and set its ownership to the kafka user. 

    sudo mkdir /var/lib/zookeeper
sudo chown -R kafka:kafka /var/lib/zookeeper
    Copy to Clipboard Toggle word wrap

  7. Create directory /var/lib/kafka for storing Kafka data and set its ownership to the kafka user. 

    sudo mkdir /var/lib/kafka
sudo chown -R kafka:kafka /var/lib/kafka
    Copy to Clipboard Toggle word wrap

2.4. Data storage considerations
Copy link

An efficient data storage infrastructure is essential to the optimal performance of AMQ Streams.

AMQ Streams requires block storage and works well with cloud-based block storage solutions, such as Amazon Elastic Block Store (EBS). The use of file storage is not recommended.

Choose local storage when possible. If local storage is not available, you can use a Storage Area Network (SAN) accessed by a protocol such as Fibre Channel or iSCSI.

2.4.1. Apache Kafka and ZooKeeper storage support
Copy link

Use separate disks for Apache Kafka and ZooKeeper.

Kafka supports JBOD (Just a Bunch of Disks) storage, a data storage configuration of multiple disks or volumes. JBOD provides increased data storage for Kafka brokers. It can also improve performance.

Solid-state drives (SSDs), though not essential, can improve the performance of Kafka in large clusters where data is sent to and received from multiple topics asynchronously. SSDs are particularly effective with ZooKeeper, which requires fast, low latency data access.

Note

You do not need to provision replicated storage because Kafka and ZooKeeper both have built-in data replication.

2.4.2. File systems
Copy link

It is recommended that you configure your storage system to use the XFS file system. AMQ Streams is also compatible with the ext4 file system, but this might require additional configuration for best results.

Additional resources

2.5. Running a single node AMQ Streams cluster
Copy link

This procedure shows how to run a basic AMQ Streams cluster consisting of single ZooKeeper and single Apache Kafka node both running on the same host. The default configuration files are used for ZooKeeper and Kafka.

Warning

A single node AMQ Streams cluster does not provide reliability and high availability and is suitable only for development purposes.

Prerequisites

  • AMQ Streams is installed on the host

Running the cluster

  1. Edit the ZooKeeper configuration file /opt/kafka/config/zookeeper.properties. Set the dataDir option to /var/lib/zookeeper/. 

    dataDir=/var/lib/zookeeper/
    Copy to Clipboard Toggle word wrap

  2. Start ZooKeeper. 

    su - kafka
/opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
    Copy to Clipboard Toggle word wrap

  3. Make sure that Apache ZooKeeper is running. 

    jcmd | grep zookeeper
    Copy to Clipboard Toggle word wrap

    Returns: 

    number org.apache.zookeeper.server.quorum.QuorumPeerMain config/zookeeper.properties
    Copy to Clipboard Toggle word wrap

  4. Edit the Kafka configuration file /opt/kafka/config/server.properties. Set the log.dirs option to /var/lib/kafka/. 

    log.dirs=/var/lib/kafka/
    Copy to Clipboard Toggle word wrap

  5. Start Kafka. 

    su - kafka
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
    Copy to Clipboard Toggle word wrap

  6. Make sure that Kafka is running. 

    jcmd | grep kafka
    Copy to Clipboard Toggle word wrap

    Returns: 

    number kafka.Kafka config/server.properties
    Copy to Clipboard Toggle word wrap

Additional resources

2.6. Using the cluster
Copy link

This procedure shows how to start the Kafka producer and consumer to send and receive messages.

Prerequisites

  • AMQ Streams is installed on the host
  • ZooKeeper and Kafka are up and running

A topic is created automatically in this procedure. Alternatively, you can configure and create topics before using the cluster. For more information on creating and managing topics, see Topics.

Procedure

  1. Start the Kafka console producer. 

    /opt/kafka/bin/kafka-console-producer.sh --broker-list <bootstrap-address> --topic <topic-name>
    Copy to Clipboard Toggle word wrap

    For example: 

    /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
    Copy to Clipboard Toggle word wrap
  2. Type your message into the console where the producer is running.

  3. Press Enter to send.

    When Kafka creates a new topic automatically, you might receive warning messages to say the topic does not exist: 

    WARN Error while fetching metadata with correlation id 39 :
{4-3-16-topic1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 :
{my-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
    Copy to Clipboard Toggle word wrap

    The warning messages should not reappear after you start sending messages to the new topic.

    The auto.create.topics.enable configuration property enables the automatic creation of topics.

  4. Start the message receiver. 

    /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <bootstrap-address> --topic <topic-name> --from-beginning
    Copy to Clipboard Toggle word wrap

    For example: 

    /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning
    Copy to Clipboard Toggle word wrap
  5. Confirm that you see the incoming messages in the consumer console.
  6. Press Crtl+C to exit the Kafka console producer and consumer.

2.7. Stopping the AMQ Streams services
Copy link

You can stop the Kafka and ZooKeeper services by running a script. All connections to the Kafka and ZooKeeper services will be terminated.

Prerequisites

  • AMQ Streams is installed on the host
  • ZooKeeper and Kafka are up and running

Procedure

  1. Stop the Kafka broker. 

    su - kafka
/opt/kafka/bin/kafka-server-stop.sh
    Copy to Clipboard Toggle word wrap

  2. Confirm that the Kafka broker is stopped. 

    jcmd | grep kafka
    Copy to Clipboard Toggle word wrap

  3. Stop ZooKeeper. 

    su - kafka
/opt/kafka/bin/zookeeper-server-stop.sh
    Copy to Clipboard Toggle word wrap

2.8. Configuring AMQ Streams
Copy link

Prerequisites

  • AMQ Streams is downloaded and installed on the host

Procedure

  1. Open ZooKeeper and Kafka broker configuration files in a text editor. The configuration files are located at :

    ZooKeeper
    /opt/kafka/config/zookeeper.properties
    Kafka
    /opt/kafka/config/server.properties

  2. Edit the configuration options. The configuration files are in the Java properties format. Every configuration option should be on separate line in the following format: 

    <option> = <value>
    Copy to Clipboard Toggle word wrap

    Lines starting with # or ! will be treated as comments and will be ignored by AMQ Streams components. 

    # This is a comment
    Copy to Clipboard Toggle word wrap

    Values can be split into multiple lines by using \ directly before the newline / carriage return. 

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="bob" \
    password="bobs-password";
    Copy to Clipboard Toggle word wrap
  3. Save the changes
  4. Restart the ZooKeeper or Kafka broker
  5. Repeat this procedure on all the nodes of the cluster.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat