Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 17. AMQ Streams and Kafka upgrades

AMQ Streams can be upgraded with no cluster downtime. Each version of AMQ Streams supports one or more versions of Apache Kafka: you can upgrade to a higher Kafka version as long as it is supported by your version of AMQ Streams. Newer versions of AMQ Streams may support newer versions of Kafka, but you need to upgrade AMQ Streams before you can upgrade to a higher supported Kafka version.

17.1. Upgrade prerequisites
Copy link

Before you begin the upgrade process, make sure that:

17.2. Upgrade process
Copy link

Upgrading AMQ Streams is a two-stage process. To upgrade brokers and clients without downtime, you must complete the upgrade procedures in the following order:

  1. Upgrade to the latest AMQ Streams version.

  2. Upgrade all Kafka brokers and client applications to the latest Kafka version

17.3. Kafka versions
Copy link

Kafka’s log message format version and inter-broker protocol version specify, respectively, the log format version appended to messages and the version of the Kafka protocol used in a cluster. To ensure the correct versions are used, the upgrade process involves making configuration changes to existing Kafka brokers and code changes to client applications (consumers and producers).

The following table shows the differences between Kafka versions:

Expand
Kafka versionInterbroker protocol versionLog message format versionZooKeeper version

3.0.0

3.0

3.0

3.6.3

2.8.0

2.8

2.8

3.5.9

Inter-broker protocol version

In Kafka, the network protocol used for inter-broker communication is called the inter-broker protocol. Each version of Kafka has a compatible version of the inter-broker protocol. The minor version of the protocol typically increases to match the minor version of Kafka, as shown in the preceding table.

The inter-broker protocol version is set cluster wide in the Kafka resource. To change it, you edit the inter.broker.protocol.version property in Kafka.spec.kafka.config.

Log message format version

When a producer sends a message to a Kafka broker, the message is encoded using a specific format. The format can change between Kafka releases, so messages specify which version of the message format they were encoded with.

The properties used to set a specific message format version are as follows:

  • message.format.version property for topics
  • log.message.format.version property for Kafka brokers

From Kafka 3.0.0, the message format version values are assumed to match the inter.broker.protocol.version and don’t need to be set. The values reflect the Kafka version used.

When upgrading to Kafka 3.0.0 or higher, you can remove these settings when you update the inter.broker.protocol.version. Otherwise, set the message format version based on the Kafka version you are upgrading to.

The default value of message.format.version for a topic is defined by the log.message.format.version that is set on the Kafka broker. You can manually set the message.format.version of a topic by modifying its topic configuration.

17.4. Upgrading to AMQ Streams 2.0
Copy link

The steps to upgrade your deployment to use AMQ Streams 2.0 are outlined in this section.

The availability of Kafka clusters managed by AMQ Streams is not affected by the upgrade operation.

Note

Refer to the documentation supporting a specific version of AMQ Streams for information on how to upgrade to that version.

17.4.1. Upgrading Kafka brokers and ZooKeeper
Copy link

This procedure describes how to upgrade Kafka brokers and ZooKeeper on a host machine to use the latest version of AMQ Streams.

Prerequisites

  • You are logged in to Red Hat Enterprise Linux as the kafka user.

Procedure

For each Kafka broker in your AMQ Streams cluster and one at a time:

  1. Download the AMQ Streams archive from the Customer Portal.

    Note

    If prompted, log in to your Red Hat account.

  2. On the command line, create a temporary directory and extract the contents of the amq-streams-x.y.z-bin.zip file. 

    mkdir /tmp/kafka
unzip amq-streams-x.y.z-bin.zip -d /tmp/kafka
    Copy to Clipboard Toggle word wrap

  3. If running, stop ZooKeeper and the Kafka broker running on the host. 

    /opt/kafka/bin/zookeeper-server-stop.sh
/opt/kafka/bin/kafka-server-stop.sh
jcmd | grep zookeeper
jcmd | grep kafka
    Copy to Clipboard Toggle word wrap

  4. Delete the libs, bin, and docs directories from your existing installation: 

    rm -rf /opt/kafka/libs /opt/kafka/bin /opt/kafka/docs
    Copy to Clipboard Toggle word wrap

  5. Copy the libs, bin, and docs directories from the temporary directory: 

    cp -r /tmp/kafka/kafka_y.y-x.x.x/libs /opt/kafka/
cp -r /tmp/kafka/kafka_y.y-x.x.x/bin /opt/kafka/
cp -r /tmp/kafka/kafka_y.y-x.x.x/docs /opt/kafka/
    Copy to Clipboard Toggle word wrap

  6. Delete the temporary directory. 

    rm -r /tmp/kafka
    Copy to Clipboard Toggle word wrap
  7. In a text editor, open the broker properties file, commonly stored in the /opt/kafka/config/ directory.

  8. Check that the inter.broker.protocol.version and log.message.format.version properties are set to the current version: 

    inter.broker.protocol.version=2.8
log.message.format.version=2.8
    Copy to Clipboard Toggle word wrap

    Leaving the inter.broker.protocol.version unchanged ensures that the brokers can continue to communicate with each other throughout the upgrade.

    If the properties are not configured, add them with the current version.

  9. Restart the updated ZooKeeper and Kafka broker: 

    /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
    Copy to Clipboard Toggle word wrap

    The Kafka broker and Zookeeper will start using the binaries for the latest Kafka version.

  10. Verify that the restarted Kafka broker has caught up with the partition replicas it is following. Use the kafka-topics.sh tool to ensure that all replicas contained in the broker are back in sync. For instructions, see Listing and describing topics.
  11. Perform the procedures to upgrade Kafka, as described in Section 17.5, “Upgrading Kafka”.

17.4.2. Upgrading Kafka Connect
Copy link

This procedure describes how to upgrade a Kafka Connect cluster on a host machine.

Prerequisites

  • You are logged in to Red Hat Enterprise Linux as the kafka user.
  • Kafka Connect is not started.

Procedure

For each Kafka broker in your AMQ Streams cluster and one at a time:

  1. Download the AMQ Streams archive from the Customer Portal.

    Note

    If prompted, log in to your Red Hat account.

  2. On the command line, create a temporary directory and extract the contents of the amq-streams-x.y.z-bin.zip file. 

    mkdir /tmp/kafka
unzip amq-streams-x.y.z-bin.zip -d /tmp/kafka
    Copy to Clipboard Toggle word wrap

  3. If running, stop the Kafka broker and ZooKeeper running on the host. 

    /opt/kafka/bin/kafka-server-stop.sh
/opt/kafka/bin/zookeeper-server-stop.sh
    Copy to Clipboard Toggle word wrap

  4. Delete the libs, bin, and docs directories from your existing installation: 

    rm -rf /opt/kafka/libs /opt/kafka/bin /opt/kafka/docs
    Copy to Clipboard Toggle word wrap

  5. Copy the libs, bin, and docs directories from the temporary directory: 

    cp -r /tmp/kafka/kafka_y.y-x.x.x/libs /opt/kafka/
cp -r /tmp/kafka/kafka_y.y-x.x.x/bin /opt/kafka/
cp -r /tmp/kafka/kafka_y.y-x.x.x/docs /opt/kafka/
    Copy to Clipboard Toggle word wrap

  6. Delete the temporary directory. 

    rm -r /tmp/kafka
    Copy to Clipboard Toggle word wrap

  7. Start Kafka Connect in either standalone or distributed mode.

    • To start in standalone mode, run the connect-standalone.sh script. Specify the Kafka Connect standalone configuration file and the configuration files of your Kafka Connect connectors. 

      su - kafka
/opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties connector1.properties
[connector2.properties ...]
      Copy to Clipboard Toggle word wrap

    • To start in distributed mode, start the Kafka Connect workers with the /opt/kafka/config/connect-distributed.properties configuration file on all Kafka Connect nodes: 

      su - kafka
/opt/kafka/bin/connect-distributed.sh /opt/kafka/config/connect-distributed.properties
      Copy to Clipboard Toggle word wrap

  8. Verify that Kafka Connect is running:

    • In standalone mode: 

      jcmd | grep ConnectStandalone
      Copy to Clipboard Toggle word wrap

    • In distributed mode: 

      jcmd | grep ConnectDistributed
      Copy to Clipboard Toggle word wrap
  9. Verify that Kafka Connect is producing and consuming data as expected.

Additional resources

17.5. Upgrading Kafka
Copy link

After you have upgraded your binaries to use the latest version of AMQ Streams, you can upgrade your brokers to use a higher supported version of Kafka.

Following the Kafka upgrade, if required, you can upgrade Kafka consumers to use the incremental cooperative rebalance protocol.

Manually configure and restart all Kafka brokers to use the new inter-broker protocol version. After performing these steps, data is transmitted between the Kafka brokers using the new inter-broker protocol version.

Note

From Kafka 3.0.0, message format version values are assumed to match the inter.broker.protocol.version and don’t need to be set. The values reflect the Kafka version used.

Messages received are still appended to the message logs in the earlier message format version.

Warning

Downgrading AMQ Streams is not possible after completing this procedure.

Prerequisites

Procedure

For each Kafka broker in your AMQ Streams cluster and one at a time:

  1. In a text editor, open the broker properties file for the Kafka broker you want to update. Broker properties files are commonly stored in the /opt/kafka/config/ directory.

  2. Set the inter.broker.protocol.version to 3.0. 

    inter.broker.protocol.version=3.0
    Copy to Clipboard Toggle word wrap

  3. On the command line, stop the Kafka broker that you modified: 

    /opt/kafka/bin/kafka-server-stop.sh
jcmd | grep kafka
    Copy to Clipboard Toggle word wrap

  4. Restart the Kafka broker that you modified: 

    /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
    Copy to Clipboard Toggle word wrap
  5. Verify that the restarted Kafka broker has caught up with the partition replicas it is following. Use the kafka-topics.sh tool to ensure that all replicas contained in the broker are back in sync. For instructions, see Listing and describing topics.

You can upgrade Kafka consumers and Kafka Streams applications to use the incremental cooperative rebalance protocol for partition rebalances instead of the default eager rebalance protocol. The new protocol was added in Kafka 2.4.0.

Consumers keep their partition assignments in a cooperative rebalance and only revoke them at the end of the process, if needed to achieve a balanced cluster. This reduces the unavailability of the consumer group or Kafka Streams application.

Note

Upgrading to the incremental cooperative rebalance protocol is optional. The eager rebalance protocol is still supported.

Prerequisites

Procedure

To upgrade a Kafka consumer to use the incremental cooperative rebalance protocol:

  1. Replace the Kafka clients .jar file with the new version.
  2. In the consumer configuration, append cooperative-sticky to the partition.assignment.strategy. For example, if the range strategy is set, change the configuration to range, cooperative-sticky.
  3. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.
  4. Reconfigure each consumer in the group by removing the earlier partition.assignment.strategy from the consumer configuration, leaving only the cooperative-sticky strategy.
  5. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.

To upgrade a Kafka Streams application to use the incremental cooperative rebalance protocol:

  1. Replace the Kafka Streams .jar file with the new version.
  2. In the Kafka Streams configuration, set the upgrade.from configuration parameter to the Kafka version you are upgrading from (for example, 2.3).
  3. Restart each of the stream processors (nodes) in turn.
  4. Remove the upgrade.from configuration parameter from the Kafka Streams configuration.
  5. Restart each consumer in the group in turn.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat