Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 8. AMQ Streams and Kafka upgrades

Each version of AMQ Streams supports a range of versions of Apache Kafka. You can upgrade to a higher Kafka version as long as that version is supported by your version of AMQ Streams. In some cases, you can also downgrade to a lower supported Kafka version.

When a newer version of AMQ Streams is available, it may provide support for newer versions of Kafka. Therefore, you will need to upgrade to the new version of AMQ Streams before you can upgrade to a higher supported Kafka version. Upgrading the version of AMQ Streams is done by upgrading the Cluster Operator deployment to the new version.

This procedure will describe how to upgrade a Cluster Operator deployment from version 1.0.0 to version 1.1.0.

The availability of Kafka clusters managed by the Cluster Operator is not affected by the upgrade operation.

Prerequisites

  • An existing version 1.0.0 Cluster Operator deployment to be upgraded.

Procedure

  1. Update your existing Kafka, KafkaConnect, KafkaConnectS2I, and KafkaMirrorMaker resources, as follows:

    • Add Kafka.spec.kafka.version with the value 2.0.0
    • Add KafkaConnect.spec.version with the value 2.0.0
    • Add KafkaConnectS2I.spec.version with the value 2.0.0

    • Add KafkaMirrorMaker.spec.version with the value 2.0.0

      Wait for the associated rolling updates to complete.

  2. Backup the existing Cluster Operator resources.

    On OpenShift use oc get: 

    oc get all -l app=strimzi -o yaml > strimzi-backup.yaml

  3. Update the Cluster Operator. You will need to modify the installation files according to the namespace the Cluster Operator is running in.

    On Linux, use: 

    sed -i 's/namespace: .*/namespace: my-namespace/' install/cluster-operator/*RoleBinding*.yaml

    On MacOS, use: 

    sed -i '' 's/namespace: .*/namespace: my-namespace/' install/cluster-operator/*RoleBinding*.yaml

    If you modified one or more environment variables in your existing Cluster Operator Deployment, edit install/cluster-operator/050-Deployment-cluster-operator.yaml to reflect the changes that you made.

  4. When you have an updated configuration you can deploy it along with the rest of the install resources.

    On OpenShift use oc apply: 

    oc apply -f install/cluster-operator

    Wait for the associated rolling updates to complete.

  5. Update existing resources to cope with deprecated custom resource properties.

    • If you have Kafka resources that specify Kafka.spec.topicOperator, rewrite them to use Kafka.spec.entityOperator.topicOperator instead.

8.2. Upgrading and downgrading Kafka versions
Copy link

8.2.1. Versions and images overview
Copy link

The Cluster Operator embeds knowledge of different Kafka versions, but not of the corresponding images in which those versions are provided. Using the STRIMZI_KAFKA_IMAGES environment variable, the Cluster Operator is configured with a mapping between the Kafka version and the image to be used when that version is requested in a given Kafka resource.

Each Kafka resource can be configured with a Kafka.spec.kafka.version. If Kafka.spec.kafka.image is not configured then the default image for the given version will be used. If Kafka.spec.kafka.image is given, this overrides the default.

Warning

The Cluster Operator cannot validate that an image actually contains a Kafka broker of the expected version. Take care to ensure that the given image corresponds to the given Kafka version.

8.2.2. Kafka upgrades using the Cluster Operator
Copy link

How the Cluster Operator will perform an upgrade depends on the differences between:

  • The interbroker protocol version of the two Kafka versions
  • The log message format version of the two Kafka versions
  • The version of Zookeeper used by the two Kafka versions

When each of these versions is the same, as is typically the case for a patch level upgrade, then the Cluster Operator can perform the upgrade using a single rolling update of the Kafka brokers. When one or more of these versions differ, the Cluster Operator will require two or three rolling updates of the Kafka brokers to perform the upgrade.

8.2.3. Upgrading brokers to a newer Kafka version
Copy link

This procedure describes how to upgrade a AMQ Streams Kafka cluster from one version of Kafka to a higher version; for example, from 2.0.0 to 2.1.1.

Prerequisites

  • The Cluster Operator, which supports both versions of Kafka, is up and running.
  • A Kafka resource to be upgraded.
  • You have checked that your Kafka.spec.kafka.config contains no options that are not supported in the version of Kafka that you are upgrading to.
  • Determine whether the new Kafka version has a different log message format version than the previous version. See the following table for help.
Expand
Kafka versionInterbroker protocol versionLog message format versionZookeeper version

2.0.0

2.0

2.0

3.4.13

2.1.1

2.1

2.1

3.4.13

Procedure

  1. If the log message format version of the previous Kafka version is the same as that of the new Kafka version, proceed to the next step. Otherwise, ensure that the Kafka.spec.kafka.config has the log.message.format.version configured to the default for the previous version.

    For example, if upgrading from Kafka 2.0.0: 

    apiVersion: v1alpha1
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.0.0
    config:
      log.message.format.version: "2.0"
      # ...
    Note

    You must format the value of log.message.format.version as a string to prevent it from being interpreted as a number.

    If the log.message.format.version is unset, set it and then wait for the resulting rolling restart of the Kafka cluster to complete.

  2. Change the Kafka.spec.kafka.version to specify the new version, but leave the log.message.format.version as the previous version. If the image to be used is different from the image for the given version of Kafka configured in the Cluster Operator’s STRIMZI_KAFKA_IMAGES then configure the Kafka.spec.kafka.image as well.

    For example, if upgrading from Kafka 2.0.0 to 2.1.1: 

    apiVersion: v1alpha1
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.1.1
    1 
    
    config:
      log.message.format.version: "2.0"
    2 
    
      # ...
    1
    This is changed to the new version
    2
    This remains at the previous version

  3. Wait for the Cluster Operator to upgrade the cluster. If the old and new versions of Kafka have different interbroker protocol versions, look in the Cluster Operator logs for an INFO level message in the following format: 

    Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 2 of 2 completed

    Alternatively, if the old and new versions of Kafka have the same interbroker protocol version, look in the Cluster Operator logs for an INFO level message in the following format: 

    Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 1 of 1 completed

    For example, using grep: 

    oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version upgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

  4. Upgrade all your client applications to use the new version of the client libraries.

    Warning

    You cannot downgrade after completing this step. If, for whatever reason, you need to revert the update at this point, follow the procedure Section 8.2.5, “Downgrading brokers to an older Kafka version”.

  5. If the log message format versions, as identified in step 1, are the same proceed to the next step. Otherwise change the log.message.format.version in Kafka.spec.kafka.config to the default version for the new version of Kafka now being used. For example, if upgrading to 2.1.1: 

    apiVersion: v1alpha1
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.1.1
    config:
      log.message.format.version: "2.1"
      # ...
  6. Wait for the Cluster Operator to update the cluster.

Additional resources

8.2.4. Kafka downgrades using the Cluster Operator
Copy link

Whether and how the Cluster Operator will perform a downgrade depends on the differences between:

  • The interbroker protocol version of the two Kafka versions
  • The log message format version of the two Kafka versions
  • The version of Zookeeper used by the two Kafka versions

If the downgraded version of Kafka has the same log message format version then downgrading is always possible. In this case the Cluster Operator will be able to downgrade by performing a single rolling restart of the brokers.

If the downgraded version of Kafka has a different log message format version, then downgrading is only possible if the running cluster has always had log.message.format.version set to the version used by the downgraded version. This is typically only the case if the upgrade procedure was aborted before the log.message.format.version was changed. In this case the downgrade will require two rolling restarts of the brokers if the interbroker protocol of the two versions is different, or a single rolling restart if they are the same.

Note

The previous version is the Kafka version that you are downgrading to (such as 2.0.0). The new version is the version that you are downgrading from (such as 2.1.1).

You can downgrade a AMQ Streams Kafka cluster from one version to a lower version; for example, from 2.1.1 to 2.0.0. Downgrading is not possible if the new version has ever used a log.message.format.version that is not supported by the previous version (including where the default value for log.message.format.version is used).

For example: 

apiVersion: v1alpha1
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.1.1
    config:
      log.message.format.version: "2.0"
      # ...

This resource can be downgraded to Kafka version 2.0.0 because the log.message.format.version has not been changed. If the log.message.format.version were absent (so that the parameter took its default value for a 2.1.0 broker of 2.1), or was "2.1" then downgrade would not be possible.

Prerequisites

  • The Cluster Operator is up and running.
  • A Kafka resource to be downgraded.
  • The Kafka.spec.kafka.config has a log.message.format.version that is supported by the version being downgraded to.
  • You have checked that your Kafka.spec.kafka.config contains no options which are not supported in the version of Kafka being downgraded to.

Procedure

  1. Change the Kafka.spec.kafka.version to specify the previous version. If the image to be used is different from the image for the given version of Kafka configured in the Cluster Operator’s STRIMZI_KAFKA_IMAGES then configure the Kafka.spec.kafka.image as well.

    For example, if downgrading from Kafka 2.1.1 to 2.0.0: 

    apiVersion: v1alpha1
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.0.0
    1 
    
    config:
      log.message.format.version: "2.0"
    2 
    
      # ...
    1
    This is changed to the previous version
    2
    This is unchanged
    Note

    It is necessary to format the value of log.message.format.version as a string to prevent it from being interpreted as a number.

  2. Wait for the Cluster Operator to downgrade the cluster. If both the previous and new versions of Kafka have a different interbroker protocol version, check the Cluster Operator logs for an INFO level message in the following format: 

    Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 2 of 2 completed

    Alternatively, if both the previous and new versions of Kafka have the same interbroker protocol version look in the Cluster Operator logs for an INFO level message in the following format: 

    Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 1 of 1 completed

    For example, using grep: 

    oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version downgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"
  3. Downgrade each client application to use the previous version of the client libraries.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top