Chapter 32. Downgrading Streams for Apache Kafka
If you are encountering issues with the version of Streams for Apache Kafka you upgraded to, you can revert to a previous version.
If you installed Streams for Apache Kafka using the YAML installation files, you can use the files from the previous release to perform the downgrade.
Downgrading Streams for Apache Kafka involves two key steps:
- Downgrading the Kafka cluster to a version supported by the target Cluster Operator (if required).
- Downgrading the Cluster Operator to the previous release.
If downgrading through multiple versions, you may need to repeat these steps.
Always downgrade Kafka before the Cluster Operator to maintain compatibility. Ensure that the Kafka version you downgrade to is supported by the target Cluster Operator. The metadata version of the Kafka cluster must not be higher than the maximum version supported by the downgraded Kafka version.
The following downgrade instructions are only suitable if you installed Streams for Apache Kafka using the installation files. If you installed Streams for Apache Kafka using another method, such as OperatorHub, downgrade may not be supported by that method unless specified in its documentation.
32.1. Downgrading KRaft-based Kafka clusters and client applications Copy linkLink copied to clipboard!
Downgrade a KRaft-based Kafka cluster to an earlier version. When downgrading a KRaft-based Kafka cluster to a lower version, like moving from 3.9.0 to 3.8.0, ensure that the metadata version used by the Kafka cluster is a version supported by the Kafka version you want to downgrade to. The metadata version for the Kafka version you are downgrading from must not be higher than the version you are downgrading to.
Consult the Apache Kafka documentation for information regarding the support and limitations associated with KRaft-based downgrades.
Prerequisites
- The Cluster Operator is up and running.
Before you downgrade the Kafka cluster, check the following for the
Kafkaresource:-
The
Kafkacustom resource does not contain options that are not supported by the Kafka version being downgraded to. -
spec.kafka.metadataVersionis set to a version that is supported by the Kafka version being downgraded to.
-
The
Procedure
Update the Kafka cluster configuration.
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change the
metadataVersionversion to a version supported by the Kafka version you are downgrading to; leave theKafka.spec.kafka.versionunchanged at the current Kafka version.For example, if downgrading from Kafka 3.9.0 to 3.8.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe value of
metadataVersionmust be a string to prevent it from being interpreted as a floating point number.-
Save the change, and wait for Cluster Operator to update
.status.kafkaMetadataVersionfor theKafkaresource. Change the
Kafka.spec.kafka.versionto the previous version.For example, if downgrading from Kafka 3.9.0 to 3.8.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the image for the Kafka version is different from the image defined in
STRIMZI_KAFKA_IMAGESfor the Cluster Operator, updateKafka.spec.kafka.image.Wait for the Cluster Operator to update the cluster.
You can check the downgrade has completed successfully from the status of the
Kafkaresource.Downgrade all client applications (consumers) to use the previous version of the client binaries.
The Kafka cluster and clients are now using the previous Kafka version.
32.2. Downgrading Kafka when using ZooKeeper Copy linkLink copied to clipboard!
If you are using Kafka in ZooKeeper mode, the downgrade process involves changing the Kafka version and the related log.message.format.version and inter.broker.protocol.version properties.
32.2.1. Kafka version compatibility for downgrades Copy linkLink copied to clipboard!
Kafka downgrades are dependent on compatible current and target Kafka versions, and the state at which messages have been logged.
You cannot revert to the previous Kafka version if that version does not support any of the inter.broker.protocol.version settings which have ever been used in that cluster, or messages have been added to message logs that use a newer log.message.format.version.
The inter.broker.protocol.version determines the schemas used for persistent metadata stored by the broker, such as the schema for messages written to __consumer_offsets. If you downgrade to a version of Kafka that does not understand an inter.broker.protocol.version that has ever been previously used in the cluster the broker will encounter data it cannot understand.
If the target downgrade version of Kafka has:
-
The same
log.message.format.versionas the current version, the Cluster Operator downgrades by performing a single rolling restart of the brokers. A different
log.message.format.version, downgrading is only possible if the running cluster has always hadlog.message.format.versionset to the version used by the downgraded version. This is typically only the case if the upgrade procedure was aborted before thelog.message.format.versionwas changed. In this case, the downgrade requires:- Two rolling restarts of the brokers if the interbroker protocol of the two versions is different
- A single rolling restart if they are the same
Downgrading is not possible if the new version has ever used a log.message.format.version that is not supported by the previous version, including when the default value for log.message.format.version is used. For example, this resource can be downgraded to Kafka version 3.8.0 because the log.message.format.version has not been changed:
The downgrade would not be possible if the log.message.format.version was set at "3.9" or a value was absent, so that the parameter took the default value for a 3.9.0 broker of 3.9.
From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set.
32.2.2. Downgrading ZooKeeper-based Kafka clusters and client applications Copy linkLink copied to clipboard!
Downgrade a ZooKeeper-based Kafka cluster to an earlier version. When downgrading a ZooKeeper-based Kafka cluster to a lower version, like moving from 3.9.0 to 3.8.0, ensure that the inter-broker protocol version used by the Kafka cluster is a version supported by the Kafka version you want to downgrade to. The inter-broker protocol version for the Kafka version you are downgrading from must not be higher than the version you are downgrading to.
Consult the Apache Kafka documentation for information regarding the support and limitations associated with ZooKeeper-based downgrades.
Prerequisites
- The Cluster Operator is up and running.
Before you downgrade the Kafka cluster, check the following for the
Kafkaresource:- IMPORTANT: Compatibility of Kafka versions.
-
The
Kafkacustom resource does not contain options that are not supported by the Kafka version being downgraded to. Kafka.spec.kafka.confighas alog.message.format.versionandinter.broker.protocol.versionthat is supported by the Kafka version being downgraded to.From Kafka 3.0.0, when the
inter.broker.protocol.versionis set to3.0or higher, thelog.message.format.versionoption is ignored and doesn’t need to be set.
Procedure
Update the Kafka cluster configuration.
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change the
inter.broker.protocol.versionversion (andlog.message.format.version, if applicable) to a version supported by the Kafka version you are downgrading to; leave theKafka.spec.kafka.versionunchanged at the current Kafka version.For example, if downgrading from Kafka 3.9.0 to 3.8.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe value of
log.message.format.versionandinter.broker.protocol.versionmust be strings to prevent them from being interpreted as floating point numbers.Save and exit the editor, then wait for rolling updates to complete.
Check the progress of the rolling updates by watching the pod state transitions:
oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow The rolling updates ensure that each pod is using the specified Kafka inter-broker protocol version.
Change the
Kafka.spec.kafka.versionto the previous version.For example, if downgrading from Kafka 3.9.0 to 3.8.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the image for the Kafka version is different from the image defined in
STRIMZI_KAFKA_IMAGESfor the Cluster Operator, updateKafka.spec.kafka.image.Wait for the Cluster Operator to update the cluster.
You can check the downgrade has completed successfully from the status of the
Kafkaresource.Downgrade all client applications (consumers) to use the previous version of the client binaries.
The Kafka cluster and clients are now using the previous Kafka version.
If you are reverting back to a version of Streams for Apache Kafka earlier than 1.7, which uses ZooKeeper for the storage of topic metadata, delete the internal topic store topics from the Kafka cluster.
oc run kafka-admin -ti --image=registry.redhat.io/amq-streams/kafka-39-rhel9:2.9.3 --rm=true --restart=Never -- ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi-topic-operator-kstreams-topic-store-changelog --delete && ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi_store_topic --delete
oc run kafka-admin -ti --image=registry.redhat.io/amq-streams/kafka-39-rhel9:2.9.3 --rm=true --restart=Never -- ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi-topic-operator-kstreams-topic-store-changelog --delete && ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi_store_topic --deleteCopy to Clipboard Copied! Toggle word wrap Toggle overflow
32.3. Downgrading the Cluster Operator to a previous version Copy linkLink copied to clipboard!
If you are encountering issues with Streams for Apache Kafka, you can revert your installation.
This procedure describes how to downgrade a Cluster Operator deployment to a previous version.
Prerequisites
- An existing Cluster Operator deployment is available.
- You have downloaded the installation files for the previous version.
Before you begin
Check the downgrade requirements of the Streams for Apache Kafka feature gates. If a feature gate is permanently enabled, you may need to downgrade to a version that allows you to disable it before downgrading to your target version.
Procedure
Take note of any configuration changes made during the previous Cluster Operator installation.
Any changes will be overwritten by the previous version of the Cluster Operator.
- Revert your custom resources to reflect the supported configuration options available for the version of Streams for Apache Kafka you are downgrading to.
Update the Cluster Operator.
Modify the installation files for the previous version according to the namespace the Cluster Operator is running in.
On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
If you modified one or more environment variables in your existing Cluster Operator
Deployment, edit theinstall/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlfile to use those environment variables.
When you have an updated configuration, deploy it along with the rest of the installation resources:
oc replace -f install/cluster-operator
oc replace -f install/cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the rolling updates to complete.
Get the image for the Kafka pod to ensure the downgrade was successful:
oc get pod my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'oc get pod my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow The image tag shows the new Streams for Apache Kafka version followed by the Kafka version. For example,
<strimzi_version>-kafka-<kafka_version>.You can also check the downgrade has completed successfully from the status of the
Kafkaresource.