Chapter 11. Downgrading AMQ Streams
If you are encountering issues with the version of AMQ Streams you upgraded to, you can revert your installation to the previous version.
If you used the YAML installation files to install AMQ Streams, you can use the YAML installation files from the previous release to perform the following downgrade procedures:
If the previous version of AMQ Streams does not support the version of Kafka you are using, you can also downgrade Kafka as long as the log message format versions appended to messages match.
If you deployed AMQ Streams using another installation method, use a supported approach to downgrade AMQ Streams. Do not use the downgrade instructions provided here. For example, if you installed AMQ Streams using the Operator Lifecycle Manager (OLM), you can downgrade by changing the deployment channel to an earlier version of AMQ Streams.
11.1. Downgrading the Cluster Operator to a previous version
If you are encountering issues with AMQ Streams, you can revert your installation.
This procedure describes how to downgrade a Cluster Operator deployment to a previous version.
Prerequisites
- An existing Cluster Operator deployment is available.
- You have downloaded the installation files for the previous version.
Procedure
-
Take note of any configuration changes made to the existing Cluster Operator resources (in the
/install/cluster-operatordirectory). Any changes will be overwritten by the previous version of the Cluster Operator. - Revert your custom resources to reflect the supported configuration options available for the version of AMQ Streams you are downgrading to.
Update the Cluster Operator.
Modify the installation files for the previous version according to the namespace the Cluster Operator is running in.
On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
-
If you modified one or more environment variables in your existing Cluster Operator
Deployment, edit theinstall/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlfile to use those environment variables.
When you have an updated configuration, deploy it along with the rest of the installation resources:
oc replace -f install/cluster-operator
Wait for the rolling updates to complete.
Get the image for the Kafka pod to ensure the downgrade was successful:
oc get pod my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'The image tag shows the new AMQ Streams version followed by the Kafka version. For example,
NEW-STRIMZI-VERSION-kafka-CURRENT-KAFKA-VERSION.
Your Cluster Operator was downgraded to the previous version.
11.2. Downgrading Kafka
Kafka version downgrades are performed by the Cluster Operator.
11.2.1. Kafka version compatibility for downgrades
Kafka downgrades are dependent on compatible current and target Kafka versions, and the state at which messages have been logged.
You cannot revert to the previous Kafka version if that version does not support any of the inter.broker.protocol.version settings which have ever been used in that cluster, or messages have been added to message logs that use a newer log.message.format.version.
The inter.broker.protocol.version determines the schemas used for persistent metadata stored by the broker, such as the schema for messages written to __consumer_offsets. If you downgrade to a version of Kafka that does not understand an inter.broker.protocol.version that has ever been previously used in the cluster the broker will encounter data it cannot understand.
If the target downgrade version of Kafka has:
-
The same
log.message.format.versionas the current version, the Cluster Operator downgrades by performing a single rolling restart of the brokers. A different
log.message.format.version, downgrading is only possible if the running cluster has always hadlog.message.format.versionset to the version used by the downgraded version. This is typically only the case if the upgrade procedure was aborted before thelog.message.format.versionwas changed. In this case, the downgrade requires:- Two rolling restarts of the brokers if the interbroker protocol of the two versions is different
- A single rolling restart if they are the same
Downgrading is not possible if the new version has ever used a log.message.format.version that is not supported by the previous version, including when the default value for log.message.format.version is used. For example, this resource can be downgraded to Kafka version 3.2.3 because the log.message.format.version has not been changed:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
# ...
kafka:
version: 3.3.1
config:
log.message.format.version: "3.2"
# ...
The downgrade would not be possible if the log.message.format.version was set at "3.3" or a value was absent, so that the parameter took the default value for a 3.3.1 broker of 3.3.
From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set.
11.2.2. Downgrading Kafka brokers and client applications
Downgrade an AMQ Streams Kafka cluster to a lower (previous) version of Kafka, such as downgrading from 3.3.1 to 3.2.3.
Prerequisites
- The Cluster Operator is up and running.
Before you downgrade the AMQ Streams Kafka cluster, check the following for the
Kafkaresource:- IMPORTANT: Compatibility of Kafka versions.
-
Kafka.spec.kafka.configdoes not contain options that are not supported by the Kafka version being downgraded to. Kafka.spec.kafka.confighas alog.message.format.versionandinter.broker.protocol.versionthat is supported by the Kafka version being downgraded to.From Kafka 3.0.0, when the
inter.broker.protocol.versionis set to3.0or higher, thelog.message.format.versionoption is ignored and doesn’t need to be set.
Procedure
Update the Kafka cluster configuration.
oc edit kafka KAFKA-CONFIGURATION-FILEChange the
Kafka.spec.kafka.versionto specify the previous version.For example, if downgrading from Kafka 3.3.1 to 3.2.3:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka spec: # ... kafka: version: 3.2.3 1 config: log.message.format.version: "3.2" 2 inter.broker.protocol.version: "3.2" 3 # ...NoteThe value of
log.message.format.versionandinter.broker.protocol.versionmust be strings to prevent them from being interpreted as floating point numbers.If the image for the Kafka version is different from the image defined in
STRIMZI_KAFKA_IMAGESfor the Cluster Operator, updateKafka.spec.kafka.image.Save and exit the editor, then wait for rolling updates to complete.
Check the update in the logs or by watching the pod state transitions:
oc logs -f CLUSTER-OPERATOR-POD-NAME | grep -E "Kafka version downgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"oc get pod -w
Check the Cluster Operator logs for an
INFOlevel message:Reconciliation #NUM(watch) Kafka(NAMESPACE/NAME): Kafka version downgrade from FROM-VERSION to TO-VERSION, phase 1 of 1 completed
Downgrade all client applications (consumers) to use the previous version of the client binaries.
The Kafka cluster and clients are now using the previous Kafka version.
If you are reverting back to a version of AMQ Streams earlier than 1.7, which uses ZooKeeper for the storage of topic metadata, delete the internal topic store topics from the Kafka cluster.
oc run kafka-admin -ti --image=registry.redhat.io/amq7/amq-streams-kafka-33-rhel8:2.3.0 --rm=true --restart=Never -- ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi-topic-operator-kstreams-topic-store-changelog --delete && ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi_store_topic --delete
Additional resources
