Chapter 25. Upgrading Streams for Apache Kafka

1. Upgrade Streams for Apache Kafka to version 1.7 following the standard sequence.
2. Convert Streams for Apache Kafka custom resources to v1beta2 using the API conversion tool provided with Streams for Apache Kafka.

3. Do one of the following:

   • Upgrade Streams for Apache Kafka to a version between 1.8 and 0.26 (where the ControlPlaneListener feature gate is disabled by default).
   • Upgrade Streams for Apache Kafka to a version between 2.0 and 0.31 (where the ControlPlaneListener feature gate is enabled by default) with the ControlPlaneListener feature gate disabled.
4. Enable the ControlPlaneListener feature gate.
5. Upgrade to Streams for Apache Kafka 2.7 following the standard sequence.

Procedure

1. Take note of any configuration changes made to the existing Cluster Operator resources (in the /install/cluster-operator directory). Any changes will be overwritten by the new version of the Cluster Operator.
2. Update your custom resources to reflect the supported configuration options available for Streams for Apache Kafka version 2.7.

3. Update the Cluster Operator.

   1. Modify the installation files for the new Cluster Operator version according to the namespace the Cluster Operator is running in.

      On Linux, use:

      sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

      On MacOS, use:

      sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   2. If you modified one or more environment variables in your existing Cluster Operator Deployment, edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to use those environment variables.

4. When you have an updated configuration, deploy it along with the rest of the installation resources:

   oc replace -f install/cluster-operator

   Wait for the rolling updates to complete.

5. If the new Operator version no longer supports the Kafka version you are upgrading from, the Cluster Operator returns an error message to say the version is not supported. Otherwise, no error message is returned.

   • If the error message is returned, upgrade to a Kafka version that is supported by the new Cluster Operator version:

     1. Edit the Kafka custom resource.
     2. Change the spec.kafka.version property to a supported Kafka version.
   • If the error message is not returned, go to the next step. You will upgrade the Kafka version later.

6. Get the image for the Kafka pod to ensure the upgrade was successful:

   oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   The image tag shows the new Streams for Apache Kafka version followed by the Kafka version:

   registry.redhat.io/amq-streams/strimzi-kafka-37-rhel9:2.7.0

   You can also check the upgrade has completed successfully from the status of the Kafka resource.

1. Upgrade to Streams for Apache Kafka 1.7.
2. Download the Red Hat Streams for Apache Kafka API Conversion Tool provided with Streams for Apache Kafka 1.8 from the Streams for Apache Kafka software downloads page.

3. Convert custom resources and CRDs to v1beta2.

   For more information, see the Streams for Apache Kafka 1.7 upgrade documentation.

4. In the OperatorHub, delete version 1.7 of the Streams for Apache Kafka Operator.

5. If it also exists, delete version 2.7 of the Streams for Apache Kafka Operator.

   If it does not exist, go to the next step.

   If the Approval Strategy for the Streams for Apache Kafka Operator was set to Automatic, version 2.7 of the operator might already exist in your cluster. If you did not convert custom resources and CRDs to the v1beta2 API version before release, the operator-managed custom resources and CRDs will be using the old API version. As a result, the 2.7 Operator is stuck in Pending status. In this situation, you need to delete version 2.7 of the Streams for Apache Kafka Operator as well as version 1.7.

   If you delete both operators, reconciliations are paused until the new operator version is installed. Follow the next steps immediately so that any changes to custom resources are not delayed.

6. In the OperatorHub, do one of the following:

   • Upgrade to version 1.8 of the Streams for Apache Kafka Operator (where the ControlPlaneListener feature gate is disabled by default).
   • Upgrade to version 2.0 or 2.2 of the Streams for Apache Kafka Operator (where the ControlPlaneListener feature gate is enabled by default) with the ControlPlaneListener feature gate disabled.

7. Upgrade to version 2.7 of the Streams for Apache Kafka Operator immediately.

   The installed 2.7 operator begins to watch the cluster and performs rolling updates. You might notice a temporary decrease in cluster performance during this process.

Procedure

1. Update the Kafka cluster configuration:

   oc edit kafka <kafka_configuration_file>

2. If configured, check that the inter.broker.protocol.version and log.message.format.version properties are set to the current version.

   For example, the current version is 3.6 if upgrading from Kafka version 3.6.0 to 3.7.0:

   kind: Kafka
   spec:
     # ...
     kafka:
       version: 3.6.0
       config:
         log.message.format.version: "3.6"
         inter.broker.protocol.version: "3.6"
         # ...

   If log.message.format.version and inter.broker.protocol.version are not configured, Streams for Apache Kafka automatically updates these versions to the current defaults after the update to the Kafka version in the next step.

   Note

   The value of log.message.format.version and inter.broker.protocol.version must be strings to prevent them from being interpreted as floating point numbers.

3. Change the Kafka.spec.kafka.version to specify the new Kafka version; leave the log.message.format.version and inter.broker.protocol.version at the defaults for the current Kafka version.

   Note

   Changing the kafka.version ensures that all brokers in the cluster are upgraded to start using the new broker binaries. During this process, some brokers are using the old binaries while others have already upgraded to the new ones. Leaving the inter.broker.protocol.version unchanged at the current setting ensures that the brokers can continue to communicate with each other throughout the upgrade.

   For example, if upgrading from Kafka 3.6.0 to 3.7.0:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 3.7.0 1
       config:
         log.message.format.version: "3.6" 2
         inter.broker.protocol.version: "3.6" 3
         # ...

   1

   Kafka version is changed to the new version.

   2

   Message format version is unchanged.

   3

   Inter-broker protocol version is unchanged.

   Warning

   You cannot downgrade Kafka if the inter.broker.protocol.version for the new Kafka version changes. The inter-broker protocol version determines the schemas used for persistent metadata stored by the broker, including messages written to __consumer_offsets. The downgraded cluster will not understand the messages.

4. If the image for the Kafka cluster is defined in Kafka.spec.kafka.image of the Kafka custom resource, update the image to point to a container image with the new Kafka version.

   See Kafka version and image mappings

5. Save and exit the editor, then wait for rolling updates to complete.

   Check the progress of the rolling updates by watching the pod state transitions:

   oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   The rolling updates ensure that each pod is using the broker binaries for the new version of Kafka.

6. Depending on your chosen strategy for upgrading clients, upgrade all client applications to use the new version of the client binaries.

   If required, set the version property for Kafka Connect and MirrorMaker as the new version of Kafka:

   1. For Kafka Connect, update KafkaConnect.spec.version.
   2. For MirrorMaker, update KafkaMirrorMaker.spec.version.

   3. For MirrorMaker 2, update KafkaMirrorMaker2.spec.version.

      Note

      If you are using custom images that are built manually, you must rebuild those images to ensure that they are up-to-date with the latest Streams for Apache Kafka base image. For example, if you created a container image from the base Kafka Connect image, update the Dockerfile to point to the latest base image and build configuration.

7. Verify that the upgraded client applications work correctly with the new Kafka brokers.

8. If configured, update the Kafka resource to use the new inter.broker.protocol.version version. Otherwise, go to step 9.

   For example, if upgrading to Kafka 3.7.0:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 3.7.0
       config:
         log.message.format.version: "3.6"
         inter.broker.protocol.version: "3.7"
         # ...

9. Wait for the Cluster Operator to update the cluster.

10. If configured, update the Kafka resource to use the new log.message.format.version version. Otherwise, go to step 10.

    For example, if upgrading to Kafka 3.7.0:

    apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    spec:
      # ...
      kafka:
        version: 3.7.0
        config:
          log.message.format.version: "3.7"
          inter.broker.protocol.version: "3.7"
          # ...

    Important

    From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set.

11. Wait for the Cluster Operator to update the cluster.

    You can check the upgrade has completed successfully from the status of the Kafka resource.