Chapter 7. Upgrading AMQ Streams

Procedure

1. Take note of any configuration changes made to the existing Cluster Operator resources (in the /install/cluster-operator directory). Any changes will be overwritten by the new version of the Cluster Operator.

2. Update the Cluster Operator.

   1. Modify the installation files for the new version according to the namespace the Cluster Operator is running in.

      On Linux, use:

      sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

      Copy to Clipboard

      On MacOS, use:

      sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

      Copy to Clipboard
   2. If you modified one or more environment variables in your existing Cluster Operator Deployment, edit the install/cluster-operator/050-Deployment-cluster-operator.yaml file to use those environment variables.

3. When you have an updated configuration, deploy it along with the rest of the installation resources:

   oc apply -f install/cluster-operator

   Copy to Clipboard

   Wait for the rolling updates to complete.

4. Get the image for the Kafka pod to ensure the upgrade was successful:

   oc get po my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   Copy to Clipboard

   The image tag shows the new AMQ Streams version followed by the Kafka version. For example, <New AMQ Streams version>-kafka-<Current Kafka version>.

5. Update existing resources to handle deprecated custom resource properties.

   • AMQ Streams resource upgrades

Procedure

1. Update the Kafka cluster configuration in an editor, as required:

   oc edit kafka my-cluster

   Copy to Clipboard

   1. If the log.message.format.version of the current Kafka version is the same as that of the new Kafka version, proceed to the next step.

      Otherwise, ensure that Kafka.spec.kafka.config has the log.message.format.version configured to the default for the current version.

      For example, if upgrading from Kafka 2.4.1:

      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.4.1
          config:
            log.message.format.version: "2.4"
            # ...

      Copy to Clipboard

      If the log.message.format.version is unset, set it to the current version.

      Note

      The value of log.message.format.version must be a string to prevent it from being interpreted as a floating point number.

   2. Change the Kafka.spec.kafka.version to specify the new version (leaving the log.message.format.version as the current version).

      For example, if upgrading from Kafka 2.4.1 to 2.5.0:

      apiVersion: v1alpha1
      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.5.0 

      1

      
          config:
            log.message.format.version: "2.4" 

      2

      
            # ...

      Copy to Clipboard

      1

      This is changed to the new version

      2

      This remains at the current version

   3. If the image for the Kafka version is different from the image defined in STRIMZI_KAFKA_IMAGES for the Cluster Operator, update Kafka.spec.kafka.image.

      See Section 7.1.3.1, “Kafka version and image mappings”

2. Save and exit the editor, then wait for rolling updates to complete.

   Note

   Additional rolling updates occur if the new version of Kafka has a new ZooKeeper version.

   Check the update in the logs or by watching the pod state transitions:

   oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version upgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

   Copy to Clipboard

   oc get po -w

   Copy to Clipboard

   If the current and new versions of Kafka have different interbroker protocol versions, check the Cluster Operator logs for an INFO level message:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 2 of 2 completed

   Copy to Clipboard

   Alternatively, if the current and new versions of Kafka have the same interbroker protocol version, check for:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 1 of 1 completed

   Copy to Clipboard

   The rolling updates:

   • Ensure each pod is using the broker binaries for the new version of Kafka

   • Configure the brokers to send messages using the interbroker protocol of the new version of Kafka

     Note

     Clients are still using the old version, so brokers will convert messages to the old version before sending them to the clients. To minimize this additional load, update the clients as quickly as possible.

3. Depending on your chosen strategy for upgrading clients, upgrade all client applications to use the new version of the client binaries.

   See Section 7.1.3.2, “Strategies for upgrading clients”

   Warning

   You cannot downgrade after completing this step. If you need to revert the update at this point, follow the procedure Section 7.1.4.2, “Downgrading Kafka brokers and client applications”.

   If required, set the version property for Kafka Connect and MirrorMaker as the new version of Kafka:

   1. For Kafka Connect, update KafkaConnect.spec.version
   2. For MirrorMaker, update KafkaMirrorMaker.spec.version

4. If the log.message.format.version identified in step 1 is the same as the new version proceed to the next step.

   Otherwise change the log.message.format.version in Kafka.spec.kafka.config to the default version for the new version of Kafka now being used.

   For example, if upgrading to 2.5.0:

   apiVersion: v1alpha1
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.5.0
       config:
         log.message.format.version: "2.5"
         # ...

   Copy to Clipboard

5. Wait for the Cluster Operator to update the cluster.

   The Kafka cluster and clients are now using the new Kafka version.

1. Replace the Kafka clients .jar file with the new version.
2. In the consumer configuration, append cooperative-sticky to the partition.assignment.strategy. For example, if the range strategy is set, change the configuration to range, cooperative-sticky.
3. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.
4. Reconfigure each consumer in the group by removing the earlier partition.assignment.strategy from the consumer configuration, leaving only the cooperative-sticky strategy.
5. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.

1. Replace the Kafka Streams .jar file with the new version.
2. In the Kafka Streams configuration, set the upgrade.from configuration parameter to the Kafka version you are upgrading from (for example, 2.3).
3. Restart each of the stream processors (nodes) in turn.
4. Remove the upgrade.from configuration parameter from the Kafka Streams configuration.
5. Restart each consumer in the group in turn.

Procedure

1. Update the Kafka cluster configuration in an editor, as required:

   Use oc edit:

   oc edit kafka my-cluster

   Copy to Clipboard

   1. Change the Kafka.spec.kafka.version to specify the previous version.

      For example, if downgrading from Kafka 2.5.0 to 2.4.1:

      apiVersion: v1alpha1
      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.4.1 

      1

      
          config:
            log.message.format.version: "2.4" 

      2

      
            # ...

      Copy to Clipboard

      1

      This is changed to the previous version

      2

      This is unchanged

      Note

      You must format the value of log.message.format.version as a string to prevent it from being interpreted as a floating point number.

   2. If the image for the Kafka version is different from the image defined in STRIMZI_KAFKA_IMAGES for the Cluster Operator, update Kafka.spec.kafka.image.

      See Section 7.1.3.1, “Kafka version and image mappings”

2. Save and exit the editor, then wait for rolling updates to complete.

   Check the update in the logs or by watching the pod state transitions:

   oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version downgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

   Copy to Clipboard

   oc get po -w

   Copy to Clipboard

   If the previous and current versions of Kafka have different interbroker protocol versions, check the Cluster Operator logs for an INFO level message:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 2 of 2 completed

   Copy to Clipboard

   Alternatively, if the previous and current versions of Kafka have the same interbroker protocol version, check for:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 1 of 1 completed

   Copy to Clipboard

3. Downgrade all client applications (consumers) to use the previous version of the client binaries.

   The Kafka cluster and clients are now using the previous Kafka version.