搜索
支持控制台（Console）开发人员开始试用联系人

此内容没有您所选择的语言版本。

Chapter 8. Upgrading AMQ Streams

download PDF

AMQ Streams on OpenShift can be upgraded to version 1.7 to take advantage of new features and enhancements, performance improvements, and security options.

During this upgrade, you upgrade Kafka to the latest supported version. Each Kafka release introduces new features, improvements, and bug fixes to your AMQ Streams deployment.

AMQ Streams can be downgraded to the previous version if you encounter issues with the newer version.

Released versions of AMQ Streams are listed in the Product Downloads section of the Red Hat Customer Portal.

Upgrade paths

Two upgrade paths are possible:

Incremental
Upgrading AMQ Streams from the previous minor version to version 1.7.
Multi-version

Upgrading AMQ Streams from an old version to version 1.7 within a single upgrade (skipping one or more intermediate versions).

For example, upgrading from AMQ Streams 1.5 directly to 1.7.

Kafka version support

The Kafka versions table lists the supported Kafka versions for AMQ Streams 1.7. In the table:

  • The latest Kafka version is supported for production use.
  • The previous Kafka version is supported only for the purpose of upgrading to AMQ Streams 1.7.

Identify the Kafka version to upgrade to before you begin the upgrade procedures described in this chapter.

Note

You can upgrade to a higher Kafka version as long as it is supported by your version of AMQ Streams. In some cases, you can also downgrade to a previous supported Kafka version.

Downtime and availability

If topics are configured for high availability, upgrading AMQ Streams should not cause any downtime for consumers and producers that publish and read data from those topics. Highly available topics have a replication factor of at least 3 and partitions distributed evenly among the brokers.

Upgrading AMQ Streams triggers rolling updates, where all brokers are restarted in turn, at different stages of the process. During rolling updates, not all brokers are online, so overall cluster availability is temporarily reduced. A reduction in cluster availability increases the chance that a broker failure will result in lost messages.

8.1. AMQ Streams and Kafka upgrades

Upgrading AMQ Streams is a three-stage process. To upgrade brokers and clients without downtime, you must complete the upgrade procedures in the following order:

  1. Update your Cluster Operator to a new AMQ Streams version.

    The approach you take depends on how you deployed the Cluster Operator.

    • If you deployed the Cluster Operator using the installation YAML files, perform your upgrade by modifying the Operator installation files, as described in Upgrading the Cluster Operator.

    • If you deployed the Cluster Operator from the OperatorHub, use the Operator Lifecycle Manager (OLM) to change the update channel for the AMQ Streams Operators to a new AMQ Streams version.

      Depending on your chosen upgrade strategy, after updating the channel, either:

      • An automatic upgrade is initiated

      • A manual upgrade will require approval before the installation begins

        For more information on using the OperatorHub to upgrade Operators, see Upgrading installed Operators in the OpenShift documentation.

  2. Upgrade all Kafka brokers and client applications to the latest supported Kafka version.

  3. If applicable, perform the following tasks:

    1. Update existing custom resources to handle deprecated custom resource properties.

    2. Update listeners to use the GenericKafkaListener schema

Optional: incremental cooperative rebalance upgrade

Consider upgrading consumers and Kafka Streams applications to use the incremental cooperative rebalance protocol for partition rebalances.

8.1.1. Kafka versions

Kafka’s log message format version and inter-broker protocol version specify, respectively, the log format version appended to messages and the version of the Kafka protocol used in a cluster. To ensure the correct versions are used, the upgrade process involves making configuration changes to existing Kafka brokers and code changes to client applications (consumers and producers).

The following table shows the differences between Kafka versions:

Kafka versionInterbroker protocol versionLog message format versionZooKeeper version

2.6.0

2.6

2.6

3.5.8

2.7.0

2.7

2.7

3.5.8

Inter-broker protocol version

In Kafka, the network protocol used for inter-broker communication is called the inter-broker protocol. Each version of Kafka has a compatible version of the inter-broker protocol. The minor version of the protocol typically increases to match the minor version of Kafka, as shown in the preceding table.

The inter-broker protocol version is set cluster wide in the Kafka resource. To change it, you edit the inter.broker.protocol.version property in Kafka.spec.kafka.config.

Log message format version

When a producer sends a message to a Kafka broker, the message is encoded using a specific format. The format can change between Kafka releases, so messages specify which version of the format they were encoded with. You can configure a Kafka broker to convert messages from newer format versions to a given older format version before the broker appends the message to the log.

In Kafka, there are two different methods for setting the message format version:

  • The message.format.version property is set on topics.
  • The log.message.format.version property is set on Kafka brokers.

The default value of message.format.version for a topic is defined by the log.message.format.version that is set on the Kafka broker. You can manually set the message.format.version of a topic by modifying its topic configuration.

The upgrade tasks in this section assume that the message format version is defined by the log.message.format.version.

8.1.2. Upgrading the Cluster Operator

The steps to upgrade your Cluster Operator deployment to use AMQ Streams 1.7 are described in this section.

Follow this procedure if you deployed the Cluster Operator using the installation YAML files rather than OperatorHub.

The availability of Kafka clusters managed by the Cluster Operator is not affected by the upgrade operation.

Note

Refer to the documentation supporting a specific version of AMQ Streams for information on how to upgrade to that version.

8.1.2.1. Upgrading the Cluster Operator

This procedure describes how to upgrade a Cluster Operator deployment to use AMQ Streams 1.7.

Prerequisites

Procedure

  1. Take note of any configuration changes made to the existing Cluster Operator resources (in the /install/cluster-operator directory). Any changes will be overwritten by the new version of the Cluster Operator.
  2. Update your custom resources to reflect the supported configuration options available for AMQ Streams version 1.7.

  3. Update the Cluster Operator.

    1. Modify the installation files for the new Cluster Operator version according to the namespace the Cluster Operator is running in.

      On Linux, use: 

      sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

      On MacOS, use: 

      sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
    2. If you modified one or more environment variables in your existing Cluster Operator Deployment, edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to use those environment variables.

  4. When you have an updated configuration, deploy it along with the rest of the installation resources: 

    oc replace -f install/cluster-operator

    Wait for the rolling updates to complete.

  5. If the new Operator version no longer supports the Kafka version you are upgrading from, the Cluster Operator returns a "Version not found" error message. Otherwise, no error message is returned.

    For example: 

    "Version 2.4.0 is not supported. Supported versions are: 2.6.0, 2.6.1, 2.7.0."

    • If the error message is returned, upgrade to a Kafka version that is supported by the new Cluster Operator version:

      1. Edit the Kafka custom resource.
      2. Change the spec.kafka.version property to a supported Kafka version.
    • If the error message is not returned, go to the next step. You will upgrade the Kafka version later.

  6. Get the image for the Kafka pod to ensure the upgrade was successful: 

    oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

    The image tag shows the new Operator version. For example: 

    registry.redhat.io/amq7/amq-streams-kafka-27-rhel7:{ContainerVersion}

Your Cluster Operator was upgraded to version 1.7 but the version of Kafka running in the cluster it manages is unchanged.

Following the Cluster Operator upgrade, you must perform a Kafka upgrade.

8.1.3. Upgrading Kafka

After you have upgraded your Cluster Operator to 1.7, the next step is to upgrade all Kafka brokers to the latest supported version of Kafka.

Kafka upgrades are performed by the Cluster Operator through rolling updates of the Kafka brokers.

The Cluster Operator initiates rolling updates based on the Kafka cluster configuration.

If Kafka.spec.kafka.config contains…​The Cluster Operator initiates…​

Both the inter.broker.protocol.version and the log.message.format.version.

A single rolling update. After the update, the inter.broker.protocol.version must be updated manually, followed by log.message.format.version. Changing each will trigger a further rolling update.

Either the inter.broker.protocol.version or the log.message.format.version.

Two rolling updates.

No configuration for the inter.broker.protocol.version or the log.message.format.version.

Two rolling updates.

As part of the Kafka upgrade, the Cluster Operator initiates rolling updates for ZooKeeper.

  • A single rolling update occurs even if the ZooKeeper version is unchanged.
  • Additional rolling updates occur if the new version of Kafka requires a new ZooKeeper version.

Additional resources

8.1.3.1. Kafka version and image mappings

When upgrading Kafka, consider your settings for the STRIMZI_KAFKA_IMAGES environment variable and the Kafka.spec.kafka.version property.

  • Each Kafka resource can be configured with a Kafka.spec.kafka.version.

  • The Cluster Operator’s STRIMZI_KAFKA_IMAGES environment variable provides a mapping between the Kafka version and the image to be used when that version is requested in a given Kafka resource.

    • If Kafka.spec.kafka.image is not configured, the default image for the given version is used.
    • If Kafka.spec.kafka.image is configured, the default image is overridden.
Warning

The Cluster Operator cannot validate that an image actually contains a Kafka broker of the expected version. Take care to ensure that the given image corresponds to the given Kafka version.

8.1.3.2. Upgrading Kafka brokers and client applications

This procedure describes how to upgrade a AMQ Streams Kafka cluster to the latest supported Kafka version.

Compared to your current Kafka version, the new version might support a higher log message format version or inter-broker protocol version, or both. Follow the steps to upgrade these versions, if required. For more information, see Section 8.1.1, “Kafka versions”.

You should also choose a strategy for upgrading clients. Kafka clients are upgraded in step 6 of this procedure.

Prerequisites

For the Kafka resource to be upgraded, check that:

  • The Cluster Operator, which supports both versions of Kafka, is up and running.
  • The Kafka.spec.kafka.config does not contain options that are not supported in the new Kafka version.

Procedure

  1. Update the Kafka cluster configuration: 

    oc edit kafka my-cluster

  2. If configured, ensure that Kafka.spec.kafka.config has the log.message.format.version and inter.broker.protocol.version set to the defaults for the current Kafka version.

    For example, if upgrading from Kafka version 2.6.0 to 2.7.0: 

    kind: Kafka
spec:
  # ...
  kafka:
    version: 2.6.0
    config:
      log.message.format.version: "2.6"
      inter.broker.protocol.version: "2.6"
      # ...

    If log.message.format.version and inter.broker.protocol.version are not configured, AMQ Streams automatically updates these versions to the current defaults after the update to the Kafka version in the next step.

    Note

    The value of log.message.format.version and inter.broker.protocol.version must be strings to prevent them from being interpreted as floating point numbers.

  3. Change the Kafka.spec.kafka.version to specify the new Kafka version; leave the log.message.format.version and inter.broker.protocol.version at the defaults for the current Kafka version.

    Note

    Changing the kafka.version ensures that all brokers in the cluster will be upgraded to start using the new broker binaries. During this process, some brokers are using the old binaries while others have already upgraded to the new ones. Leaving the inter.broker.protocol.version unchanged ensures that the brokers can continue to communicate with each other throughout the upgrade.

    For example, if upgrading from Kafka 2.6.0 to 2.7.0: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.7.0 1
    config:
      log.message.format.version: "2.6" 2
      inter.broker.protocol.version: "2.6" 3
      # ...
    1
    Kafka version is changed to the new version.
    2
    Message format version is unchanged.
    3
    Inter-broker protocol version is unchanged.
    Warning

    You cannot downgrade Kafka if the inter.broker.protocol.version for the new Kafka version changes. The inter-broker protocol version determines the schemas used for persistent metadata stored by the broker, including messages written to __consumer_offsets. The downgraded cluster will not understand the messages.

  4. If the image for the Kafka cluster is defined in the Kafka custom resource, in Kafka.spec.kafka.image, update the image to point to a container image with the new Kafka version.

    See Kafka version and image mappings

  5. Save and exit the editor, then wait for rolling updates to complete.

    Check the progress of the rolling updates by watching the pod state transitions: 

    oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

    The rolling updates ensure that each pod is using the broker binaries for the new version of Kafka.

  6. Depending on your chosen strategy for upgrading clients, upgrade all client applications to use the new version of the client binaries.

    If required, set the version property for Kafka Connect and MirrorMaker as the new version of Kafka:

    1. For Kafka Connect, update KafkaConnect.spec.version.
    2. For MirrorMaker, update KafkaMirrorMaker.spec.version.
    3. For MirrorMaker 2.0, update KafkaMirrorMaker2.spec.version.

  7. If configured, update the Kafka resource to use the new inter.broker.protocol.version version. Otherwise, go to step 9.

    For example, if upgrading to Kafka 2.7.0: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.7.0
    config:
      log.message.format.version: "2.6"
      inter.broker.protocol.version: "2.7"
      # ...
  8. Wait for the Cluster Operator to update the cluster.

  9. If configured, update the Kafka resource to use the new log.message.format.version version. Otherwise, go to step 10.

    For example, if upgrading to Kafka 2.7.0: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
  # ...
  kafka:
    version: 2.7.0
    config:
      log.message.format.version: "2.7"
      inter.broker.protocol.version: "2.7"
      # ...

  10. Wait for the Cluster Operator to update the cluster.

    • The Kafka cluster and clients are now using the new Kafka version.
    • The brokers are configured to send messages using the inter-broker protocol version and message format version of the new version of Kafka.

Following the Kafka upgrade, if required, you can:

8.1.4. Updating listeners to the generic listener configuration

AMQ Streams provides a GenericKafkaListener schema for the configuration of Kafka listeners in a Kafka resource.

GenericKafkaListener replaces the KafkaListeners schema, which has been removed from AMQ Streams.

With the GenericKafkaListener schema, you can configure as many listeners as required, as long as their names and ports are unique. The listeners configuration is defined as an array, but the deprecated format is also supported.

For clients inside the OpenShift cluster, you can create plain (without encryption) or tls internal listeners.

For clients outside the OpenShift cluster, you create external listeners and specify a connection mechanism, which can be nodeport, loadbalancer, ingress or route.

The KafkaListeners schema used sub-properties for plain, tls and external listeners, with fixed ports for each. At any stage in the upgrade process, you must convert listeners configured using the KafkaListeners schema into the format of the GenericKafkaListener schema.

For example, if you are currently using the following configuration in your Kafka configuration:

Old listener configuration

listeners:
  plain:
    # ...
  tls:
    # ...
  external:
    type: loadbalancer
    # ...

Convert the listeners into the new format using:

New listener configuration

listeners:
  #...
  - name: plain
    port: 9092
    type: internal
    tls: false 1
  - name: tls
    port: 9093
    type: internal
    tls: true
  - name: external
    port: 9094
    type: EXTERNAL-LISTENER-TYPE 2
    tls: true

1
The TLS property is now required for all listeners.
2
Options: ingress, loadbalancer, nodeport, route.

Make sure to use the exact names and port numbers shown.

For any additional configuration or overrides properties used with the old format, you need to update them to the new format.

Changes introduced to the listener configuration:

  • overrides is merged with the configuration section
  • dnsAnnotations has been renamed annotations
  • preferredAddressType has been renamed preferredNodePortAddressType
  • address has been renamed alternativeNames
  • loadBalancerSourceRanges and externalTrafficPolicy move to the listener configuration from the now deprecated template

For example, this configuration:

Old additional listener configuration

listeners:
  external:
    type: loadbalancer
    authentication:
      type: tls
    overrides:
      bootstrap:
        dnsAnnotations:
          #...

Changes to:

New additional listener configuration

listeners:
    #...
  - name: external
    port: 9094
    type:loadbalancer
    tls: true
    authentication:
      type: tls
    configuration:
      bootstrap:
        annotations:
          #...

Important

The name and port numbers shown in the new listener configuration must be used for backwards compatibility. Using any other values will cause renaming of the Kafka listeners and OpenShift services.

For more information on the configuration options available for each type of listener, see the GenericKafkaListener schema reference.

8.1.5. Strategies for upgrading clients

The right approach to upgrading your client applications (including Kafka Connect connectors) depends on your particular circumstances.

Consuming applications need to receive messages in a message format that they understand. You can ensure that this is the case in one of two ways:

  • By upgrading all the consumers for a topic before upgrading any of the producers.
  • By having the brokers down-convert messages to an older format.

Using broker down-conversion puts extra load on the brokers, so it is not ideal to rely on down-conversion for all topics for a prolonged period of time. For brokers to perform optimally they should not be down converting messages at all.

Broker down-conversion is configured in two ways:

  • The topic-level message.format.version configures it for a single topic.
  • The broker-level log.message.format.version is the default for topics that do not have the topic-level message.format.version configured.

Messages published to a topic in a new-version format will be visible to consumers, because brokers perform down-conversion when they receive messages from producers, not when they are sent to consumers.

There are a number of strategies you can use to upgrade your clients:

Consumers first
  1. Upgrade all the consuming applications.
  2. Change the broker-level log.message.format.version to the new version.

  3. Upgrade all the producing applications.

    This strategy is straightforward, and avoids any broker down-conversion. However, it assumes that all consumers in your organization can be upgraded in a coordinated way, and it does not work for applications that are both consumers and producers. There is also a risk that, if there is a problem with the upgraded clients, new-format messages might get added to the message log so that you cannot revert to the previous consumer version.

Per-topic consumers first

For each topic:

  1. Upgrade all the consuming applications.
  2. Change the topic-level message.format.version to the new version.

  3. Upgrade all the producing applications.

    This strategy avoids any broker down-conversion, and means you can proceed on a topic-by-topic basis. It does not work for applications that are both consumers and producers of the same topic. Again, it has the risk that, if there is a problem with the upgraded clients, new-format messages might get added to the message log.

Per-topic consumers first, with down conversion

For each topic:

  1. Change the topic-level message.format.version to the old version (or rely on the topic defaulting to the broker-level log.message.format.version).
  2. Upgrade all the consuming and producing applications.
  3. Verify that the upgraded applications function correctly.

  4. Change the topic-level message.format.version to the new version.

    This strategy requires broker down-conversion, but the load on the brokers is minimized because it is only required for a single topic (or small group of topics) at a time. It also works for applications that are both consumers and producers of the same topic. This approach ensures that the upgraded producers and consumers are working correctly before you commit to using the new message format version.

    The main drawback of this approach is that it can be complicated to manage in a cluster with many topics and applications.

Other strategies for upgrading client applications are also possible.

Note

It is also possible to apply multiple strategies. For example, for the first few applications and topics the "per-topic consumers first, with down conversion" strategy can be used. When this has proved successful another, more efficient strategy can be considered acceptable to use instead.

8.2. AMQ Streams custom resource upgrades

After you have upgraded AMQ Streams to 1.7, you must ensure that your custom resources are using API version v1beta2. You can do this any time after upgrading to 1.7, but the upgrades must be completed before the next AMQ Streams minor version update.

Important

Upgrade of the custom resources to v1beta2 must be performed after upgrading the Cluster Operator, so the Cluster Operator can understand the resources.

Note

Upgrade of the custom resources to v1beta2 prepares AMQ Streams for a move to OpenShift CRD v1, which is required for Kubernetes v1.22.

CLI upgrades to custom resources

AMQ Streams provides an API conversion tool with its release artifacts.

You can download its ZIP or TAR.GZ from AMQ Streams download site. To use the tool, extract it and use the scripts in the bin directory.

From its CLI, you can then use the tool to convert the format of your custom resources to v1beta2 in one of two ways:

After the conversion of your custom resources, you must set v1beta2 as the storage API version in your CRDs:

Manual upgrades to custom resources

Instead of using the API conversion tool to update custom resources to v1beta2, you can manually update each custom resource to use v1beta2:

Update the Kafka custom resource, including the configurations for the other components:

Update the other custom resources that apply to your deployment:

The manual procedures show the changes that are made to each custom resource. After these changes, you must use the API conversion tool to upgrade your CRDs.

8.2.1. API versioning

Custom resources are edited and controlled using APIs added to OpenShift by CRDs. Put another way, CRDs extend the Kubernetes API to allow the creation of custom resources. CRDs are themselves resources within OpenShift. They are installed in an OpenShift cluster to define the versions of API for the custom resource. Each version of the custom resource API can define its own schema for that version. OpenShift clients, including the AMQ Streams Operators, access the custom resources served by the Kubernetes API server using a URL path (API path), which includes the API version.

The introduction of v1beta2 updates the schemas of the custom resources. Older API versions are deprecated.

The v1alpha1 API version is deprecated for the following AMQ Streams custom resources:

  • Kafka
  • KafkaConnect
  • KafkaConnectS2I
  • KafkaConnector
  • KafkaMirrorMaker
  • KafkaMirrorMaker2
  • KafkaTopic
  • KafkaUser
  • KafkaBridge
  • KafkaRebalance

The v1beta1 API version is deprecated for the following AMQ Streams custom resources:

  • Kafka
  • KafkaConnect
  • KafkaConnectS2I
  • KafkaMirrorMaker
  • KafkaTopic
  • KafkaUser
Important

The v1alpha1 and v1beta1 versions will be removed in the next minor release.

Additional resources

8.2.2. Converting custom resources configuration files using the API conversion tool

This procedure describes how to use the API conversion tool to convert YAML files describing the configuration for AMQ Streams custom resources into a format applicable to v1beta2. To do so, you use the convert-file (cf) command.

The convert-file command can convert YAML files containing multiple documents. For a multi-document YAML file, all the AMQ Streams custom resources it contains are converted. Any non-AMQ Streams OpenShift resources are replicated unmodified in the converted output file.

After you have converted the YAML file, you must apply the configuration to update the custom resource in the cluster. Alternatively, if the GitOps synchronization mechanism is being used for updates on your cluster, you can use it to apply the changes. The conversion is only complete when the custom resource is updated in the OpenShift cluster.

Alternatively, you can use the convert-resource procedure to convert custom resources directly.

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.
  • The API conversion tool, which is provided with the release artifacts.
  • The tool requires Java 11.

Use the CLI help for more information on the API conversion tool, and the flags available for the convert-file command: 

bin/api-conversion.sh help
bin/api-conversion.sh help convert-file

Use bin/api-conversion.cmd for this procedure if you are using Windows.

Table 8.1. Flags for YAML file conversion
FlagDescription

-f, --file=NAME-OF-YAML-FILE

Specifies the YAML file for the AMQ Streams custom resource being converted

-o, --output=NAME-OF-CONVERTED-YAML-FILE

Creates an output YAML file for the converted custom resource

--in-place

Updates the original source file with the converted YAML

Procedure

  1. Run the API conversion tool with the convert-file command and appropriate flags.

    Example 1, converts a YAML file and displays the output, though the file does not change: 

    bin/api-conversion.sh convert-file --file input.yaml

    Example 2, converts a YAML file, and writes the changes into the original source file: 

    bin/api-conversion.sh convert-file --file input.yaml --in-place

    Example 3, converts a YAML file, and writes the changes into a new output file: 

    bin/api-conversion.sh convert-file --file input.yaml --output output.yaml

  2. Update the custom resources using the converted configuration file. 

    oc apply -f CONVERTED-CONFIG-FILE

  3. Verify that the custom resources have been converted. 

    oc get KIND CUSTOM-RESOURCE-NAME -o yaml

8.2.3. Converting custom resources directly using the API conversion tool

This procedure describes how to use the API conversion tool to convert AMQ Streams custom resources directly in the OpenShift cluster into a format applicable to v1beta2. To do so, you use the convert-resource (cr) command. The command uses Kubernetes APIs to make the conversions.

You can specify one or more of types of AMQ Streams custom resources, based on the kind property, or you can convert all types. You can also target a specific namespace or all namespaces for conversion. When targeting a namespace, you can convert all custom resources in that namespace, or convert a single custom resource by specifying its name and kind.

Alternatively, you can use the convert-file procedure to convert and apply the YAML files describing the custom resources.

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.
  • The API conversion tool, which is provided with the release artifacts.
  • The tool requires Java 11 (OpenJDK).

  • The steps require a user admin account with RBAC permission to:

    • Get the AMQ Streams custom resources being converted using the --name option
    • List the AMQ Streams custom resources being converted without using the --name option
    • Replace the AMQ Streams custom resources being converted

Use the CLI help for more information on the API conversion tool, and the flags available for the convert-resource command: 

bin/api-conversion.sh help
bin/api-conversion.sh help convert-resource

Use bin/api-conversion.cmd for this procedure if you are using Windows.

Table 8.2. Flags for converting custom resources
FlagDescription

-k, --kind

Specifies the kinds of custom resources to be converted, or converts all resources if not specified

-a, --all-namespaces

Converts custom resources in all namespaces

-n, --namespace

Specifies an OpenShift namespace or OpenShift project, or uses the current namespace if not specified

--name

If --namespace and a single custom resource --kind is used, specifies the name of the custom resource being converted

Procedure

  1. Run the API conversion tool with the convert-resource command and appropriate flags.

    Example 1, converts all AMQ Streams resources in current namespace: 

    bin/api-conversion.sh convert-resource

    Example 2, converts all AMQ Streams resources in all namespaces: 

    bin/api-conversion.sh convert-resource --all-namespaces

    Example 3, converts all AMQ Streams resources in the my-kafka namespace: 

    bin/api-conversion.sh convert-resource --namespace my-kafka

    Example 4, converts only Kafka resources in all namespaces: 

    bin/api-conversion.sh convert-resource --all-namespaces --kind Kafka

    Example 5, converts Kafka and Kafka Connect resources in all namespaces: 

    bin/api-conversion.sh convert-resource --all-namespaces --kind Kafka --kind KafkaConnect

    Example 6, converts a Kafka custom resource named my-cluster in the my-kafka namespace: 

    bin/api-conversion.sh convert-resource --kind Kafka --namespace my-kafka --name my-cluster

  2. Verify that the custom resources have been converted. 

    oc get KIND CUSTOM-RESOURCE-NAME -o yaml

8.2.4. Upgrading CRDs to v1beta2 using the API conversion tool

This procedure describes how to use the API conversion tool to convert the CRDs that define the schemas used to instantiate and manage AMQ Streams-specific resources in a format applicable to v1beta2. To do so, you use the crd-upgrade command.

Perform this procedure after converting all AMQ Streams custom resources in the whole OpenShift cluster to v1beta2. If you upgrade your CRDs first, and then convert your custom resources, you will need to run this command again.

The command updates spec.versions in the CRDs to declare v1beta2 as the storage API version. The command also updates custom resources so they are stored under v1beta2. New custom resource instances are created from the specification of the storage API version, so only one API version is ever marked as the storage version.

When you have upgraded the CRDs to use v1beta2 as the storage version, you should only use v1beta2 properties in your custom resources.

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.
  • The API conversion tool, which is provided with the release artifacts.
  • The tool requires Java 11 (OpenJDK).
  • Custom resources have been converted to v1beta2.

  • The steps require a user admin account with RBAC permission to:

    • List the AMQ Streams custom resources in all namespaces
    • Replace the AMQ Streams custom resources being converted
    • Update CRDs
    • Replace the status of the CRDs

Use the CLI help for more information on the API conversion tool: 

bin/api-conversion.sh help

Use bin/api-conversion.cmd for this procedure if you are using Windows.

Procedure

  1. If you have not done so, convert your custom resources to use v1beta2.

    You can use the API conversion tool to do this in one of two ways:

  2. Run the API conversion tool with the crd-upgrade command. 

    bin/api-conversion.sh crd-upgrade

  3. Verify that the CRDs have been upgraded so that v1beta2 is the storage version.

    For example, for the Kafka topic CRD: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: CustomResourceDefinition
metadata:
  name: kafkatopics.kafka.strimzi.io
  #...
spec:
  group: kafka.strimzi.io
  #...
  versions:
  - name: v1beta2
    served: true
    storage: true
    #...
status:
  #...
  storedVersions:
  - v1beta2

8.2.5. Upgrading Kafka resources to support v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each Kafka custom resource in your deployment.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER

  2. If you have not already done so, update .spec.kafka.listener to the new generic listener format, as described in Section 8.1.4, “Updating listeners to the generic listener configuration”.

    Warning

    The old listener format is not supported in API version v1beta2.

  3. If present, move affinity from .spec.kafka.affinity to .spec.kafka.template.pod.affinity.
  4. If present, move tolerations from .spec.kafka.tolerations to .spec.kafka.template.pod.tolerations.
  5. If present, remove .spec.kafka.template.tlsSidecarContainer.
  6. If present, remove .spec.kafka.tlsSidecarContainer.

  7. If either of the following policy configurations exist:

    • .spec.kafka.template.externalBootstrapService.externalTrafficPolicy

    • .spec.kafka.template.perPodService.externalTrafficPolicy

      1. Move the configuration to .spec.kafka.listeners[].configuration.externalTrafficPolicy, for both type: loadbalancer and type: nodeport listeners.
      2. Remove .spec.kafka.template.externalBootstrapService.externalTrafficPolicy or .spec.kafka.template.perPodService.externalTrafficPolicy.

  8. If either of the following loadbalancer listener configurations exist:

    • .spec.kafka.template.externalBootstrapService.loadBalancerSourceRanges

    • .spec.kafka.template.perPodService.loadBalancerSourceRanges

      1. Move the configuration to .spec.kafka.listeners[].configuration.loadBalancerSourceRanges, for type: loadbalancer listeners.
      2. Remove .spec.kafka.template.externalBootstrapService.loadBalancerSourceRanges or .spec.kafka.template.perPodService.loadBalancerSourceRanges.

  9. If type: external logging is configured in .spec.kafka.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  10. If the .spec.kafka.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.kafka.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.kafka.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-metrics
      key: kafka-metrics-config.yaml
    3. Delete the old .spec.kafka.metrics field.
  11. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

What to do next

For each Kafka custom resource, upgrade the configurations for ZooKeeper, Topic Operator, Entity Operator, and Cruise Control (if deployed) to support version v1beta2. This is described in the following procedures.

When all Kafka configurations are updated to support v1beta2, you can upgrade the Kafka custom resource to v1beta2.

8.2.6. Upgrading ZooKeeper to support v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each Kafka custom resource in your deployment.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER
  2. If present, move affinity from .spec.zookeeper.affinity to .spec.zookeeper.template.pod.affinity.
  3. If present, move tolerations from .spec.zookeeper.tolerations to .spec.zookeeper.template.pod.tolerations.
  4. If present, remove .spec.zookeeper.template.tlsSidecarContainer.
  5. If present, remove .spec.zookeeper.tlsSidecarContainer.

  6. If type: external logging is configured in .spec.kafka.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  7. If the .spec.zookeeper.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.zookeeper.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  zookeeper-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.zookeeper.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-metrics
      key: zookeeper-metrics-config.yaml
    3. Delete the old .spec.zookeeper.metrics field.
  8. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.7. Upgrading the Topic Operator to support v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each Kafka custom resource in your deployment.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER

  2. If Kafka.spec.topicOperator is used:

    1. Move affinity from .spec.topicOperator.affinity to .spec.entityOperator.template.pod.affinity.
    2. Move tolerations from .spec.topicOperator.tolerations to .spec.entityOperator.template.pod.tolerations.
    3. Move .spec.topicOperator.tlsSidecar to .spec.entityOperator.tlsSidecar.
    4. After moving affinity, tolerations, and tlsSidecar, move the remaining configuration in .spec.topicOperator to .spec.entityOperator.topicOperator.

  3. If type: external logging is configured in .spec.topicOperator.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j2.properties
    Note

    You can also complete this step as part of the Entity Operator upgrade.

  4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.8. Upgrading the Entity Operator to support v1beta2

Prerequisites

Procedure

Perform the following steps for each Kafka custom resource in your deployment.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER
  2. Move affinity from .spec.entityOperator.affinity to .spec.entityOperator.template.pod.affinity.
  3. Move tolerations from .spec.entityOperator.tolerations to .spec.entityOperator.template.pod.tolerations.

  4. If type: external logging is configured in .spec.entityOperator.userOperator.logging or .spec.entityOperator.topicOperator.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j2.properties
  5. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.9. Upgrading Cruise Control to support v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.
  • Cruise Control is configured and deployed. See Deploying Cruise Control in the Using AMQ Streams on OpenShift guide.

Procedure

Perform the following steps for each Kafka.spec.cruiseControl configuration in your Kafka cluster.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER

  2. If type: external logging is configured in .spec.cruiseControl.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j2.properties

  3. If the .spec.cruiseControl.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.cruiseControl.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  cruise-control-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.cruiseControl.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-metrics
      key: cruise-control-metrics-config.yaml
    3. Delete the old .spec.cruiseControl.metrics field.
  4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.10. Upgrading the API version of Kafka resources to v1beta2

Prerequisites

Procedure

Perform the following steps for each Kafka custom resource in your deployment.

  1. Update the Kafka custom resource in an editor. 

    oc edit kafka KAFKA-CLUSTER

  2. Update the apiVersion of the Kafka custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.11. Upgrading Kafka Connect resources to v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each KafkaConnect custom resource in your deployment.

  1. Update the KafkaConnect custom resource in an editor. 

    oc edit kafkaconnect KAFKA-CONNECT-CLUSTER

  2. If present, move: 

    KafkaConnect.spec.affinity
    KafkaConnect.spec.tolerations

    to: 

    KafkaConnect.spec.template.pod.affinity
    KafkaConnect.spec.template.pod.tolerations

    For example, move: 

    spec:
  # ...
  affinity:
    # ...
  tolerations:
    # ...

    to: 

    spec:
  # ...
  template:
    pod:
      affinity:
        # ...
      tolerations:
        # ...

  3. If type: external logging is configured in .spec.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  4. If the .spec.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-connect-metrics
  labels:
    app: strimzi
data:
  connect-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-connect-metrics
      key: connect-metrics-config.yaml
    3. Delete the old .spec.metrics field.

  5. Update the apiVersion of the KafkaConnect custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.12. Upgrading Kafka Connect S2I resources to v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each KafkaConnectS2I custom resource in your deployment.

  1. Update the KafkaConnectS2I custom resource in an editor. 

    oc edit kafkaconnects2i S2I-CLUSTER

  2. If present, move: 

    KafkaConnectS2I.spec.affinity
    KafkaConnectS2I.spec.tolerations

    to: 

    KafkaConnectS2I.spec.template.pod.affinity
    KafkaConnectS2I.spec.template.pod.tolerations

    For example, move: 

    spec:
  # ...
  affinity:
    # ...
  tolerations:
    # ...

    to: 

    spec:
  # ...
  template:
    pod:
      affinity:
        # ...
      tolerations:
        # ...

  3. If type: external logging is configured in .spec.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  4. If the .spec.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-connect-s2i-metrics
  labels:
    app: strimzi
data:
  connect-s2i-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-connect-s2i-metrics
      key: connect-s2i-metrics-config.yaml
    3. Delete the old .spec.metrics field

  5. Update the apiVersion of the KafkaConnectS2I custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.13. Upgrading Kafka MirrorMaker resources to v1beta2

Prerequisites

Procedure

Perform the following steps for each KafkaMirrorMaker custom resource in your deployment.

  1. Update the KafkaMirrorMaker custom resource in an editor. 

    oc edit kafkamirrormaker MIRROR-MAKER

  2. If present, move: 

    KafkaMirrorMaker.spec.affinity
    KafkaMirrorMaker.spec.tolerations

    to: 

    KafkaMirrorMaker.spec.template.pod.affinity
    KafkaMirrorMaker.spec.template.pod.tolerations

    For example, move: 

    spec:
  # ...
  affinity:
    # ...
  tolerations:
    # ...

    to: 

    spec:
  # ...
  template:
    pod:
      affinity:
        # ...
      tolerations:
        # ...

  3. If type: external logging is configured in .spec.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  4. If the .spec.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-mm-metrics
  labels:
    app: strimzi
data:
  mm-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-mm-metrics
      key: mm-metrics-config.yaml
    3. Delete the old .spec.metrics field.

  5. Update the apiVersion of the KafkaMirrorMaker custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.14. Upgrading Kafka MirrorMaker 2.0 resources to v1beta2

Prerequisites

Procedure

Perform the following steps for each KafkaMirrorMaker2 custom resource in your deployment.

  1. Update the KafkaMirrorMaker2 custom resource in an editor. 

    oc edit kafkamirrormaker2 MIRROR-MAKER-2
  2. If present, move affinity from .spec.affinity to .spec.template.pod.affinity.
  3. If present, move tolerations from .spec.tolerations to .spec.template.pod.tolerations.

  4. If type: external logging is configured in .spec.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j.properties

  5. If the .spec.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field. 

      kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-mm2-metrics
  labels:
    app: strimzi
data:
  mm2-metrics-config.yaml: |
      <YAML>

    2. Add a .spec.metricsConfig property that points to the ConfigMap and key: 

      metricsConfig:
  type: jmxPrometheusExporter
  valueFrom:
    configMapKeyRef:
      name: kafka-mm2-metrics
      key: mm2-metrics-config.yaml
    3. Delete the old .spec.metrics field.

  6. Update the apiVersion of the KafkaMirrorMaker2 custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1alpha1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  7. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.15. Upgrading Kafka Bridge resources to v1beta2

Prerequisites

Procedure

Perform the following steps for each KafkaBridge resource in your deployment.

  1. Update the KafkaBridge custom resource in an editor. 

    oc edit kafkabridge KAFKA-BRIDGE

  2. If type: external logging is configured in KafkaBridge.spec.logging:

    Replace the name of the ConfigMap containing the logging configuration: 

    logging:
  type: external
  name: my-config-map

    With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored: 

    logging:
  type: external
  valueFrom:
    configMapKeyRef:
      name: my-config-map
      key: log4j2.properties

  3. Update the apiVersion of the KafkaBridge custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1alpha1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.16. Upgrading Kafka User resources to v1beta2

Prerequisites

  • A User Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each KafkaUser custom resource in your deployment.

  1. Update the KafkaUser custom resource in an editor. 

    oc edit kafkauser KAFKA-USER

  2. Update the apiVersion of the KafkaUser custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.17. Upgrading Kafka Topic resources to v1beta2

Prerequisites

  • A Topic Operator supporting the v1beta2 API version is up and running.

Procedure

Perform the following steps for each KafkaTopic custom resource in your deployment.

  1. Update the KafkaTopic custom resource in an editor. 

    oc edit kafkatopic KAFKA-TOPIC

  2. Update the apiVersion of the KafkaTopic custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1beta1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.18. Upgrading Kafka Connector resources to v1beta2

Prerequisites

Procedure

Perform the following steps for each KafkaConnector custom resource in your deployment.

  1. Update the KafkaConnector custom resource in an editor. 

    oc edit kafkaconnector KAFKA-CONNECTOR

  2. Update the apiVersion of the KafkaConnector custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1alpha1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.2.19. Upgrading Kafka Rebalance resources to v1beta2

Prerequisites

  • A Cluster Operator supporting the v1beta2 API version is up and running.
  • Cruise Control is configured and deployed. See Deploying Cruise Control in the Using AMQ Streams on OpenShift guide.

Procedure

Perform the following steps for each KafkaRebalance custom resource in your deployment.

  1. Update the KafkaRebalance custom resource in an editor. 

    oc edit kafkarebalance KAFKA-REBALANCE

  2. Update the apiVersion of the KafkaRebalance custom resource to v1beta2:

    Replace: 

    apiVersion: kafka.strimzi.io/v1alpha1

    with: 

    apiVersion: kafka.strimzi.io/v1beta2
  3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

8.3. Upgrading consumers to cooperative rebalancing

You can upgrade Kafka consumers and Kafka Streams applications to use the incremental cooperative rebalance protocol for partition rebalances instead of the default eager rebalance protocol. The new protocol was added in Kafka 2.4.0.

Consumers keep their partition assignments in a cooperative rebalance and only revoke them at the end of the process, if needed to achieve a balanced cluster. This reduces the unavailability of the consumer group or Kafka Streams application.

Note

Upgrading to the incremental cooperative rebalance protocol is optional. The eager rebalance protocol is still supported.

Prerequisites

Procedure

To upgrade a Kafka consumer to use the incremental cooperative rebalance protocol:

  1. Replace the Kafka clients .jar file with the new version.
  2. In the consumer configuration, append cooperative-sticky to the partition.assignment.strategy. For example, if the range strategy is set, change the configuration to range, cooperative-sticky.
  3. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.
  4. Reconfigure each consumer in the group by removing the earlier partition.assignment.strategy from the consumer configuration, leaving only the cooperative-sticky strategy.
  5. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.

To upgrade a Kafka Streams application to use the incremental cooperative rebalance protocol:

  1. Replace the Kafka Streams .jar file with the new version.
  2. In the Kafka Streams configuration, set the upgrade.from configuration parameter to the Kafka version you are upgrading from (for example, 2.3).
  3. Restart each of the stream processors (nodes) in turn.
  4. Remove the upgrade.from configuration parameter from the Kafka Streams configuration.
  5. Restart each consumer in the group in turn.

Additional resources

Red Hat logoGithubRedditYoutubeTwitter

学习

尝试、购买和销售

社区

关于红帽文档

通过我们的产品和服务，以及可以信赖的内容，帮助红帽用户创新并实现他们的目标。

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情，请参阅红帽博客.

關於紅帽

我们提供强化的解决方案，使企业能够更轻松地跨平台和环境（从核心数据中心到网络边缘）工作。

© 2024 Red Hat, Inc.