Release Notes for Streams for Apache Kafka 2.9 on OpenShift

You can scale a Kafka cluster by adjusting the number of brokers using the spec.replicas property in the Kafka or KafkaNodePool custom resource used in deployment.

Enable auto-rebalancing to automatically redistribute topic partitions when scaling a cluster up or down. Auto-rebalancing requires a Cruise Control deployment, a rebalancing template for the operation, and autorebalance configuration in the Kafka resource referencing the template. When enabled, auto-rebalancing rebalances clusters that have been scaled up or down without further intervention.

For more information, see Triggering auto-rebalances when scaling clusters.

If you are using JBOD storage and have Cruise Control installed with Streams for Apache Kafka, you can now reassign partitions between the JBOD disks used for storage on the same broker. This capability also allows you to remove JBOD disks without data loss.

You configure a KafkaRebalance resource in remove-disks mode and specify a list of broker IDs with corresponding volume IDs for partition reassignment. Cruise Control generates an optimization proposal based on the configuration and reassigns the partitions when approved manually or automatically.

For more information, see Using Cruise Control to reassign partitions on JBOD disks.

A new mechanism allows connector offsets to be managed through KafkaConnect and KafkaMirrorMaker2 resources. It’s now possible to list, alter, and reset offsets.

For more information, see Configuring Kafka Connect connectors.

Hostnames and advertised hostnames for individual brokers can be specified using the host and advertisedHost properties. This release introduces support for using variables, such as {nodeId} or {nodePodName}, in the following templates:

By using templates, you no longer need to configure each broker individually. Streams for Apache Kafka automatically replaces the template variables with the corresponding values for each broker.

For more information, see Overriding advertised addresses for brokers and Specifying listener types.

Environment variables for any container deployed by Streams for Apache Kafka may now be based on values specified in a Secret or ConfigMap. This replaces the requirement to use the ExternalConfiguration schema for Kafka Connect and MirrorMaker 2 containers, which is now deprecated.

Values are referenced in the container configuration using the valueFrom.secretKeyRef or valueFrom.configMapKeyRef properties.

For more information, see Loading configuration values from environment variables.

Strimzi generates pod disruption budget resources for Kafka, Kafka Connect worker, MirrorMaker2 worker, and Kafka Bridge worker nodes.

If you want to use custom pod disruption budget resources, you can now set the STRIMZI_POD_DISRUPTION_BUDGET_GENERATION environment variable to false in the Cluster Operator configuration.

For more information, see Disabling pod disruption budget generation.

In order to support the CSI volumes, a new property named csi has been added to the AdditionalVolume schema. This property maps to the Kubernetes API CSIVolumeSource structure, allowing CSI volumes to be defined in container template fields.

For more information, see AdditionalVolume schema reference and Additional volumes.

Use the new admin/topics endpoint of the Kafka Bridge API to create topics. You can specify the topic name, partition count, replication factor in the request body.

For more information, see Securing connections from clients.

When configuring proxies, you can now use trust properties to configure virtual clusters to use TLS client authentication.

For more information, see Securing connections from clients.

The console (user interface) for Streams for Apache Kafka moves to GA. It is designed to seamlessly integrate with your Streams for Apache Kafka deployment, providing a centralized hub for monitoring and managing Kafka clusters. Deploy the console and connect it to Kafka clusters managed by Streams for Apache Kafka.

Gain insights into each connected cluster through dedicated console pages covering brokers, topics, and consumer groups. View essential information, such as the status of a Kafka cluster, before looking into specific details about brokers, topics, or connected consumer groups.

For more information, see the Streams for Apache Kafka Console guide.

You can now reset consumer offsets of a specific consumer group from the Consumer Groups page.

For more information, see Resetting consumer offsets.

When you configure KafkaRebalance resources to generate optimization proposals on a cluster, you can manage the proposals and any resulting rebalances from the Brokers page.

For more information, see Managing rebalances.

Pause and resume cluster reconciliations from the Cluster overview page. While paused, any changes to the cluster configuration using the Kafka custom resource are ignored until reconciliation is resumed.

For more information, see Pausing reconciliation of clusters.

The console now supports configuration of authorization rules in the console deployment configuration. Enable secure console connections to Kafka clusters using an OpenID Connect (OIDC) provider, such as Red Hat build of Keycloak. The configuration can be set up for all clusters or at the cluster level.

For more information, see Deploying the console.

The Strimzi Quotas plugin moves to GA (General Availability). Use the plugin properties to set throughput and storage limits on brokers in your Kafka cluster configuration.

For more information, see Setting limits on brokers using the Kafka Static Quota plugin.

When finalizers are enabled (default), the Topic Operator no longer restores them on unmanaged KafkaTopic resources if removed. This behavior aligns with paused topics, where finalizers are also not restored.

The technology preview of the ContinueReconciliationOnManualRollingUpdateFailure feature gate moves to beta stage and is enabled by default. If required, ContinueReconciliationOnManualRollingUpdateFailure can be disabled in the feature gates configuration in the Cluster Operator.

Pods are now rolled only when the cluster CA key is replaced, not when the clients CA key is replaced, which is used solely for trust. Consequently, the restart event reason ClientCaCertKeyReplaced has been removed, and either CaCertRenewed or CaCertHasOldGeneration is now used as the event reason.

Rolling updates for new CA certificate generations now resume from where they left off after an interruption, instead of restarting the process and rolling all pods again.

See the Streams for Apache Kafka Custom Resource API Reference.

Support for Java 11 is deprecated from Kafka 3.7.0 and Streams for Apache Kafka 2.7. Java 11 will be unsupported for all Streams for Apache Kafka components, including clients, in release 3.0.

Streams for Apache Kafka supports Java 17. Use Java 17 when developing new applications. Plan to migrate any applications that currently use Java 11 to 17.

If you want to continue using Java 11 for the time being, Streams for Apache Kafka 2.5 provides Long Term Support (LTS). For information on the LTS terms and dates, see the Streams for Apache Kafka LTS Support Policy.

The storage overrides (*.storage.overrides) for configuring per-broker storage are deprecated and will be removed in Streams for Apache Kafka 3.0. If you are using the storage overrides, migrate to KafkaNodePool resources and use multiple node pools with a different storage class each.

For more information, see PersistentClaimStorage schema reference.

You can use configuration providers to load configuration data from external sources for all Kafka components, including producers and consumers.

Previously, you could enable the io.strimzi.kafka.EnvVarConfigProvider environment variable configuration provider using the config.providers properties in the spec configuration of a component. However, this provider is now deprecated and will be removed in Streams for Apache Kafka 3.0. Therefore, it is recommended to update your implementation to use Kafka’s own environment variable configuration provider (org.apache.kafka.common.config.provider.EnvVarConfigProvider) to provide configuration properties as environment variables.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...
  config:
    # ...
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  # ...

Identity replication policy is a feature used with MirrorMaker 2 to override the automatic renaming of remote topics. Instead of prepending the name with the source cluster’s name, the topic retains its original name. This setting is particularly useful for active/passive backups and data migration scenarios.

To implement an identity replication policy, you must specify a replication policy class (replication.policy.class) in the MirrorMaker 2 configuration. Previously, you could specify the io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy class included with the Streams for Apache Kafka mirror-maker-2-extensions component. However, this component is now deprecated and will be removed in Streams for Apache Kafka 3.0. Therefore, it is recommended to update your implementation to use Kafka’s own replication policy class (org.apache.kafka.connect.mirror.IdentityReplicationPolicy).

For more information, see Configuring Kafka MirrorMaker 2.

Kafka MirrorMaker replicates data between two or more active Kafka clusters, within or across data centers. Kafka MirrorMaker 1 was deprecated in Kafka 3.0 and will be removed in Streams for Apache Kafka 3.0, including the KafkaMirrorMaker custom resource, and Kafka 4.0.0. MirrorMaker 2 will be the only version available. MirrorMaker 2 is based on the Kafka Connect framework, connectors managing the transfer of data between clusters. To avoid disruptions, please transition to MirrorMaker 2 before support ends.

If you’re using MirrorMaker 1, you can replicate its functionality in MirrorMaker 2 by using the KafkaMirrorMaker2 custom resource with the IdentityReplicationPolicy class.. By default, MirrorMaker 2 renames topics replicated to a target cluster, but IdentityReplicationPolicy preserves the original topic names, enabling the same active/passive unidirectional replication as MirrorMaker 1.

For more information, see Configuring Kafka MirrorMaker 2.

Support for OpenAPI v2 is now deprecated and will be removed in Streams for Apache Kafka 3.0. OpenAPI v3 is now supported. Plan to move to using OpenAPI v3.

During the transition to using OpenAPI v2, the /openapi endpoint returns the OpenAPI v2 specification using an additional /openapi/v2 endpoint. A new /openapi/v3 endpoint returns the OpenAPI v3 specification.

The following Kafka Bridge span attributes are deprecated with replacements shown where applicable:

Kafka Bridge uses OpenTelemetry for distributed tracing. The changes are inline with changes to OpenTelemetry semantic conventions. The attributes will be removed in a future release of the Kafka Bridge