Release Notes for Streams for Apache Kafka 2.8 on OpenShift

Streams for Apache Kafka 2.8 is supported on OpenShift Container Platform 4.12 and 4.14 to 4.17.

For more information, see Chapter 10, Supported Configurations.

Streams for Apache Kafka now supports and uses Apache Kafka version 3.8.0. Only Kafka distributions built by Red Hat are supported.

You must upgrade the Cluster Operator to Streams for Apache Kafka version 2.8 before you can upgrade brokers and client applications to Kafka 3.8.0. For upgrade instructions, see Upgrading Streams for Apache Kafka.

Refer to the Kafka 3.8.0 Release Notes for additional information.

Kafka 3.7.x is supported only for the purpose of upgrading to Streams for Apache Kafka 2.8.

The UseKRaft feature gate moves to GA (General Availability) and is now permanently enabled. To deploy Kafka clusters in KRaft (Kafka Raft metadata) mode without ZooKeeper, the Kafka custom resource must include the annotation strimzi.io/kraft="enabled", and you must use KafkaNodePool resources to manage the configuration of groups of nodes.

For more information, see Deploying a Kafka cluster in KRaft mode and Configuring Kafka in KRaft mode.

KRaft limitations

The following Kafka features are currently not supported in KRaft:

The KafkaNodePools feature gate moves to GA (General Availability) and is now permanently enabled. The feature gate enables the configuration of different pools of Apache Kafka nodes through the KafkaNodePool custom resource. To use the KafkaNodePool resources, you still need to use the strimzi.io/node-pools: enabled annotation on the Kafka custom resources.

For more information, see Configuring node pools.

The UnidirectionalTopicOperator feature gate moves to GA (General Availability) and is now permanently enabled. The feature gate introduces KRaft-compatible unidirectional topic management, enabling the creation of Kafka topics using the KafkaTopic resource. These topics are then managed by the Topic Operator.

For more information, see Using the Topic Operator.

A new configuration mechanism supports quotas management. Configure a Kafka resource to enable the Strimzi Quotas plugin (strimzi) or Kafka’s built-in quotas management plugin (kafka).

The Strimzi Quotas plugin is now configured using .spec.kafka.quotas properties. Any configuration of the plugin inside .spec.kafka.config, as used in previous releases, is ignored and should be removed.

For more information, see Setting throughput and storage limits on brokers.

With the necessary permissions, you can now create REST API users to safely access a secured Cruise Control REST API directly. Standard Cruise Control USER and VIEWER roles are supported.

For more information, see API users.

It’s now possible to change the replication factor of topics by updating the replicas property value in a KafkaTopic resource managed by the Topic Operator. The Topic Operator uses Cruise Control to make the necessary changes, so Cruise Control must be deployed with Streams for Apache Kafka.

For more information, see Using Cruise Control to modify topic replication factor.

The Record Validation filter validates records sent by a producer. Only records that pass the validation are sent to the broker. This filter can be used to prevent poison messages—such as those containing corrupted data or invalid formats—from entering the Kafka system, which may otherwise lead to consumer failure.

For more information, see Record Validation filter.

The Record Encryption filter now supports integration with AWS Key Management Service (AWS KMS) as a Key Management Service (KMS). You can use AWS KMS to create encryption keys and give them aliases through which the filter references them.

For more information, see Record Encryption filter.

The new operator simplifies and streamlines the process of deploying the console using the Operator Lifecycle Manager (OLM).

For more information, see Deploying and connecting console to a Kafka cluster.

This release introduces per-cluster authentication support. You can now log in to provide credentials when accessing each Kafka cluster through the console.

For more information, see Logging into a Kafka cluster.

JBOD storage is now supported in KRaft mode. Use the kraftMetadata property in the storage configuration of a KafkaNodePool resource to specify the JBOD volume that stores the KRaft metadata log. By default, the log is stored on the volume with the lowest ID.

For more information, see Configuring the storage volume used to store the KRaft metadata log

When nodes are removed from a cluster, they are now deregistered so they are no longer tracked. The .status.nodeIds property in the Kafka custom resource stores a full list of node IDs, which is used to determine which nodes were removed and deregister them.

Additional OAuth configuration options have been added for OAuth 2.0 authentication on the listener and the client.

For more information, see Configuring OAuth 2.0 authentication on listeners and Setting up OAuth 2.0 on Kafka components.

Streams for Apache Kafka now supports specifying additional volumes and volume mounts for Kafka components, the User Operator, and the Topic Operator. You can configure volumes in the pod template (template.pod) and define volume mounts in the container template (template.KafkaContainer) within the component’s resource. All additional mounted paths are located inside /mnt to ensure compatibility with future Kafka and Streams for Apache Kafka updates.

For more information, see Additional volumes.

It’s now possible to specify certificates in resource configuration based on patterns instead of using certificate names using the new pattern property. For example, you can specify pattern: "*.crt" rather than specific certificate names when configuring trusted certificates. The means that the related custom resource does not need to be updated if the certificate file name changes.

You can add this configuration to the Kafka Connect, Kafka MirrorMaker, and Kafka Bridge components for TLS connections to the Kafka cluster. You can also use the pattern property in the configuration for oauth, keycloak, and opa authentication and authorization types that integrate with authorization servers.

For more information, see CertSecretSource schema reference.

You can now configure external Kafka listeners with the publishNotReadyAddresses property to consider service endpoints as ready even if the pods are not.

For more information, see GenericKafkaListenerConfiguration schema properties.

Support for specifying external IP addresses is now available when configuring node ports. Use the externalIPs property to associate external IP addresses with Kafka bootstrap and node port services. These addresses are used by clients external to the Kubernetes cluster to access the Kafka brokers.

For more information, see GenericKafkaListenerConfigurationBootstrap schema properties and GenericKafkaListenerConfigurationBroker schema properties.

If you need to configure custom SASL authentication, you can now define the necessary authentication properties using the STRIMZI_SASL_CUSTOM_CONFIG_JSON environment variable for the standalone Topic Operator. For example, this configuration may be used for accessing a Kafka cluster in a cloud provider with a custom login module

For more information, see Deploying the standalone Topic Operator.

Supported feature gates are now applicable to all Streams for Apache Kafka operators. While a particular feature gate might be used by one operator and ignored by the others, it can still be configured in all operators. When deploying the User Operator and Topic Operator within the context of the`Kafka` custom resource, the Cluster Operator automatically propagates the feature gates configuration to them. When the User Operator and Topic Operator are deployed standalone, without a Cluster Operator available to configure the feature gates, they must be directly configured within their deployments.

A warning is now triggered if the connectCluster configuration for a KafkaMirrorMaker2 resource does not specify the target Kafka cluster.

New alerts for failing connectors and tasks have been added to the metrics examples (prometheus-rules.yaml).

Metrics are now available for monitoring certificate expiration. The example Grafana dashboard for operators (strimzi-operators.json) presents the time certificates expire per cluster.

Kafka Bridge now supports OpenAPI v3. Support for OpenAPI v2 is now deprecated.

Producers can now specify a timestamp explicitly in ProducerRecord objects. A timestamp in the ConsumerRecord can also be read in a request response.

For more information, see ProducerRecord and ConsumerRecord.

The json embedded data format for Kafka messages now supports JSON arrays for record keys and values in the OpenAPI definition.

See the Streams for Apache Kafka Custom Resource API Reference.

Support for Java 11 is deprecated from Kafka 3.7.0 and Streams for Apache Kafka 2.7.0. Java 11 will be unsupported for all Streams for Apache Kafka components, including clients, in release 3.0.0.

Streams for Apache Kafka supports Java 17. Use Java 17 when developing new applications. Plan to migrate any applications that currently use Java 11 to 17.

If you want to continue using Java 11 for the time being, Streams for Apache Kafka 2.5 provides Long Term Support (LTS). For information on the LTS terms and dates, see the Streams for Apache Kafka LTS Support Policy.

The storage overrides (*.storage.overrides) for configuring per-broker storage are deprecated and will be removed in the future. If you are using the storage overrides, migrate to KafkaNodePool resources and use multiple node pools with a different storage class each.

For more information, see PersistentClaimStorage schema reference.

You can use configuration providers to load configuration data from external sources for all Kafka components, including producers and consumers.

Previously, you could enable the io.strimzi.kafka.EnvVarConfigProvider environment variable configuration provider using the config.providers properties in the spec configuration of a component. However, this provider is now deprecated and will be removed in the future. Therefore, it is recommended to update your implementation to use Kafka’s own environment variable configuration provider (org.apache.kafka.common.config.provider.EnvVarConfigProvider) to provide configuration properties as environment variables.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...
  config:
    # ...
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...
  config:
    # ...
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  # ...

Identity replication policy is a feature used with MirrorMaker 2 to override the automatic renaming of remote topics. Instead of prepending the name with the source cluster’s name, the topic retains its original name. This setting is particularly useful for active/passive backups and data migration scenarios.

To implement an identity replication policy, you must specify a replication policy class (replication.policy.class) in the MirrorMaker 2 configuration. Previously, you could specify the io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy class included with the Streams for Apache Kafka mirror-maker-2-extensions component. However, this component is now deprecated and will be removed in the future. Therefore, it is recommended to update your implementation to use Kafka’s own replication policy class (org.apache.kafka.connect.mirror.IdentityReplicationPolicy).

For more information, see Configuring Kafka MirrorMaker 2.

Kafka MirrorMaker replicates data between two or more active Kafka clusters, within or across data centers. Kafka MirrorMaker 1 was deprecated in Kafka 3.0.0 and will be removed in Kafka 4.0.0. MirrorMaker 2 will be the only version available. MirrorMaker 2 is based on the Kafka Connect framework, connectors managing the transfer of data between clusters.

As a result, MirrorMaker 1 (referred to as MirrorMaker in the documentation) has been deprecated in Streams for Apache Kafka, including the KafkaMirrorMaker custom resource, and support will be removed when Kafka 4.0.0 is adopted. To avoid disruptions, please transition to MirrorMaker 2 before support ends.

If you’re using MirrorMaker 1, you can replicate its functionality in MirrorMaker 2 by using the KafkaMirrorMaker2 custom resource with the IdentityReplicationPolicy class.. By default, MirrorMaker 2 renames topics replicated to a target cluster, but IdentityReplicationPolicy preserves the original topic names, enabling the same active/passive unidirectional replication as MirrorMaker 1.

For more information, see Configuring Kafka MirrorMaker 2.

Support for OpenAPI v2 is now deprecated and will be removed in the future. OpenAPI v3 is now supported. Plan to move to using OpenAPI v3.

During the transition to using OpenAPI v2, the /openapi endpoint returns the OpenAPI v2 specification using an aditional /openapi/v2 endpoint. A new /openapi/v3 endpoint returns the OpenAPI v3 specification.

The following Kafka Bridge span attributes are deprecated with replacements shown where applicable:

Kafka Bridge uses OpenTelemetry for distributed tracing. The changes are inline with changes to OpenTelemetry semantic conventions. The attributes will be removed in a future release of the Kafka Bridge