Chapter 5. Features

You can scale a Kafka cluster by adjusting the number of brokers using the spec.replicas property in the Kafka or KafkaNodePool custom resource used in deployment.

Enable auto-rebalancing to automatically redistribute topic partitions when scaling a cluster up or down. Auto-rebalancing requires a Cruise Control deployment, a rebalancing template for the operation, and autorebalance configuration in the Kafka resource referencing the template. When enabled, auto-rebalancing rebalances clusters that have been scaled up or down without further intervention.

For more information, see Triggering auto-rebalances when scaling clusters.

If you are using JBOD storage and have Cruise Control installed with Streams for Apache Kafka, you can now reassign partitions between the JBOD disks used for storage on the same broker. This capability also allows you to remove JBOD disks without data loss.

You configure a KafkaRebalance resource in remove-disks mode and specify a list of broker IDs with corresponding volume IDs for partition reassignment. Cruise Control generates an optimization proposal based on the configuration and reassigns the partitions when approved manually or automatically.

For more information, see Using Cruise Control to reassign partitions on JBOD disks.

A new mechanism allows connector offsets to be managed through KafkaConnect and KafkaMirrorMaker2 resources. It’s now possible to list, alter, and reset offsets.

For more information, see Configuring Kafka Connect connectors.

Hostnames and advertised hostnames for individual brokers can be specified using the host and advertisedHost properties. This release introduces support for using variables, such as {nodeId} or {nodePodName}, in the following templates:

By using templates, you no longer need to configure each broker individually. Streams for Apache Kafka automatically replaces the template variables with the corresponding values for each broker.

For more information, see Overriding advertised addresses for brokers and Specifying listener types.

Environment variables for any container deployed by Streams for Apache Kafka may now be based on values specified in a Secret or ConfigMap. This replaces the requirement to use the ExternalConfiguration schema for Kafka Connect and MirrorMaker 2 containers, which is now deprecated.

Values are referenced in the container configuration using the valueFrom.secretKeyRef or valueFrom.configMapKeyRef properties.

For more information, see Loading configuration values from environment variables.

Strimzi generates pod disruption budget resources for Kafka, Kafka Connect worker, MirrorMaker2 worker, and Kafka Bridge worker nodes.

If you want to use custom pod disruption budget resources, you can now set the STRIMZI_POD_DISRUPTION_BUDGET_GENERATION environment variable to false in the Cluster Operator configuration.

For more information, see Disabling pod disruption budget generation.

In order to support the CSI volumes, a new property named csi has been added to the AdditionalVolume schema. This property maps to the Kubernetes API CSIVolumeSource structure, allowing CSI volumes to be defined in container template fields.

For more information, see AdditionalVolume schema reference and Additional volumes.

Use the new admin/topics endpoint of the Kafka Bridge API to create topics. You can specify the topic name, partition count, replication factor in the request body.

For more information, see Securing connections from clients.

When configuring proxies, you can now use trust properties to configure virtual clusters to use TLS client authentication.

For more information, see Securing connections from clients.

The console (user interface) for Streams for Apache Kafka moves to GA. It is designed to seamlessly integrate with your Streams for Apache Kafka deployment, providing a centralized hub for monitoring and managing Kafka clusters. Deploy the console and connect it to Kafka clusters managed by Streams for Apache Kafka.

Gain insights into each connected cluster through dedicated console pages covering brokers, topics, and consumer groups. View essential information, such as the status of a Kafka cluster, before looking into specific details about brokers, topics, or connected consumer groups.

For more information, see the Streams for Apache Kafka Console guide.

You can now reset consumer offsets of a specific consumer group from the Consumer Groups page.

For more information, see Resetting consumer offsets.

When you configure KafkaRebalance resources to generate optimization proposals on a cluster, you can manage the proposals and any resulting rebalances from the Brokers page.

For more information, see Managing rebalances.

Pause and resume cluster reconciliations from the Cluster overview page. While paused, any changes to the cluster configuration using the Kafka custom resource are ignored until reconciliation is resumed.

For more information, see Pausing reconciliation of clusters.

The console now supports configuration of authorization rules in the console deployment configuration. Enable secure console connections to Kafka clusters using an OpenID Connect (OIDC) provider, such as Red Hat build of Keycloak. The configuration can be set up for all clusters or at the cluster level.

For more information, see Deploying the console.