Chapter 2. Streams for Apache Kafka operators

Streams for Apache Kafka uses the Cluster Operator to deploy and manage clusters. By default, when you deploy Streams for Apache Kafka a single Cluster Operator replica is deployed. You can add replicas with leader election so that additional Cluster Operators are on standby in case of disruption.

The Cluster Operator manages the clusters of the following Kafka components:

The clusters are deployed using custom resources.

For example, to deploy a Kafka cluster:

The Cluster Operator can also deploy the following Streams for Apache Kafka operators through configuration of the Kafka resource:

The Topic Operator and User Operator function within the Entity Operator on deployment.

The Topic Operator provides a way of managing topics in a Kafka cluster through KafkaTopic resources. The Cluster Operator can deploy the Topic Operator as part of the Entity Operator configuration at the same time as a Kafka cluster. The operator can also be deployed standalone to help manage topics for Kafka clusters not operated by Streams for Apache Kafka.

The Topic Operator manages Kafka topics by watching for KafkaTopic resources that describe Kafka topics, and ensuring that they are configured properly in the Kafka cluster.

When a KafkaTopic is created, deleted, or changed, the Topic Operator performs the corresponding action on the Kafka topic.

You can declare a KafkaTopic as part of your application’s deployment and the Topic Operator manages the Kafka topic for you.

The User Operator provides a way of managing users in a Kafka cluster through KafkaUser resources. The Cluster Operator can deploy the User Operator as part of the Entity Operator configuration at the same time as a Kafka cluster. The operator can also be deployed standalone to help manage users for Kafka clusters not operated by Streams for Apache Kafka.

The User Operator manages Kafka users for a Kafka cluster by watching for KafkaUser resources that describe Kafka users, and ensuring that they are configured properly in the Kafka cluster.

When a KafkaUser is created, deleted, or changed, the User Operator performs the corresponding action on the Kafka user.

You can declare a KafkaUser resource as part of your application’s deployment and the User Operator manages the Kafka user for you. You can specify the authentication and authorization mechanism for the user. You can also configure user quotas that control usage of Kafka resources to ensure, for example, that a user does not monopolize access to a broker.

When the user is created, the user credentials are created in a Secret. Your application needs to use the user and its credentials for authentication and to produce or consume messages.

In addition to managing credentials for authentication, the User Operator also manages authorization rules by including a description of the user’s access rights in the KafkaUser declaration.

If you are using the Cluster Operator to manage your Kafka cluster, you can deploy and use the Drain Cleaner to streamline the process of moving Kafka pods from OpenShift nodes scheduled for maintenance.

By deploying the Streams for Apache Kafka Drain Cleaner, you can use the Cluster Operator to move Kafka pods instead of OpenShift. Streams for Apache Kafka Drain Cleaner annotates pods being evicted with a rolling update annotation. The annotation informs the Cluster Operator to perform the rolling update.

Drain Cleaner ensures that no partition replicas become under-replicated during node draining, maintaining data availability and fault tolerance. This controlled approach minimizes potential disruptions to your Kafka cluster when draining pods.

Streams for Apache Kafka operators use feature gates to enable or disable specific features and functions. Enabling a feature gate alters the behavior of the associated operator, introducing the corresponding feature to your Streams for Apache Kafka deployment.

Feature gates are set in the operator configuration and have three stages of maturity: alpha, beta, or graduated. Graduated feature gates have reached General Availability (GA) and are permanently enabled features.

For more information, see Feature gates.