Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 6. Streams for Apache Kafka Operators
Operators are a method of packaging, deploying, and managing OpenShift applications. They provide a way to extend the Kubernetes API and simplify the administration tasks associated with specific applications.
Streams for Apache Kafka operators support tasks related to a Kafka deployment. Streams for Apache Kafka custom resources provide the deployment configuration. This includes configuration for Kafka clusters, topics, users, and other components. Leveraging custom resource configuration, Streams for Apache Kafka operators create, configure, and manage Kafka components within an OpenShift environment. Using operators reduces the need for manual intervention and streamlines the process of managing Kafka in an OpenShift cluster.
Streams for Apache Kafka provides the following operators for managing a Kafka cluster running within an OpenShift cluster.
- Cluster Operator
- Deploys and manages Apache Kafka clusters, Kafka Connect, Kafka MirrorMaker, Kafka Bridge, Kafka Exporter, Cruise Control, and the Entity Operator
- Entity Operator
- Comprises the Topic Operator and User Operator
- Topic Operator
- Manages Kafka topics
- User Operator
- Manages Kafka users
The Cluster Operator can deploy the Topic Operator and User Operator as part of an Entity Operator configuration at the same time as a Kafka cluster.
Operators within the Streams for Apache Kafka architecture
6.1. Cluster Operator
Streams for Apache Kafka uses the Cluster Operator to deploy and manage clusters. By default, when you deploy Streams for Apache Kafka a single Cluster Operator replica is deployed. You can add replicas with leader election so that additional Cluster Operators are on standby in case of disruption.
The Cluster Operator manages the clusters of the following Kafka components:
- Kafka (including ZooKeeper, Entity Operator, Kafka Exporter, and Cruise Control)
- Kafka Connect
- Kafka MirrorMaker
- Kafka Bridge
The clusters are deployed using custom resources.
For example, to deploy a Kafka cluster:
-
A
Kafka
resource with the cluster configuration is created within the OpenShift cluster. -
The Cluster Operator deploys a corresponding Kafka cluster, based on what is declared in the
Kafka
resource.
The Cluster Operator can also deploy the following Streams for Apache Kafka operators through configuration of the Kafka
resource:
-
Topic Operator to provide operator-style topic management through
KafkaTopic
custom resources -
User Operator to provide operator-style user management through
KafkaUser
custom resources
The Topic Operator and User Operator function within the Entity Operator on deployment.
You can use the Cluster Operator with a deployment of Streams for Apache Kafka Drain Cleaner to help with pod evictions. By deploying the Streams for Apache Kafka Drain Cleaner, you can use the Cluster Operator to move Kafka pods instead of OpenShift. Streams for Apache Kafka Drain Cleaner annotates pods being evicted with a rolling update annotation. The annotation informs the Cluster Operator to perform the rolling update.
Example architecture for the Cluster Operator
6.2. Topic Operator
The Topic Operator provides a way of managing topics in a Kafka cluster through KafkaTopic
resources.
Example architecture for the Topic Operator
The Topic Operator manages Kafka topics by watching for KafkaTopic
resources that describe Kafka topics, and ensuring that they are configured properly in the Kafka cluster.
When a KafkaTopic
is created, deleted, or changed, the Topic Operator performs the corresponding action on the Kafka topic.
You can declare a KafkaTopic
as part of your application’s deployment and the Topic Operator manages the Kafka topic for you.
The Topic Operator operates in the following modes:
- Unidirectional mode
-
Unidirectional mode means that the Topic Operator solely manages topics through the
KafkaTopic
resource. This mode does not require ZooKeeper and is compatible with using Streams for Apache Kafka in KRaft mode. - Bidirectional mode
Bidirectional mode means that the Topic Operator can reconcile changes to a
KafkaTopic
resource to and from a Kafka cluster. This means that you can update topics either through theKafkaTopic
resource or directly in Kafka, and the Topic Operator will ensure that both sources are updated to reflect the changes. This mode requires ZooKeeper for cluster management.The Topic Operator maintains information about each topic in a topic store, which is continually synchronized with updates from OpenShift
KafkaTopic
custom resources or Kafka topics. Updates from operations applied to a local in-memory topic store are persisted to a backup topic store on disk.
6.3. User Operator
The User Operator provides a way of managing users in a Kafka cluster through KafkaUser
resources.
The User Operator manages Kafka users for a Kafka cluster by watching for KafkaUser
resources that describe Kafka users, and ensuring that they are configured properly in the Kafka cluster.
When a KafkaUser
is created, deleted, or changed, the User Operator performs the corresponding action on the Kafka user.
You can declare a KafkaUser
resource as part of your application’s deployment and the User Operator manages the Kafka user for you. You can specify the authentication and authorization mechanism for the user. You can also configure user quotas that control usage of Kafka resources to ensure, for example, that a user does not monopolize access to a broker.
When the user is created, the user credentials are created in a Secret
. Your application needs to use the user and its credentials for authentication and to produce or consume messages.
In addition to managing credentials for authentication, the User Operator also manages authorization rules by including a description of the user’s access rights in the KafkaUser
declaration.
6.4. Feature gates in Streams for Apache Kafka Operators
Streams for Apache Kafka operators use feature gates to enable or disable specific features and functions. Enabling a feature gate alters the behavior of the associated operator, introducing the corresponding feature to your Streams for Apache Kafka deployment.
Feature gates are set in the operator configuration and have three stages of maturity: alpha, beta, or graduated. Graduated feature gates have reached General Availability (GA) and are permanently enabled features.
For more information, see Feature gates.