第5章 Using Kafka in KRaft mode

KRaft (Kafka Raft metadata) mode replaces Kafka’s dependency on ZooKeeper for cluster management. KRaft mode simplifies the deployment and management of Kafka clusters by bringing metadata management and coordination of clusters into Kafka.

Kafka in KRaft mode is designed to offer enhanced reliability, scalability, and throughput. Metadata operations become more efficient as they are directly integrated. And by removing the need to maintain a ZooKeeper cluster, there’s also a reduction in the operational and security overhead.

To deploy a Kafka cluster in KRaft mode, you must use Kafka and KafkaNodePool custom resources. The Kafka resource using KRaft mode must also have the annotations strimzi.io/kraft: enabled and strimzi.io/node-pools: enabled. For more details and examples, see 「Deploying a Kafka cluster in KRaft mode」.

Through node pool configuration using KafkaNodePool resources, nodes are assigned the role of broker, controller, or both:

Controllers use a metadata log, stored as a single-partition topic (__cluster_metadata) on every node, which records the state of the cluster. When requests are made to change the cluster configuration, an active (lead) controller manages updates to the metadata log, and follower controllers replicate these updates. The metadata log stores information on brokers, replicas, topics, and partitions, including the state of in-sync replicas and partition leadership. Kafka uses this metadata to coordinate changes and manage the cluster effectively.

Broker nodes act as observers, storing the metadata log passively to stay up-to-date with the cluster’s state. Each node fetches updates to the log independently.

In the following example, a Kafka cluster comprises a quorum of controller and broker nodes for fault tolerance and high availability.

図5.1 Example cluster with separate broker and controller nodes

In a typical production environment, use dedicated broker and controller nodes. However, you might want to use nodes in a dual-role configuration for development or testing.

You can use a combination of nodes that combine roles with nodes that perform a single role. In the following example, three nodes perform a dual role and two nodes act only as brokers.

図5.2 Example cluster with dual-role nodes and dedicated broker nodes