Chapter 8. Migrating to KRaft mode

Procedure

1. For the Kafka cluster, create a node pool with a controller role.

   The node pool adds a quorum of controller nodes to the cluster.

   Example configuration for a controller node pool

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: controller
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - controller
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 20Gi
           deleteClaim: false
       resources:
         requests:
           memory: 64Gi
           cpu: "8"
         limits:
           memory: 64Gi
           cpu: "12"

   Note

   For the migration, you cannot use a node pool of nodes that share the broker and controller roles.

2. Apply the new KafkaNodePool resource to create the controllers.

   Errors related to using controllers in a ZooKeeper-based environment are expected in the Cluster Operator logs. The errors can block reconciliation. To prevent this, perform the next step immediately.

3. Enable KRaft migration in the Kafka resource by setting the strimzi.io/kraft annotation to migration:

   oc annotate kafka my-cluster strimzi.io/kraft="migration" --overwrite

   Enabling KRaft migration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: my-project
     annotations:
       strimzi.io/kraft="migration"
   # ...

   Applying the annotation to the Kafka resource configuration starts the migration.

4. Check the controllers have started and the brokers have rolled:

   oc get pods -n my-project

   Output shows nodes in broker and controller node pools

   NAME                     READY  STATUS   RESTARTS
   my-cluster-kafka-0       1/1    Running  0
   my-cluster-kafka-1       1/1    Running  0
   my-cluster-kafka-2       1/1    Running  0
   my-cluster-controller-3  1/1    Running  0
   my-cluster-controller-4  1/1    Running  0
   my-cluster-controller-5  1/1    Running  0
   # ...

5. Check the status of the migration:

   oc get kafka my-cluster -n my-project -w

   Updates to the metadata state

   NAME        ...  METADATA STATE
   my-cluster  ...  Zookeeper
   my-cluster  ...  KRaftMigration
   my-cluster  ...  KRaftDualWriting
   my-cluster  ...  KRaftPostMigration

   METADATA STATE shows the mechanism used to manage Kafka metadata and coordinate operations. At the start of the migration this is ZooKeeper.

   • ZooKeeper is the initial state when metadata is only stored in ZooKeeper.
   • KRaftMigration is the state when the migration is in progress. The flag to enable ZooKeeper to KRaft migration (zookeeper.metadata.migration.enable) is added to the brokers and they are rolled to register with the controllers. The migration can take some time at this point depending on the number of topics and partitions in the cluster.
   • KRaftDualWriting is the state when the Kafka cluster is working as a KRaft cluster, but metadata are being stored in both Kafka and ZooKeeper. Brokers are rolled a second time to remove the flag to enable migration.
   • KRaftPostMigration is the state when KRaft mode is enabled for brokers. Metadata are still being stored in both Kafka and ZooKeeper.

   The migration status is also represented in the status.kafkaMetadataState property of the Kafka resource.

   Warning

   You can roll back to using ZooKeeper from this point. The next step is to enable KRaft. Rollback cannot be performed after enabling KRaft.

6. When the metadata state has reached KRaftPostMigration, enable KRaft in the Kafka resource configuration by setting the strimzi.io/kraft annotation to enabled:

   oc annotate kafka my-cluster strimzi.io/kraft="enabled" --overwrite

   Enabling KRaft migration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: my-project
     annotations:
       strimzi.io/kraft="enabled"
   # ...

7. Check the status of the move to full KRaft mode:

   oc get kafka my-cluster -n my-project -w

   Updates to the metadata state

   NAME        ...  METADATA STATE
   my-cluster  ...  Zookeeper
   my-cluster  ...  KRaftMigration
   my-cluster  ...  KRaftDualWriting
   my-cluster  ...  KRaftPostMigration
   my-cluster  ...  PreKRaft
   my-cluster  ...  KRaft

   • PreKRaft is the state when all ZooKeeper-related resources have been automatically deleted.
   • KRaft is the final state (after the controllers have rolled) when the KRaft migration is finalized.
   Note

   Depending on how deleteClaim is configured for ZooKeeper, its Persistent Volume Claims (PVCs) and persistent volumes (PVs) may not be deleted. deleteClaim specifies whether the PVC is deleted when the cluster is uninstalled. The default is false.

8. Remove any ZooKeeper-related configuration from the Kafka resource.

   If present, you can remove the following:

   • log.message.format.version
   • inter.broker.protocol.version

   • spec.zookeeper.* properties

     Removing log.message.format.version and inter.broker.protocol.version causes the brokers and controllers to roll again. Removing ZooKeeper properties removes any warning messages related to ZooKeeper configuration being present in a KRaft-operated cluster.

1. Apply the strimzi.io/kraft="rollback" annotation to the Kafka resource to roll back the brokers.

   oc annotate kafka my-cluster strimzi.io/kraft="rollback" --overwrite

   Rolling back KRaft migration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: my-project
     annotations:
       strimzi.io/kraft="rollback"
   # ...

   The migration process must be in the KRaftPostMigration state to do this. The brokers are rolled back so that they can be connected to ZooKeeper again and the state returns to KRaftDualWriting.

2. Delete the controllers node pool:

   oc delete KafkaNodePool controller -n my-project

3. Apply the strimzi.io/kraft="disabled" annotation to the Kafka resource to return the metadata state to ZooKeeper.

   oc annotate kafka my-cluster strimzi.io/kraft="disabled" --overwrite

   Switching back to using ZooKeeper

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     namespace: my-project
     annotations:
       strimzi.io/kraft="disabled"
   # ...