Chapter 23. Using Cruise Control to reassign partitions on JBOD disks

Procedure

1. Run a new interactive pod container using the Kafka image to connect to a running Kafka broker.

   oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-40-rhel9:3.0.1 helper-pod -- /bin/sh -c "sleep 3600"

   In this procedure, we use a pod named helper-pod.

2. (Optional) Check the partition replica data on broker 0 by opening a terminal inside the interactive pod and running the Kafka kafka-log-dirs.sh tool:

   oc exec -n myproject -ti my-cluster-pool-a-0 bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 0,1,2 --topic-list my-topic

   my-cluster-pool-a-0 is the pod name for broker 0. The tool returns topic information for each log directory. In this example, we are restricting the information to my-topic to show the steps against a single topic. The JBOD volumes used for log directories are mounted at /var/lib/kafka/data-<volume_id>/kafka-log<pod_id>.

   Example output data for each log directory

   {
     "brokers": [
       {
         "broker": 0, 

   1

   
         "logDirs": [
           {
             "partitions": [ 

   2

   
               {
                 "partition": "my-topic-5",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-2",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null, 

   3

   
             "logDir": "/var/lib/kafka/data-2/kafka-log0" 

   4

   
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-0",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-3",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-0/kafka-log0"
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-4",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-1",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-1/kafka-log0"
           }
         ]
       }

   1

   The broker ID.

   2

   Partition details: name, size, offset lag. The (isFuture) property indicates that the partition is moving between log directories when showing as true.

   3

   If error is not null, there is an issue with the disk hosting the log directory.

   4

   The path and name of the log directory.

3. Create a KafkaRebalance resource in remove-disks mode, listing the brokers and volume IDs to reassign partitions from. Without specific configuration, the default rebalance goals are used.

   Example Cruise Control configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     name: my-rebalance
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     mode: remove-disks
     moveReplicasOffVolumes:
       - brokerId: 0 

   1

   
         volumeIds: [1, 2] 

   2

   1

   The broker from which to reassign partitions.

   2

   The volume IDs to reassign partitions from.

   In this example, my-rebalance reassigns partitions from volumes with IDs 1 and 2 on broker 0.

4. (Optional) To approve the optimization proposal automatically, set the strimzi.io/rebalance-auto-approval annotation to true:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     name: my-rebalance
     labels:
       strimzi.io/cluster: my-cluster
     annotations:
       strimzi.io/rebalance-auto-approval: "true"
   spec:
     mode: remove-disks
     moveReplicasOffVolumes:
       - brokerId: 0
         volumeIds: [1, 2]

5. Apply the KafkaRebalance configuration.

6. If manually approving, wait for the status of the proposal to move to ProposalReady before approving the changes.

   1. Check the summary of the changes in the KafkaRebalance status:

      oc get kafkarebalance my-rebalance -n my-project -o yaml

      Example summary of changes

      apiVersion: kafka.strimzi.io/v1beta2
      kind: KafkaRebalance
      metadata:
        name: my-rebalance
        labels:
          strimzi.io/cluster: my-cluster
      spec:
        mode: remove-disks
        moveReplicasOffVolumes:
          - brokerId: 0
            volumeIds: [1, 2]
      status:
        - lastTransitionTime: "2024-11-13T06:55:42.217794891Z"
          status: "True"
          type: ProposalReady
        observedGeneration: 1
        optimizationResult:
          afterBeforeLoadConfigMap: my-rebalance
          dataToMoveMB: 0
          excludedBrokersForLeadership: []
          excludedBrokersForReplicaMove: []
          excludedTopics: []
          intraBrokerDataToMoveMB: 0
          monitoredPartitionsPercentage: 100
          numIntraBrokerReplicaMovements: 26
          numLeaderMovements: 0
          numReplicaMovements: 0
          onDemandBalancednessScoreAfter: 100
          onDemandBalancednessScoreBefore: 0
          provisionRecommendation: ""
          provisionStatus: UNDECIDED
          recentWindows: 1
        sessionId: 24537b9c-a315-4715-8e86-01481e914771

      Note

      The summary only shows the changes after optimization, not the load before optimization.

   2. Annotate the KafkaRebalance resource to approve the changes:

      oc annotate kafkarebalance my-rebalance strimzi.io/rebalance="approve"

7. Wait for the status of the proposal to change to Ready.

8. Use the Kafka kafka-log-dirs.sh tool again to verify data movement.

   In this example, the log directories for volumes 1 and 2 no longer have partitions assigned to them and volume 0 holds 6 partitions for my-topic, indicating that the partitions have been successfully reassigned.

   Example output data following reassignment of partitions

   {
     "brokers": [
       {
         "broker": 0,
         "logDirs": [
           {
             "partitions": [],
             "error": null,
             "logDir": "/var/lib/kafka/data-2/kafka-log0"
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-4",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-5",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-0",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-1",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-2",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-3",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-0/kafka-log0"
           },
           {
             "partitions": [],
             "error": null,
             "logDir": "/var/lib/kafka/data-1/kafka-log0"
           }
         ]
       }

9. To prevent empty volumes from being used in future rebalances or topic allocations, update the configuration and remove the associated persistent volume claims (PVCs).

   1. Update the node pool configuration to exclude the volumes.

      Warning

      Before making changes, verify that all partitions have been successfully moved using kafka-log-dirs.sh. Removing volumes prematurely can cause data loss.

      In this example, volumes 1 and 2 are removed, and only volume 0 is retained:

      Updated node pool configuration with single volume JBOD storage

      apiVersion: kafka.strimzi.io/v1beta2
      kind: KafkaNodePool
      metadata:
        name: pool-a
        labels:
          strimzi.io/cluster: my-cluster
      spec:
        replicas: 3
        roles:
          - broker
        storage:
          type: jbod
          volumes:
            - id: 0
              type: persistent-claim
              size: 2000Gi
              deleteClaim: false
        # ...

   2. Delete the unused PVCs.

      PVCs are named using the format data-<id>-<kafka_cluster_name>-kafka-<pod_id>. You can list them using:

      oc get pvc -n my-project

      Then delete the unused PVCs:

      oc delete pvc data-<id>-<kafka_cluster_name>-kafka-<pod_id> -n my-project

      Note

      Deleting a PVC removes the underlying storage unless deleteClaim: false is set in the volume configuration.

   3. (Optional) Delete the helper pod used earlier:

      oc delete pod helper-pod -n my-project