Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 23. Using Cruise Control to reassign partitions on JBOD disks

If you are using JBOD storage and have Cruise Control installed with Streams for Apache Kafka, you can reassign partitions between the JBOD disks used for storage on the same broker. This capability also allows you to remove JBOD disks without data loss.

To reassign partitions, configure a KafkaRebalance resource in remove-disks mode and specify a list of broker IDs with corresponding volume IDs for partition reassignment. Cruise Control generates an optimization proposal based on the configuration and reassigns the partitions when approved manually or automatically.

Use the Kafka kafka-log-dirs.sh tool to check information about Kafka topic partitions and their location on brokers before and after moving them. Use an interactive pod to avoid running the tool within the broker container and causing any disruptions.

Prerequisites

In this procedure, we use a Kafka cluster named my-cluster, which is deployed to the my-project namespace with node pools and Cruise Control enabled.

Example Kafka cluster configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: my-project
  annotations:
    strimzi.io/node-pools: enabled
spec:
  kafka:
    # ...
  cruiseControl: {}
    # ...
Copy to Clipboard Toggle word wrap

A node pool named pool-a is configured with three broker replicas that use three JBOD storage volumes. In the procedure, we show how partitions are reassigned from volume 1 and 2 to volume 0.

Example node pool configuration with JBOD storage

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 2000Gi
        deleteClaim: false
      - id: 1
        type: persistent-claim
        size: 2000Gi
        deleteClaim: false
      - id: 2
        type: persistent-claim
        size: 2000Gi
        deleteClaim: false
  # ...
Copy to Clipboard Toggle word wrap

Procedure

  1. Run a new interactive pod container using the Kafka image to connect to a running Kafka broker. 

    oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-39-rhel9:2.9.3 helper-pod -- /bin/sh -c "sleep 3600"
    Copy to Clipboard Toggle word wrap

    In this procedure, we use a pod named helper-pod.

  2. (Optional) Check the partition replica data on broker 0 by opening a terminal inside the interactive pod and running the Kafka kafka-log-dirs.sh tool: 

    oc exec -n myproject -ti my-cluster-pool-a-0 bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 0,1,2 --topic-list my-topic
    Copy to Clipboard Toggle word wrap

    my-cluster-pool-a-0 is the pod name for broker 0. The tool returns topic information for each log directory. In this example, we are restricting the information to my-topic to show the steps against a single topic. The JBOD volumes used for log directories are mounted at /var/lib/kafka/data-<volume_id>/kafka-log<pod_id>.

    Example output data for each log directory

    {
  "brokers": [
    {
      "broker": 0,
    1 
    
      "logDirs": [
        {
          "partitions": [
    2 
    
            {
              "partition": "my-topic-5",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-2",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            }
          ],
          "error": null,
    3 
    
          "logDir": "/var/lib/kafka/data-2/kafka-log0"
    4 
    
        },
        {
          "partitions": [
            {
              "partition": "my-topic-0",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-3",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            }
          ],
          "error": null,
          "logDir": "/var/lib/kafka/data-0/kafka-log0"
        },
        {
          "partitions": [
            {
              "partition": "my-topic-4",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-1",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            }
          ],
          "error": null,
          "logDir": "/var/lib/kafka/data-1/kafka-log0"
        }
      ]
    }
    Copy to Clipboard Toggle word wrap

    1
    The broker ID.
    2
    Partition details: name, size, offset lag. The (isFuture) property indicates that the partition is moving between log directories when showing as true.
    3
    If error is not null, there is an issue with the disk hosting the log directory.
    4
    The path and name of the log directory.

  3. Create a KafkaRebalance resource in remove-disks mode, listing the brokers and volume IDs to reassign partitions from. Without specific configuration, the default rebalance goals are used.

    Example Cruise Control configuration

    apiVersion: {KafkaRebalanceApiVersion}
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: remove-disks
  moveReplicasOffVolumes:
    - brokerId: 0
    1 
    
      volumeIds: [1, 2]
    2
    Copy to Clipboard Toggle word wrap

    1
    The broker from which to reassign partitions.
    2
    The volume IDs to reassign partitions from.

    In this example, my-rebalance reassigns partitions from volumes with IDs 1 and 2 on broker 0.

  4. (Optional) To approve the optimization proposal automatically, set the strimzi.io/rebalance-auto-approval annotation to true: 

    apiVersion: {KafkaRebalanceApiVersion}
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
  annotations:
    strimzi.io/rebalance-auto-approval: "true"
spec:
  mode: remove-disks
  moveReplicasOffVolumes:
    - brokerId: 0
      volumeIds: [1, 2]
    Copy to Clipboard Toggle word wrap
  5. Apply the KafkaRebalance configuration.

  6. If manually approving, wait for the status of the proposal to move to ProposalReady before approving the changes.

    1. Check the summary of the changes in the KafkaRebalance status: 

      oc get kafkarebalance my-rebalance -n my-project -o yaml
      Copy to Clipboard Toggle word wrap

      Example summary of changes

      apiVersion: {KafkaRebalanceApiVersion}
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: remove-disks
  moveReplicasOffVolumes:
    - brokerId: 0
      volumeIds: [1, 2]
status:
  - lastTransitionTime: "2024-11-13T06:55:42.217794891Z"
    status: "True"
    type: ProposalReady
  observedGeneration: 1
  optimizationResult:
    afterBeforeLoadConfigMap: my-rebalance
    dataToMoveMB: 0
    excludedBrokersForLeadership: []
    excludedBrokersForReplicaMove: []
    excludedTopics: []
    intraBrokerDataToMoveMB: 0
    monitoredPartitionsPercentage: 100
    numIntraBrokerReplicaMovements: 26
    numLeaderMovements: 0
    numReplicaMovements: 0
    onDemandBalancednessScoreAfter: 100
    onDemandBalancednessScoreBefore: 0
    provisionRecommendation: ""
    provisionStatus: UNDECIDED
    recentWindows: 1
  sessionId: 24537b9c-a315-4715-8e86-01481e914771
      Copy to Clipboard Toggle word wrap

      Note

      The summary only shows the changes after optimization, not the load before optimization.

    2. Annotate the KafkaRebalance resource to approve the changes: 

      oc annotate kafkarebalance my-rebalance strimzi.io/rebalance="approve"
      Copy to Clipboard Toggle word wrap
  7. Wait for the status of the proposal to change to Ready.

  8. Use the Kafka kafka-log-dirs.sh tool again to verify data movement.

    In this example, the log directories for volumes 1 and 2 no longer have partitions assigned to them and volume 0 holds 6 partitions for my-topic, indicating that the partitions have been successfully reassigned.

    Example output data following reassignment of partitions

    {
  "brokers": [
    {
      "broker": 0,
      "logDirs": [
        {
          "partitions": [],
          "error": null,
          "logDir": "/var/lib/kafka/data-2/kafka-log0"
        },
        {
          "partitions": [
            {
              "partition": "my-topic-4",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-5",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-0",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-1",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-2",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            },
            {
              "partition": "my-topic-3",
              "size": 0,
              "offsetLag": 0,
              "isFuture": false
            }
          ],
          "error": null,
          "logDir": "/var/lib/kafka/data-0/kafka-log0"
        },
        {
          "partitions": [],
          "error": null,
          "logDir": "/var/lib/kafka/data-1/kafka-log0"
        }
      ]
    }
    Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat