Chapter 17. Using Cruise Control to reassign partitions on JBOD disks

Procedure

1. Start the Cruise Control server. The server starts on port 9092 by default; optionally, specify a different port.

   cd ./cruise-control/
   ./kafka-cruise-control-start.sh config/cruisecontrol.properties <port_number>

2. To verify that Cruise Control is running, send a GET request to the /state endpoint of the Cruise Control server:

   curl -X GET 'http://<cc_host>:<cc_port>/kafkacruisecontrol/state'

3. (Optional) Check the partition replica data on the broker by using the Kafka kafka-log-dirs.sh tool:

   kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 1 --topic-list my-topic

   The tool returns topic information for each log directory. In this example, we are restricting the information to my-topic to show the steps against a single topic. The JBOD volumes used for log directories are mounted at /var/lib/kafka/<volume_id>.

   Example output data for each log directory

   {
     "brokers": [
       {
         "broker": 1, 

   1

   
         "logDirs": [
           {
             "partitions": [  

   2

   
               {
                 "partition": "my-topic-0",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null, 

   3

   
             "logDir": "/var/lib/kafka/data-0" 

   4

   
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-1",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-1"
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-2",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-2"
           }
         ]
       }
     ]
   }

   1

   The broker ID.

   2

   Partition details: name, size, offset lag. The (isFuture) property indicates that the partition is moving between log directories when showing as true.

   3

   If error is not null, there is an issue with the disk hosting the log directory.

   4

   The path and name of the log directory.

4. Remove the volume from the node:

   curl -v -X POST 'http://<cc_host>:<cc_port>/kafkacruisecontrol/remove_disks?dryrun=false&brokerid_and_logdirs=1-/var/lib/kafka/data-0'

   The command specifies the broker ID and log directory for the volume being removed. If successful, partitions are reassigned from volume 0 on broker 1.

   Note

   If required, you can perform a dry run of this operation before applying the changes.

5. Use the Kafka kafka-log-dirs.sh tool again to verify volume removal and data movement.

   In this example, volume 0 has been removed and my-topic-0 partition reassigned to /var/lib/kafka/data-1.

   Example output data following reassignment of partitions

   {
     "brokers": [
       {
         "broker": 1,
         "logDirs": [
           {
             "partitions": [
               {
                 "partition": "my-topic-0",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               },
               {
                 "partition": "my-topic-1",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-1"
           },
           {
             "partitions": [
               {
                 "partition": "my-topic-2",
                 "size": 0,
                 "offsetLag": 0,
                 "isFuture": false
               }
             ],
             "error": null,
             "logDir": "/var/lib/kafka/data-2"
           }
         ]
       }
     ]
   }