20.3. Reassigning partitions after adding brokers

Procedure

1. Add as many new brokers as you need by increasing the Kafka.spec.kafka.replicas configuration option.
2. Verify that the new broker pods have started.
3. If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named reassignment.json.

4. Copy the reassignment.json file to the interactive pod container.

   oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json

   Replace <interactive_pod_name> with the name of the pod.

5. Start a shell process in the interactive pod container.

   oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash

   Replace <namespace> with the OpenShift namespace where the pod is running.

6. Run the partition reassignment using the kafka-reassign-partitions.sh script from the interactive pod container.

   bin/kafka-reassign-partitions.sh --bootstrap-server
    <cluster_name>-kafka-bootstrap:9093 \
    --command-config /tmp/config.properties \
    --reassignment-json-file /tmp/reassignment.json \
    --execute

   Replace <cluster_name> with the name of your Kafka cluster. For example, my-cluster-kafka-bootstrap:9093

   If you are going to throttle replication, you can also pass the --throttle option with an inter-broker throttled rate in bytes per second. For example:

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --throttle 5000000 \
     --execute

   This command will print out two reassignment JSON objects. The first records the current assignment for the partitions being moved. You should save this to a local file (not a file in the pod) in case you need to revert the reassignment later on. The second JSON object is the target reassignment you have passed in your reassignment JSON file.

   If you need to change the throttle during reassignment, you can use the same command with a different throttled rate. For example:

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --throttle 10000000 \
     --execute

7. Verify that the reassignment has completed using the kafka-reassign-partitions.sh command line tool from any of the broker pods. This is the same command as the previous step, but with the --verify option instead of the --execute option.

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --verify

   The reassignment has finished when the --verify command reports that each of the partitions being moved has completed successfully. This final --verify will also have the effect of removing any reassignment throttles.

8. You can now delete the revert file if you saved the JSON for reverting the assignment to their original brokers.