20.4. Reassigning partitions before removing brokers

Procedure

1. If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named reassignment.json.

2. Copy the reassignment.json file to the interactive pod container.

   oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json

   Replace <interactive_pod_name> with the name of the pod.

3. Start a shell process in the interactive pod container.

   oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash

   Replace <namespace> with the OpenShift namespace where the pod is running.

4. Run the partition reassignment using the kafka-reassign-partitions.sh script from the interactive pod container.

   bin/kafka-reassign-partitions.sh --bootstrap-server
    <cluster_name>-kafka-bootstrap:9093 \
    --command-config /tmp/config.properties \
    --reassignment-json-file /tmp/reassignment.json \
    --execute

   Replace <cluster_name> with the name of your Kafka cluster. For example, my-cluster-kafka-bootstrap:9093

   If you are going to throttle replication, you can also pass the --throttle option with an inter-broker throttled rate in bytes per second. For example:

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --throttle 5000000 \
     --execute

   This command will print out two reassignment JSON objects. The first records the current assignment for the partitions being moved. You should save this to a local file (not a file in the pod) in case you need to revert the reassignment later on. The second JSON object is the target reassignment you have passed in your reassignment JSON file.

   If you need to change the throttle during reassignment, you can use the same command with a different throttled rate. For example:

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --throttle 10000000 \
     --execute

5. Verify that the reassignment has completed using the kafka-reassign-partitions.sh command line tool from any of the broker pods. This is the same command as the previous step, but with the --verify option instead of the --execute option.

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --verify

   The reassignment has finished when the --verify command reports that each of the partitions being moved has completed successfully. This final --verify will also have the effect of removing any reassignment throttles.

6. You can now delete the revert file if you saved the JSON for reverting the assignment to their original brokers.

7. When all the partition reassignments have finished, the brokers being removed should not have responsibility for any of the partitions in the cluster. You can verify this by checking that the broker’s data log directory does not contain any live partition logs. If the log directory on the broker contains a directory that does not match the extended regular expression \.[a-z0-9]-delete$, the broker still has live partitions and should not be stopped.

   You can check this by executing the command:

   oc exec my-cluster-kafka-0 -c kafka -it -- \
     /bin/bash -c \
     "ls -l /var/lib/kafka/kafka-log_<n>_ | grep -E '^d' | grep -vE '[a-zA-Z0-9.-]+\.[a-z0-9]+-delete$'"

   where n is the number of the pods being deleted.

   If the above command prints any output then the broker still has live partitions. In this case, either the reassignment has not finished or the reassignment JSON file was incorrect.

8. When you have confirmed that the broker has no live partitions, you can edit the Kafka.spec.kafka.replicas property of your Kafka resource to reduce the number of brokers.