20.5. Changing the replication factor of topics

Procedure

1. If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named reassignment.json.

   Example reassignment JSON file showing the current and proposed replica assignment

   Current partition replica assignment
   {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[3,4,2,0],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[0,2,3,1],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[1,3,0,4],"log_dirs":["any","any","any","any"]}]}
   
   Proposed partition reassignment configuration
   {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3,4],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4,0],"log_dirs":["any","any","any","any"]}]}

   Save a copy of this file locally in case you need to revert the changes later on.

2. Edit the reassignment.json to remove a replica from each partition.

   For example use the jq command line JSON parser tool to remove the last replica in the list for each partition of the topic:

   Removing the last topic replica for each partition

   jq '.partitions[].replicas |= del(.[-1])' reassignment.json > reassignment.json

   Example reassignment file showing the updated replicas

   {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4],"log_dirs":["any","any","any","any"]}]}

3. Copy the reassignment.json file to the interactive pod container.

   oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json

   Replace <interactive_pod_name> with the name of the pod.

4. Start a shell process in the interactive pod container.

   oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash

   Replace <namespace> with the OpenShift namespace where the pod is running.

5. Make the topic replica change using the kafka-reassign-partitions.sh script from the interactive pod container.

   bin/kafka-reassign-partitions.sh --bootstrap-server
    <cluster_name>-kafka-bootstrap:9093 \
    --command-config /tmp/config.properties \
    --reassignment-json-file /tmp/reassignment.json \
    --execute

   注意

   Removing replicas from a broker does not require any inter-broker data movement, so there is no need to throttle replication. If you are adding replicas, then you may want to change the throttle rate.

6. Verify that the change to the topic replicas has completed using the kafka-reassign-partitions.sh command line tool from any of the broker pods. This is the same command as the previous step, but with the --verify option instead of the --execute option.

   bin/kafka-reassign-partitions.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --reassignment-json-file /tmp/reassignment.json \
     --verify

   The reassignment has finished when the --verify command reports that each of the partitions being moved has completed successfully. This final --verify will also have the effect of removing any reassignment throttles.

7. Run the bin/kafka-topics.sh command with the --describe option to see the results of the change to the topics.

   bin/kafka-topics.sh --bootstrap-server
     <cluster_name>-kafka-bootstrap:9093 \
     --command-config /tmp/config.properties \
     --describe

   Results of reducing the number of replicas for a topic

   my-topic  Partition: 0  Leader: 0  Replicas: 0,1,2 Isr: 0,1,2
   my-topic  Partition: 1  Leader: 2  Replicas: 1,2,3 Isr: 1,2,3
   my-topic  Partition: 2  Leader: 3  Replicas: 2,3,4 Isr: 2,3,4

8. Finally, edit the KafkaTopic custom resource to change .spec.replicas to 3, and then wait the reconciliation.

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaTopic
   metadata:
     name: my-topic
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     partitions: 3
     replicas: 3