Chapter 15. Using Cruise Control for cluster rebalancing

Procedure

1. Download the latest version of the Red Hat Streams for Apache Kafka Cruise Control archive from the Red Hat Customer Portal.

2. Create the /opt/cruise-control directory:

   sudo mkdir /opt/cruise-control

   Copy to Clipboard Toggle word wrap

3. Extract the contents of the Cruise Control ZIP file to the new directory:

   unzip amq-streams-<version>-cruise-control-bin.zip -d /opt/cruise-control

   Copy to Clipboard Toggle word wrap

4. Change the ownership of the /opt/cruise-control directory to the Kafka user:

   sudo chown -R kafka:kafka /opt/cruise-control

   Copy to Clipboard Toggle word wrap

1. Stop the Kafka broker:

   ./bin/kafka-server-stop.sh

   Copy to Clipboard Toggle word wrap

2. Edit the Kafka configuration properties file to configure the Cruise Control Metrics Reporter.

   1. Add the CruiseControlMetricsReporter class to the metric.reporters configuration option. Do not remove any existing Metrics Reporters.

      metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter

      Copy to Clipboard Toggle word wrap

   2. Add the following configuration options and values:

      cruise.control.metrics.topic.auto.create=true
      cruise.control.metrics.topic.num.partitions=1
      cruise.control.metrics.topic.replication.factor=1

      Copy to Clipboard Toggle word wrap

      These options enable the Cruise Control Metrics Reporter to create the __CruiseControlMetrics topic with a log cleanup policy of DELETE. For more information, see Auto-created topics and Configuring logging and cleanup policy.

3. Configure SSL, if required.

   1. In the Kafka configuration properties file, configure SSL between the Cruise Control Metrics Reporter and the Kafka broker by setting the relevant client configuration properties.

      The Metrics Reporter accepts all standard producer-specific configuration properties with the cruise.control.metrics.reporter prefix. For example: cruise.control.metrics.reporter.ssl.truststore.password.

   2. In the Cruise Control properties file (./cruise-control/config/cruisecontrol.properties) configure SSL between the Kafka broker and the Cruise Control server by setting the relevant client configuration properties.

      Cruise Control inherits SSL client property options from Kafka and uses those properties for all Cruise Control server clients.

4. Restart the Kafka broker:

   ./bin/kafka-server-start.sh -daemon ./config/server.properties

   Copy to Clipboard Toggle word wrap

   For information on restarting brokers in a multi-node cluster, see Section 4.3, “Performing a graceful rolling restart of Kafka brokers”.

5. Repeat steps 1-5 for the remaining brokers.

Procedure

1. Edit the Cruise Control properties file (./cruise-control/config/cruisecontrol.properties).

2. Configure the properties shown in the following example configuration:

   # The Kafka cluster to control.
   bootstrap.servers=localhost:9092 

   1

   
   
   # The replication factor of Kafka metric sample store topic
   sample.store.topic.replication.factor=2 

   2

   
   
   # The configuration for the BrokerCapacityConfigFileResolver (supports JBOD, non-JBOD, and heterogeneous CPU core capacities)
   #capacity.config.file=config/capacity.json
   #capacity.config.file=config/capacityCores.json
   capacity.config.file=config/capacityJBOD.json 

   3

   
   
   # The list of goals to optimize the Kafka cluster for with pre-computed proposals
   default.goals={List of default optimization goals} 

   4

   
   
   # The list of supported goals
   goals={list of supported optimization goals} 

   5

   
   
   # The list of supported hard goals
   hard.goals={List of hard goals} 

   6

   
   
   # How often should the cached proposal be expired and recalculated if necessary
   proposal.expiration.ms=60000 

   7

   
   
   # The zookeeper connect of the Kafka cluster
   zookeeper.connect=localhost:2181 

   8

   Copy to Clipboard Toggle word wrap

   1

   Host and port numbers of the Kafka broker (always port 9092).

   2

   Replication factor of the Kafka metric sample store topic. If you are evaluating Cruise Control in a single-node Kafka and ZooKeeper cluster, set this property to 1. For production use, set this property to 2 or more.

   3

   The configuration file that sets the maximum capacity limits for broker resources. Use the file that applies to your Kafka deployment configuration. For more information, see Capacity configuration.

   4

   Comma-separated list of default optimization goals, using fully-qualified domain names (FQDNs). A number of supported optimization goals (see 5) are already set as default optimization goals; you can add or remove goals if desired.

   5

   Comma-separated list of supported optimization goals, using FQDNs. To completely exclude goals from being used to generate optimization proposals, remove them from the list.

   6

   Comma-separated list of hard goals, using FQDNs. Seven of the supported optimization goals are already set as hard goals; you can add or remove goals if desired.

   7

   The interval, in milliseconds, for refreshing the cached optimization proposal that is generated from the default optimization goals.

   8

   Host and port numbers of the ZooKeeper connection (always port 2181).

3. Start the Cruise Control server. The server starts on port 9092 by default; optionally, specify a different port.

   cd ./cruise-control/
   ./kafka-cruise-control-start.sh config/cruisecontrol.properties <port_number>

   Copy to Clipboard Toggle word wrap

4. To verify that Cruise Control is running, send a GET request to the /state endpoint of the Cruise Control server:

   curl -X GET 'http://<cc_host>:<cc_port>/kafkacruisecontrol/state'

   Copy to Clipboard Toggle word wrap

1. Get the current configuration of the __CruiseControlMetrics topic:

   opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --entity-type topics --entity-name __CruiseControlMetrics --describe

   Copy to Clipboard Toggle word wrap

2. Change the log cleanup policy in the topic configuration:

   ./bin/kafka-configs.sh --bootstrap-server <broker_address> --entity-type topics --entity-name __CruiseControlMetrics --alter --add-config cleanup.policy=delete

   Copy to Clipboard Toggle word wrap

Procedure

1. Generate an optimization proposal using a POST request to the /rebalance, /add_broker, or /remove_broker endpoint.

   Example request to /rebalance using default goals

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/rebalance'

   Copy to Clipboard Toggle word wrap

   The cached optimization proposal is immediately returned.

   Note

   If NotEnoughValidWindows is returned, Cruise Control has not yet recorded enough metrics data to generate an optimization proposal. Wait a few minutes and then resend the request.

   Example request to /rebalance using specified goals

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/rebalance?goals=RackAwareGoal,ReplicaCapacityGoal'

   Copy to Clipboard Toggle word wrap

   If the request satisfies the supplied goals, the cached optimization proposal is immediately returned. Otherwise, a new optimization proposal is generated using the supplied goals; this takes longer to calculate. You can enforce this behavior by adding the ignore_proposal_cache=true parameter to the request.

   Example request to /rebalance using specified goals without hard goals

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/rebalance?goals=RackAwareGoal,ReplicaCapacityGoal,ReplicaDistributionGoal&skip_hard_goal_check=true'

   Copy to Clipboard Toggle word wrap

   Example request to /add_broker that includes specified brokers

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/add_broker?brokerid=3,4'

   Copy to Clipboard Toggle word wrap

   The request includes the IDs of the new brokers only. For example, this request adds brokers with the IDs 3 and 4. Replicas are moved to the new brokers from existing brokers when rebalancing.

   Example request to /remove_broker that excludes specified brokers

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/remove_broker?brokerid=3,4'

   Copy to Clipboard Toggle word wrap

   The request includes the IDs of the brokers being excluded only. For example, this request excludes brokers with the IDs 3 and 4. Replicas are moved from the brokers being removed to other existing brokers when rebalancing.

   Note

   If a broker that is being removed has excluded topics, replicas are still moved.

2. Review the optimization proposal contained in the response. The properties describe the pending cluster rebalance operation.

   The proposal contains a high level summary of the proposed optimization, followed by summaries for each default optimization goal, and the expected cluster state after the proposal has executed.

   Pay particular attention to the following information:

   • The Cluster load after rebalance summary. If it meets your requirements, you should assess the impact of the proposed changes using the high level summary.
   • n inter-broker replica (y MB) moves indicates how much data will be moved across the network between brokers. The higher the value, the greater the potential performance impact on the Kafka cluster during the rebalance.
   • n intra-broker replica (y MB) moves indicates how much data will be moved within the brokers themselves (between disks). The higher the value, the greater the potential performance impact on individual brokers (although less than that of n inter-broker replica (y MB) moves).
   • The number of leadership moves. This has a negligible impact on the performance of the cluster during the rebalance.

Procedure

1. Send a POST request to the /rebalance, /add_broker, or /remove_broker endpoint with the dryrun=false parameter:

   If you used the /add_broker or /remove_broker endpoint to generate a proposal that included or excluded brokers, use the same endpoint to perform the rebalance with or without the specified brokers.

   Example request to /rebalance

   curl -X POST 'cruise-control-server:9090/kafkacruisecontrol/rebalance?dryrun=false'

   Copy to Clipboard Toggle word wrap

   Example request to /add_broker

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/add_broker?dryrun=false&brokerid=3,4'

   Copy to Clipboard Toggle word wrap

   Example request to /remove_broker

   curl -v -X POST 'cruise-control-server:9090/kafkacruisecontrol/remove_broker?dryrun=false&brokerid=3,4'

   Copy to Clipboard Toggle word wrap

   Cruise Control initiates the cluster rebalance and returns the optimization proposal.

2. Check the changes that are summarized in the optimization proposal. If the changes are not what you expect, you can stop the rebalance.

3. Check the progress of the cluster rebalance using the /user_tasks endpoint. The cluster rebalance in progress has a status of "Active".

   To view all cluster rebalance tasks executed on the Cruise Control server:

   curl 'cruise-control-server:9090/kafkacruisecontrol/user_tasks'
   
   USER TASK ID      CLIENT ADDRESS  START TIME     STATUS  REQUEST URL
   c459316f-9eb5-482f-9d2d-97b5a4cd294d  0:0:0:0:0:0:0:1       2020-06-01_16:10:29 UTC  Active      POST /kafkacruisecontrol/rebalance?dryrun=false
   445e2fc3-6531-4243-b0a6-36ef7c5059b4  0:0:0:0:0:0:0:1       2020-06-01_14:21:26 UTC  Completed   GET /kafkacruisecontrol/state?json=true
   05c37737-16d1-4e33-8e2b-800dee9f1b01  0:0:0:0:0:0:0:1       2020-06-01_14:36:11 UTC  Completed   GET /kafkacruisecontrol/state?json=true
   aebae987-985d-4871-8cfb-6134ecd504ab  0:0:0:0:0:0:0:1       2020-06-01_16:10:04 UTC

   Copy to Clipboard Toggle word wrap

4. To view the status of a particular cluster rebalance task, supply the user-task-ids parameter and the task ID:

   curl 'cruise-control-server:9090/kafkacruisecontrol/user_tasks?user_task_ids=c459316f-9eb5-482f-9d2d-97b5a4cd294d'

   Copy to Clipboard Toggle word wrap

1. Check that each broker being removed does not have any live partitions in its log (log.dirs).

   ls -l <LogDir> | grep -E '^d' | grep -vE '[a-zA-Z0-9.-]+\.[a-z0-9]+-delete$'

   Copy to Clipboard Toggle word wrap

   If a log directory does not match the regular expression \.[a-z0-9]-delete$, active partitions are still present. If you have active partitions, check the rebalance has finished or the configuration for the optimization proposal. You can run the proposal again. Make sure that there are no active partitions before moving on to the next step.

2. Stop the broker.

   ./bin/kafka-server-stop.sh

   Copy to Clipboard Toggle word wrap

3. Confirm that the broker has stopped.

   jcmd | grep kafka

   Copy to Clipboard Toggle word wrap