Red Hat Summit Red Hat Summit지원콘솔개발자평가판 시작연락처

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 21. Using Cruise Control for cluster rebalancing

Cruise Control is an open-source application designed to run alongside Kafka to help optimize use of cluster resources by doing the following:

  • Monitoring cluster workload
  • Rebalancing partitions based on predefined constraints

Cruise Control operations help with running a more balanced Kafka cluster that uses brokers more efficiently.

As Kafka clusters evolve, some brokers may become overloaded while others remain underutilized. Cruise Control addresses this imbalance by modeling resource utilization at the replica level—​including, CPU, disk, network load—​and generating optimization proposals (which you can approve or reject) for balanced partition assignments based on configurable optimization goals.

Optimization proposals are configured and generated using a KafkaRebalance resource. You can configure the resource using an annotation so that optimization proposals are approved automatically or manually.

Note

Streams for Apache Kafka provides example configuration files for Cruise Control.

21.1. Cruise Control components and features
링크 복사

Cruise Control comprises four main components:

Load Monitor
Load Monitor collects the metrics and analyzes cluster workload data.
Analyzer
Analyzer generates optimization proposals based on collected data and configured goals.
Anomaly Detector
Anomaly Detector identifies and reports irregularities in cluster behavior.
Executor
Executor applies approved optimization proposals to the cluster.
REST API

Cruise Control provides a REST API for client interactions, which Streams for Apache Kafka uses to support these features:

  • Generating optimization proposals from optimization goals
  • Rebalancing a Kafka cluster based on an optimization proposal
  • Changing topic replication factor
  • Reassigning partitions between JBOD disks
Note

Other Cruise Control features are not currently supported, including self healing, notifications, and write-your-own goals.

21.1.1. Optimization goals
링크 복사

Optimization goals define objectives for rebalancing, such as distributing topic replicas evenly across brokers.

They are categorized as follows:

  • Supported goals are a list of goals supported by the Cruise Control instance that can be used in its operations. By default, this list includes all goals included with Cruise Control. For a goal to be used in other categories, such as default or hard goals, it must first be listed in supported goals. To prevent a goal’s usage, remove it from this list.
  • Hard goals are preset and must be satisfied for a proposal to succeed.
  • Soft goals are preset goals with objectives that are prioritized during optimization as much as possible, without preventing a proposal from being created if all hard goals are satisfied.
  • Default goals refer to the goals used by default when generating proposals. They match the supported goals unless specifically set by the user.
  • Proposal-specific goals are a subset of supported goals configured for specific proposals.

Configure optimization goals in the Kafka and KafkaRebalance custom resources.

  • Kafka resource for supported, hard, and default goals.

    • Supported goals: Kafka.spec.cruiseControl.config.goals
    • Hard goals: Kafka.spec.cruiseControl.config.hard.goals
    • Default goals: Kafka.spec.cruiseControl.config.default.goals

  • KafkaRebalance resource for proposal-specific goals.

    • Proposal-specific goals: KafkaRebalance.spec.goals

21.1.1.1. Supported goals
링크 복사

Supported goals are predefined and available to use for generating Cruise Control optimization proposals. Goals not listed as supported goals cannot be used in Cruise Control operations. Some supported goals are preset as hard goals.

Configure supported goals in Kafka.spec.cruiseControl.config.goals:

  • To accept inherited supported goals, omit the goals property.
  • To modify supported goals, specify the goals in descending priority order in the goals property.

21.1.1.2. Hard and soft goals
링크 복사

Hard goals must be satisfied for optimization proposals to be generated. Soft goals are best-effort objectives that Cruise Control tries to meet after all hard goals are satisfied. The classification of hard and soft goals is fixed in Cruise Control code and cannot be changed.

Cruise Control first prioritizes satisfying hard goals, and then addresses soft goals in the order they are listed. A proposal meeting all hard goals is valid, even if it violates some soft goals.

For example, a soft goal might be to evenly distribute a topic’s replicas. Cruise Control continues to generate an optimization proposal even if the soft goal isn’t completely satisfied.

Configure hard goals in your Cruise Control deployment using Kafka.spec.cruiseControl.config.hard.goals:

  • To enforce all hard goals, omit the hard.goals property.
  • To specify hard goals, list them in hard.goals.
  • To exclude a hard goal, ensure it’s not in either default.goals or hard.goals.

Increasing the number of configured hard goals will reduce the likelihood of Cruise Control generating optimization proposals.

21.1.1.3. Default goals
링크 복사

Cruise Control uses default goals to generate an optimization proposal.

If default.goals is not specified in the Cruise Control deployment configuration, Streams for Apache Kafka configures default.goals to the list of supported goals specified in goals.

The optimization proposal based on this supported goals list is then generated and cached.

Configure default goals in Kafka.spec.cruiseControl.config.default.goals:

  • To use supported goals as default, omit the default.goals property.
  • To modify default goals, specify a subset of supported goals in the default.goals property.
    You can adjust the priority order in the default goals configuration.

21.1.1.4. Proposal-specific goals
링크 복사

Proposal-specific optimization goals support the creation of optimization proposals based on a specific list of goals. If proposal-specific goals are not set in the KafkaRebalance resource, then default goals are used

Configure proposal-specific goals in KafkaRebalance.spec.goals, specifying a subset of supported optimization goals for customization.

For example, you can optimize topic leader replica distribution across the Kafka cluster without considering disk capacity or utilization by defining a single proposal-specific goal.

21.1.1.5. Goals order of priority
링크 복사

Unless you change the Cruise Control deployment configuration, Streams for Apache Kafka inherits goals from Cruise Control, in descending priority order.

The following list shows supported goals inherited by Streams for Apache Kafka from Cruise Control in descending priority order. Goals labeled as hard are mandatory constraints that must be satisfied for optimization proposals.

  • RackAwareGoal (hard)
  • MinTopicLeadersPerBrokerGoal (hard)
  • ReplicaCapacityGoal (hard)
  • DiskCapacityGoal (hard)
  • NetworkInboundCapacityGoal (hard)
  • NetworkOutboundCapacityGoal (hard)
  • CpuCapacityGoal (hard)
  • ReplicaDistributionGoal
  • PotentialNwOutGoal
  • DiskUsageDistributionGoal
  • NetworkInboundUsageDistributionGoal
  • NetworkOutboundUsageDistributionGoal
  • CpuUsageDistributionGoal
  • TopicReplicaDistributionGoal
  • LeaderReplicaDistributionGoal
  • LeaderBytesInDistributionGoal
  • PreferredLeaderElectionGoal
  • IntraBrokerDiskCapacityGoal (hard)
  • IntraBrokerDiskUsageDistributionGoal

Resource distribution goals are subject to capacity limits on broker resources.

For more information on each optimization goal, see Goals in the Cruise Control Wiki.

Note

"Write your own" goals and Kafka assigner goals are not supported.

Example Kafka configuration for default and hard goals

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
  zookeeper:
    # ...
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 10000KB/s
      outboundNetwork: 10000KB/s
    config:
      #`default.goals` (superset) must also include all `hard.goals` (subset)
      default.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      # ...
Copy to Clipboard Toggle word wrap

Important

Ensure that the supported goals, default.goals, and (unless skipHardGoalCheck is set to true) proposal-specific spec.goals include all hard goals specified in hard.goals to avoid errors when generating optimization proposals. Hard goals must be included as a subset in the supported, default, and proposal-specific goals.

Example KafkaRebalance configuration for proposal-specific goals

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  goals:
    - RackAwareGoal
    - TopicReplicaDistributionGoal
  skipHardGoalCheck: true
Copy to Clipboard Toggle word wrap

21.1.1.6. Skipping hard goal checks
링크 복사

If skipHardGoalCheck: true is specified in the KafkaRebalance custom resource, Cruise Control does not verify that the proposal-specific goals include all the configured hard goals. This allows for more flexibility in generating optimization proposals, but may lead to proposals that do not satisfy all hard goals.

However, any hard goals included in the proposal-specific goals will still be treated as hard goals by Cruise Control, even with skipHardGoalCheck: true.

21.1.2. Optimization proposals
링크 복사

Optimization proposals are summaries of proposed changes based on the defined optimization goals, assessed in a specific order of priority. You can approve or reject proposals and rerun them with adjusted goals if needed.

With Cruise Control deployed for use in Streams for Apache Kafka, the process to generate and approve an optimization proposal is as follows:

  1. Create a KafkaRebalance resource specifying optimization goals and any specific configurations. This resource triggers Cruise Control to initiate the optimization proposal generation process.
  2. A Cruise Control Metrics Reporter runs in every Kafka broker, collecting raw metrics and publishing them to a dedicated Kafka topic (strimzi.cruisecontrol.metrics). Metrics for brokers, topics, and partitions are aggregated, sampled, and stored in other topics automatically created when Cruise Control is deployed.
  3. Load Monitor collects, processes, and stores the metrics as a workload model--including CPU, disk, and network utilization data—​which is used by the Analyzer and Anomaly Detector.
  4. Anomaly Detector continuously monitors the health and performance of the Kafka cluster, checking for things like broker failures or disk capacity issues, that could impact cluster stability.
  5. Analyzer creates optimization proposals based on the workload model from the Load Monitor. Based on configured goals and capacities, it generates an optimization proposal for balancing partitions across brokers. Through the REST API, a summary of the proposal is reflected in the status of the KafkaRebalance resource.
  6. The optimization proposal is approved or rejected (manually or automatically) based on its alignment with cluster management goals.
  7. If approved, the Executor applies the optimization proposal to rebalance the Kafka cluster. This involves reassigning partitions and redistributing workload across brokers according to the approved proposal.

Figure 21.1. Cruise Control optimization process

Cruise Control process
View larger image
Cruise Control process

Optimization proposals comprise a list of partition reassignment mappings. When you approve a proposal, the Cruise Control server applies these partition reassignments to the Kafka cluster.

A partition reassignment consists of either of the following types of operations:

  • Partition movement: Involves transferring the partition replica and its data to a new location. Partition movements can take one of two forms:

    • Inter-broker movement: The partition replica is moved to a log directory on a different broker.
    • Intra-broker movement: The partition replica is moved to a different log directory on the same broker.
  • Leadership movement: Involves switching the leader of the partition’s replicas.

Cruise Control issues partition reassignments to the Kafka cluster in batches. The performance of the cluster during the rebalance is affected by the number and magnitude of each type of movement contained in each batch.

21.1.2.1. Rebalancing modes
링크 복사

Proposals for rebalances can be generated in four modes, which are specified using the spec.mode property of the KafkaRebalance custom resource.

full mode
The full mode runs a full rebalance by moving replicas across all the brokers in the cluster. This is the default mode if the spec.mode property is not defined in the KafkaRebalance custom resource.
add-brokers mode
The add-brokers mode is used after scaling up a Kafka cluster by adding one or more brokers. Normally, after scaling up a Kafka cluster, new brokers are used to host only the partitions of newly created topics. If no new topics are created, the newly added brokers are not used and the existing brokers remain under the same load. By using the add-brokers mode immediately after adding brokers to the cluster, the rebalancing operation moves replicas from existing brokers to the newly added brokers. You specify the new brokers as a list using the spec.brokers property of the KafkaRebalance custom resource.
remove-brokers mode
The remove-brokers mode is used before scaling down a Kafka cluster by removing one or more brokers. The remove-brokers mode moves replicas off the brokers that are going to be removed. When these brokers are not hosting replicas anymore, you can safely run the scaling down operation. You specify the brokers you’re removing as a list in the spec.brokers property in the KafkaRebalance custom resource.
remove-disks mode
The remove-disks mode is used specifically to reassign partitions between JBOD disks used for storage on the same broker. You specify a list of broker IDs with corresponding volume IDs for partition reassignment.
Note

Brokers are shut down even if they host replicas when checks are skipped on scale-down operations.

In general, use the full rebalance mode to rebalance a Kafka cluster by spreading the load across brokers. Use the add-brokers and remove-brokers modes only if you want to scale your cluster up or down and rebalance the replicas accordingly.

The procedure to run a rebalance is actually the same across the three different modes. The only difference is with specifying a mode through the spec.mode property and, if needed, listing brokers that have been added or will be removed through the spec.brokers property.

21.1.2.2. The results of an optimization proposal
링크 복사

When an optimization proposal is generated, a summary and broker load is returned.

Summary
The summary is contained in the KafkaRebalance resource. The summary provides an overview of the proposed cluster rebalance and indicates the scale of the changes involved. A summary of a successfully generated optimization proposal is contained in the Status.optimizationResult property of the KafkaRebalance resource. The information provided is a summary of the full optimization proposal.
Broker load
The broker load is stored in a ConfigMap that contains data as a JSON string. The broker load shows before and after values for the proposed rebalance, so you can see the impact on each of the brokers in the cluster.

21.1.2.3. Manually approving or rejecting an optimization proposal
링크 복사

An optimization proposal summary shows the proposed scope of changes.

You can use the name of the KafkaRebalance resource to return a summary from the command line.

Returning an optimization proposal summary

oc describe kafkarebalance <kafka_rebalance_resource_name> -n <namespace>
Copy to Clipboard Toggle word wrap

You can also use the jq command line JSON parser tool.

Returning an optimization proposal result using jq

oc get kafkarebalance <kafka_rebalance_resource_name> -n <namespace> -o json | jq '.status.optimizationResult'
Copy to Clipboard Toggle word wrap

Use the summary to decide whether to approve or reject an optimization proposal.

Approving an optimization proposal
You approve the optimization proposal by setting the strimzi.io/rebalance annotation of the KafkaRebalance resource to approve. Cruise Control applies the proposal to the Kafka cluster and starts a cluster rebalance operation.
Rejecting an optimization proposal
If you choose not to approve an optimization proposal, you can change the optimization goals or update any of the rebalance performance tuning options, and then generate another proposal. You can generate a new optimization proposal for a KafkaRebalance resource by setting the strimzi.io/rebalance annotation to refresh.

Use optimization proposals to assess the movements required for a rebalance. For example, a summary describes inter-broker and intra-broker movements. Inter-broker rebalancing moves data between separate brokers. Intra-broker rebalancing moves data between disks on the same broker when you are using a JBOD storage configuration. Such information can be useful even if you don’t go ahead and approve the proposal.

You might reject an optimization proposal, or delay its approval, because of the additional load on a Kafka cluster when rebalancing. If the proposal is delayed for too long, the cluster load may change significantly, so it may be better to request a new proposal.

In the following example, the proposal suggests the rebalancing of data between separate brokers. The rebalance involves the movement of 55 partition replicas, totaling 12MB of data, across the brokers. The proposal will also move 24 partition leaders to different brokers. This requires a change to the cluster metadata, which has a low impact on performance.

The balancedness scores are measurements of the overall balance of the Kafka cluster before and after the optimization proposal is approved. A balancedness score is based on optimization goals. If all goals are satisfied, the score is 100. The score is reduced for each goal that will not be met. Compare the balancedness scores to see whether the Kafka cluster is less balanced than it could be following a rebalance.

Example optimization proposal summary

Name:         my-rebalance
Namespace:    myproject
Labels:       strimzi.io/cluster=my-cluster
Annotations:  API Version:  kafka.strimzi.io/v1alpha1
Kind:         KafkaRebalance
Metadata:
# ...
Status:
  Conditions:
    Last Transition Time:  2022-04-05T14:36:11.900Z
    Status:                ProposalReady
    Type:                  State
  Observed Generation:     1
  Optimization Result:
    Data To Move MB:  0
    Excluded Brokers For Leadership:
    Excluded Brokers For Replica Move:
    Excluded Topics:
    Intra Broker Data To Move MB:         12
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 24
    Num Replica Movements:                55
    On Demand Balancedness Score After:   82.91290759174306
    On Demand Balancedness Score Before:  78.01176356230222
    Recent Windows:                       5
  Session Id:                             a4f833bd-2055-4213-bfdd-ad21f95bf184
Copy to Clipboard Toggle word wrap

Though the inter-broker movement of partition replicas has a high impact on performance, the total amount of data is not large. If the total data was much larger, you could reject the proposal, or time when to approve the rebalance to limit the impact on the performance of the Kafka cluster.

Rebalance performance tuning options can help reduce the impact of data movement. If you can extend the rebalance period, you can divide the rebalance into smaller batches. Fewer data movements at a single time reduces the load on the cluster.

21.1.2.4. Optimization proposal summary properties
링크 복사

The following table explains the properties contained in the optimization proposal’s summary.

Expand
Table 21.1. Properties contained in an optimization proposal summary
JSON propertyDescription

numIntraBrokerReplicaMovements

The total number of partition replicas that will be transferred between the disks of the cluster’s brokers.

Performance impact during rebalance operation: Relatively high, but lower than numReplicaMovements.

excludedBrokersForLeadership

Not yet supported. An empty list is returned.

numReplicaMovements

The number of partition replicas that will be moved between separate brokers.

Performance impact during rebalance operation: Relatively high.

onDemandBalancednessScoreBefore
onDemandBalancednessScoreAfter

A measurement of the overall balancedness of a Kafka Cluster, before and after the optimization proposal was generated.

The score is calculated by subtracting the sum of the BalancednessScore of each violated soft goal from 100. Cruise Control assigns a BalancednessScore to every optimization goal based on several factors, including priority—​the goal’s position in the list of default.goals or proposal-specific goals.

The Before score is based on the workload model of the Kafka cluster. The After score is based on the predicted workload model after applying the generated optimization proposal.

intraBrokerDataToMoveMB

The sum of the size of each partition replica that will be moved between disks on the same broker (see also numIntraBrokerReplicaMovements).

Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see dataToMoveMB).

recentWindows

The number of metrics windows upon which the optimization proposal is based.

dataToMoveMB

The sum of the size of each partition replica that will be moved to a separate broker (see also numReplicaMovements).

Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete.

monitoredPartitionsPercentage

The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of excludedTopics.

excludedTopics

If you specified a regular expression in the spec.excludedTopicsRegex property in the KafkaRebalance resource, all topic names matching that expression are listed here. These topics are excluded from the calculation of partition replica/leader movements in the optimization proposal.

numLeaderMovements

The number of partitions whose leaders will be switched to different replicas. This involves a change to ZooKeeper configuration.

Performance impact during rebalance operation: Relatively low.

excludedBrokersForReplicaMove

Not yet supported. An empty list is returned.

21.1.2.5. Automatically approving an optimization proposal
링크 복사

To save time, you can automate the process of approving optimization proposals. With automation, when you generate an optimization proposal it goes straight into a cluster rebalance.

To enable the optimization proposal auto-approval mechanism, create the KafkaRebalance resource with the strimzi.io/rebalance-auto-approval annotation set to true. If the annotation is not set or set to false, the optimization proposal requires manual approval.

Example rebalance request with auto-approval mechanism enabled

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
  annotations:
    strimzi.io/rebalance-auto-approval: "true"
spec:
  mode: # any mode
  # ...
Copy to Clipboard Toggle word wrap

You can still check the status when automatically approving an optimization proposal. The status of the KafkaRebalance resource moves to Ready when the rebalance is complete.

21.1.2.6. Comparing broker load data
링크 복사

Broker load data provides insights into current and anticipated usage of resources following a rebalance. The data is stored in a ConfigMap (with the same name as the KafkaRebalance resource) as a JSON formatted string

When a Kafka rebalance proposal reaches the ProposalReady state, Streams for Apache Kafka creates a ConfigMap (named after the KafkaRebalance custom resource) containing a JSON string of broker metrics generated from Cruise Control. Each broker has a set of key metrics represented by three values:

  • The current metric value before the optimization proposal is applied
  • The expected metric value after applying the proposal
  • The difference between the two values (after minus before)

This ConfigMap remains accessible even after the rebalance completes.

To view this data from the command line, use the ConfigMap name.

Returning ConfigMap data

oc describe configmaps <my_rebalance_configmap_name> -n <namespace>
Copy to Clipboard Toggle word wrap

You can also use the jq command line JSON parser tool to extract the JSON string.

Extracting the JSON string from the ConfigMap using jq

oc get configmaps <my_rebalance_configmap_name> -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.'
Copy to Clipboard Toggle word wrap

Expand
Table 21.2. Properties captured in the config map
JSON propertyDescription

leaders

The number of replicas on this broker that are partition leaders.

replicas

The number of replicas on this broker.

cpuPercentage

The CPU utilization as a percentage of the defined capacity.

diskUsedPercentage

The disk utilization as a percentage of the defined capacity.

diskUsedMB

The absolute disk usage in MB.

networkOutRate

The total network output rate for the broker.

leaderNetworkInRate

The network input rate for all partition leader replicas on this broker.

followerNetworkInRate

The network input rate for all follower replicas on this broker.

potentialMaxNetworkOutRate

The hypothetical maximum network output rate that would be realized if this broker became the leader of all the replicas it currently hosts.

21.1.2.7. Adjusting the cached proposal refresh rate
링크 복사

Cruise Control maintains a cached optimization proposal based on the configured default optimization goals. This proposal is generated from the workload model and updated every 15 minutes to reflect the current state of the Kafka cluster. When you generate an optimization proposal using the default goals, Cruise Control returns the latest cached version.

For clusters with rapidly changing workloads, you may want to shorten the refresh interval to ensure the optimization proposal reflects the most recent state. However, reducing the interval increases the load on the Cruise Control server. To adjust the refresh rate, modify the proposal.expiration.ms setting in the Cruise Control deployment configuration.

21.1.3. Tuning options for rebalances
링크 복사

Configuration options allow you to fine-tune cluster rebalance performance. These settings control the movement of partition replicas and leadership, as well as the bandwidth allocated for rebalances.

21.1.3.1. Selecting replica movement strategies
링크 복사

Cluster rebalance performance is also influenced by the replica movement strategy that is applied to the batches of partition reassignment commands. By default, Cruise Control uses the BaseReplicaMovementStrategy, which applies the reassignments in the order they were generated. However, this strategy could lead to the delay of other partition reassignments if large partition reassignments are generated then ordered first.

Cruise Control provides four alternative replica movement strategies that can be applied to optimization proposals:

  • PrioritizeSmallReplicaMovementStrategy: Reassign smaller partitions first.
  • PrioritizeLargeReplicaMovementStrategy: Reassign larger partitions first.
  • PostponeUrpReplicaMovementStrategy: Prioritize partitions without out-of-sync replicas.
  • PrioritizeMinIsrWithOfflineReplicasStrategy: Prioritize reassignments for partitions at or below their minimum in-sync replicas (MinISR) with offline replicas.
    Set cruiseControl.config.concurrency.adjuster.min.isr.check.enabled to true in the Kafka resource to enable this strategy.

These strategies can be configured as a sequence. The first strategy attempts to compare two partition reassignments using its internal logic. If the reassignments are equivalent, then it passes them to the next strategy in the sequence to decide the order, and so on.

21.1.3.2. Intra-broker disk balancing
링크 복사

Intra-broker balancing shifts data between disks on the same broker, useful for deployments with JBOD storage and multiple disks. This type of balancing incurs less network overhead than inter-broker balancing.

Note

If you are using JBOD storage with a single disk, intra-broker disk balancing will result in a proposal with 0 partition movements since there are no disks to balance.

To enable intra-broker balancing, set rebalanceDisk to true in KafkaRebalance.spec. When this is enabled, do not specify a goals field, as Cruise Control will automatically configure intra-broker goals and disregard inter-broker goals. Cruise Control does not perform inter-broker and intra-broker balancing at the same time.

21.1.3.3. Rebalance tuning
링크 복사

You can set the following rebalance tuning options when configuring Cruise Control or individual rebalances:

  • Set Cruise Control server configurations in Kafka.spec.cruiseControl.config in the Kafka resource.
  • Set proposal-specific configurations in KafkaRebalance.spec in the KafkaRebalance resource.
Expand
Table 21.3. Rebalance configuration tuning properties
Cruise Control propertiesKafkaRebalance propertiesDefaultDescription

num.concurrent.partition.movements.per.broker

concurrentPartitionMovementsPerBroker

5

The maximum number of inter-broker partition movements in each partition reassignment batch

num.concurrent.intra.broker.partition.movements

concurrentIntraBrokerPartitionMovements

2

The maximum number of intra-broker partition movements in each partition reassignment batch

num.concurrent.leader.movements

concurrentLeaderMovements

1000

The maximum number of partition leadership changes in each partition reassignment batch

default.replication.throttle

replicationThrottle

Null (no limit)

The bandwidth (in bytes per second) to assign to partition reassignment

default.replica.movement.strategies

replicaMovementStrategies

BaseReplicaMovementStrategy

The list of strategies (in priority order) used to determine the order in which partition reassignment commands are executed for generated proposals. For the server setting, use a comma separated string with the fully qualified names of the strategy class (add com.linkedin.kafka.cruisecontrol.executor.strategy. to the start of each class name). For the KafkaRebalance resource setting use a YAML array of strategy class names.

-

rebalanceDisk

false

Enables intra-broker disk balancing, which balances disk space utilization between disks on the same broker. Only applies to Kafka deployments that use JBOD storage with multiple disks.

Changing the default settings affects the length of time that the rebalance takes to complete, as well as the load placed on the Kafka cluster during the rebalance. Using lower values reduces the load but increases the amount of time taken, and vice versa.

21.2. Deploying Cruise Control with Kafka
링크 복사

Configure a Kafka resource to deploy Cruise Control alongside a Kafka cluster. You can use the cruiseControl properties of the Kafka resource to configure the deployment. Deploy one instance of Cruise Control per Kafka cluster.

Use goals configuration in the Cruise Control config to specify optimization goals for generating optimization proposals. You can use brokerCapacity to change the default capacity limits for goals related to resource distribution. If brokers are running on nodes with heterogeneous network resources, you can use overrides to set network capacity limits for each broker.

If an empty object ({}) is used for the cruiseControl configuration, all properties use their default values.

Streams for Apache Kafka provides example configuration files, which include Kafka custom resources with Cruise Control configuration. For more information on the configuration options for Cruise Control, see the Streams for Apache Kafka Custom Resource API Reference.

Prerequisites

Procedure

  1. Edit the cruiseControl property for the Kafka resource.

    The properties you can configure are shown in this example configuration: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  # ...
  cruiseControl:
    brokerCapacity:
    1 
    
      inboundNetwork: 10000KB/s
      outboundNetwork: 10000KB/s
      overrides:
    2 
    
      - brokers: [0]
        inboundNetwork: 20000KiB/s
        outboundNetwork: 20000KiB/s
      - brokers: [1, 2]
        inboundNetwork: 30000KiB/s
        outboundNetwork: 30000KiB/s
      # ...
    config:
    3 
    
      # Note that `default.goals` (superset) must also include all `hard.goals` (subset)
      default.goals: >
    4 
    
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal
        # ...
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal
        # ...
      cpu.balance.threshold: 1.1
      metadata.max.age.ms: 300000
      send.buffer.bytes: 131072
      webserver.http.cors.enabled: true
    5 
    
      webserver.http.cors.origin: "*"
      webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type"
      # ...
    resources:
    6 
    
      requests:
        cpu: 1
        memory: 512Mi
      limits:
        cpu: 2
        memory: 2Gi
    logging:
    7 
    
        type: inline
        loggers:
          rootLogger.level: INFO
    template:
    8 
    
      pod:
        metadata:
          labels:
            label1: value1
        securityContext:
          runAsUser: 1000001
          fsGroup: 0
        terminationGracePeriodSeconds: 120
    readinessProbe:
    9 
    
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    metricsConfig:
    10 
    
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml
# ...
    Copy to Clipboard Toggle word wrap
    1
    Capacity limits for broker resources.
    2
    Overrides set network capacity limits for specific brokers when running on nodes with heterogeneous network resources.
    3
    Cruise Control configuration. Standard Cruise Control configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka.
    4
    Optimization goals configuration, which can include configuration for default optimization goals (default.goals), supported optimization goals (goals), and hard goals (hard.goals).
    5
    CORS enabled and configured for read-only access to the Cruise Control API.
    6
    Requests for reservation of supported resources, currently cpu and memory, and limits to specify the maximum resources that can be consumed.
    7
    Cruise Control loggers and log levels added directly (inline) or indirectly (external) through a ConfigMap. A custom Log4j configuration must be placed under the log4j.properties key in the ConfigMap. Cruise Control has a single logger named rootLogger.level. You can set the log level to INFO, ERROR, WARN, TRACE, DEBUG, FATAL or OFF.
    8
    Template customization. Here a pod is scheduled with additional security attributes.
    9
    Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).
    10
    Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter (the default metrics exporter).

  2. Create or update the resource: 

    oc apply -f <kafka_configuration_file>
    Copy to Clipboard Toggle word wrap

  3. Check the status of the deployment: 

    oc get deployments -n <my_cluster_operator_namespace>
    Copy to Clipboard Toggle word wrap

    Output shows the deployment name and readiness

    NAME                      READY  UP-TO-DATE  AVAILABLE
my-cluster-cruise-control 1/1    1           1
    Copy to Clipboard Toggle word wrap

    my-cluster is the name of the Kafka cluster.

    READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

21.2.1. Auto-created Cruise Control topics
링크 복사

The following table shows the three topics that are automatically created when Cruise Control is deployed. These topics are required for Cruise Control to work properly and must not be deleted or changed. You can change the name of the topic using the specified configuration option.

Expand
Table 21.4. Topics created when Cruise Control is deployed
Auto-created topic configurationDefault topic nameCreated byFunction

metric.reporter.topic

strimzi.cruisecontrol.metrics

Streams for Apache Kafka Metrics Reporter

Stores the raw metrics from the Metrics Reporter in each Kafka broker.

partition.metric.sample.store.topic

strimzi.cruisecontrol.partitionmetricsamples

Cruise Control

Stores the derived metrics for each partition. These are created by the Metric Sample Aggregator.

broker.metric.sample.store.topic

strimzi.cruisecontrol.modeltrainingsamples

Cruise Control

Stores the metrics samples used to create the Cluster Workload Model.

To prevent the removal of records that are needed by Cruise Control, log compaction is disabled in the auto-created topics.

Note

If the names of the auto-created topics are changed in a Kafka cluster that already has Cruise Control enabled, the old topics will not be deleted and should be manually removed.

What to do next

After configuring and deploying Cruise Control, you can generate optimization proposals.

21.3. Generating optimization proposals
링크 복사

When you create or update a KafkaRebalance resource, Cruise Control generates an optimization proposal for the Kafka cluster based on a set of optimization goals. Analyze the information in the optimization proposal and decide whether to approve it. You can use the results of the optimization proposal to rebalance your Kafka cluster.

This procedure covers using the following modes for generating optimization proposals related to rebalances:

  • full (default)
  • add-brokers
  • remove-brokers

The mode you use depends on whether you are rebalancing across all the brokers already running in the Kafka cluster; or you want to rebalance after scaling up or before scaling down your Kafka cluster. For more information, see Rebalancing modes with broker scaling.

Prerequisites

  • You have deployed Cruise Control to your Streams for Apache Kafka cluster.
  • You have configured optimization goals and, optionally, capacity limits on broker resources.

For more information on configuring Cruise Control, see Section 21.2, “Deploying Cruise Control with Kafka”.

Procedure

  1. Create a KafkaRebalance resource and specify the appropriate mode.

    full mode (default)

    To use the default optimization goals defined in the Kafka resource, leave the spec property empty. Cruise Control rebalances a Kafka cluster in full mode by default.

    Example configuration with full rebalancing by default

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec: {}
    Copy to Clipboard Toggle word wrap

    You can also run a full rebalance by specifying the full mode through the spec.mode property.

    Example configuration specifying full mode

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: full
    Copy to Clipboard Toggle word wrap

    add-brokers mode

    If you want to rebalance a Kafka cluster after scaling up, specify the add-brokers mode.

    In this mode, existing replicas are moved to the newly added brokers. You need to specify the brokers as a list.

    Example configuration specifying add-brokers mode

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: add-brokers
  brokers: [3, 4]
    1
    Copy to Clipboard Toggle word wrap

    1
    List of newly added brokers added by the scale up operation. This property is mandatory.
    remove-brokers mode

    If you want to rebalance a Kafka cluster before scaling down, specify the remove-brokers mode.

    In this mode, replicas are moved off the brokers that are going to be removed. You need to specify the brokers that are being removed as a list.

    Example configuration specifying remove-brokers mode

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: remove-brokers
  brokers: [3, 4]
    1
    Copy to Clipboard Toggle word wrap

    1
    List of brokers to be removed by the scale down operation. This property is mandatory.
    Note

    The following steps and the steps to approve or stop a rebalance are the same regardless of the rebalance mode you are using.

  2. To configure proposal-specific optimization goals instead of using the default goals, add the goals property and enter one or more goals.

    In the following example, rack awareness and replica capacity are configured as proposal-specific optimization goals: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  goals:
    - RackAwareGoal
    - ReplicaCapacityGoal
    Copy to Clipboard Toggle word wrap

  3. To ignore the configured hard goals, add the skipHardGoalCheck: true property: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  goals:
    - RackAwareGoal
    - ReplicaCapacityGoal
  skipHardGoalCheck: true
    Copy to Clipboard Toggle word wrap

  4. (Optional) To approve the optimization proposal automatically, set the strimzi.io/rebalance-auto-approval annotation to true: 

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
  annotations:
    strimzi.io/rebalance-auto-approval: "true"
spec:
  goals:
    - RackAwareGoal
    - ReplicaCapacityGoal
  skipHardGoalCheck: true
    Copy to Clipboard Toggle word wrap

  5. Create or update the resource: 

    oc apply -f <kafka_rebalance_configuration_file>
    Copy to Clipboard Toggle word wrap

    The Cluster Operator requests the optimization proposal from Cruise Control. This might take a few minutes depending on the size of the Kafka cluster.

  6. If you used the automatic approval mechanism, wait for the status of the optimization proposal to change to Ready. If you haven’t enabled the automatic approval mechanism, wait for the status of the optimization proposal to change to ProposalReady: 

    oc get kafkarebalance -o wide -w -n <namespace>
    Copy to Clipboard Toggle word wrap
    PendingProposal
    A PendingProposal status means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready.
    ProposalReady
    A ProposalReady status means the optimization proposal is ready for review and approval.

    When the status changes to ProposalReady, the optimization proposal is ready to approve.

  7. Review the optimization proposal.

    The optimization proposal is contained in the Status.Optimization Result property of the KafkaRebalance resource. 

    oc describe kafkarebalance <kafka_rebalance_resource_name>
    Copy to Clipboard Toggle word wrap

    Example optimization proposal

    Status:
  Conditions:
    Last Transition Time:  2020-05-19T13:50:12.533Z
    Status:                ProposalReady
    Type:                  State
  Observed Generation:     1
  Optimization Result:
    Data To Move MB:  0
    Excluded Brokers For Leadership:
    Excluded Brokers For Replica Move:
    Excluded Topics:
    Intra Broker Data To Move MB:         0
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 0
    Num Replica Movements:                26
    On Demand Balancedness Score After:   81.8666802863978
    On Demand Balancedness Score Before:  78.01176356230222
    Recent Windows:                       1
  Session Id:                             05539377-ca7b-45ef-b359-e13564f1458c
    Copy to Clipboard Toggle word wrap

    The properties in the Optimization Result section describe the pending cluster rebalance operation. For descriptions of each property, see Contents of optimization proposals.

Insufficient CPU capacity

If a Kafka cluster is overloaded in terms of CPU utilization, you might see an insufficient CPU capacity error in the KafkaRebalance status. It’s worth noting that this utilization value is unaffected by the excludedTopics configuration. Although optimization proposals will not reassign replicas of excluded topics, their load is still considered in the utilization calculation.

Example CPU utilization error

com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException:
[CpuCapacityGoal] Insufficient capacity for cpu (Utilization 615.21, Allowed Capacity 420.00, Threshold: 0.70). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0.
Copy to Clipboard Toggle word wrap

Note

The error shows CPU capacity as a percentage rather than the number of CPU cores. For this reason, it does not directly map to the number of CPUs configured in the Kafka custom resource. It is like having a single virtual CPU per broker, which has the cycles of the CPUs configured in Kafka.spec.kafka.resources.limits.cpu. This has no effect on the rebalance behavior, since the ratio between CPU utilization and capacity remains the same.

What to do next

Section 21.4, “Approving optimization proposals”

21.4. Approving optimization proposals
링크 복사

You can approve an optimization proposal generated by Cruise Control, if its status is ProposalReady. Cruise Control will then apply the optimization proposal to the Kafka cluster, reassigning partitions to brokers and changing partition leadership.

Important

This is not a dry run. Before you approve an optimization proposal, you must:

Prerequisites

Procedure

Perform these steps for the optimization proposal that you want to approve.

  1. Unless the optimization proposal is newly generated, check that it is based on current information about the state of the Kafka cluster. To do so, annotate the KafkaRebalance resource to refresh the optimization proposal and make sure it uses the latest cluster metrics: 

    oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"
    Copy to Clipboard Toggle word wrap

  2. Wait for the status of the optimization proposal to change to ProposalReady: 

    oc get kafkarebalance -o wide -w -n <namespace>
    Copy to Clipboard Toggle word wrap
    PendingProposal
    A PendingProposal status means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready.
    ProposalReady
    A ProposalReady status means the optimization proposal is ready for review and approval.

    When the status changes to ProposalReady, the optimization proposal is ready to approve.

  3. Annotate the KafkaRebalance resource to approve the optimization proposal: 

    oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="approve"
    Copy to Clipboard Toggle word wrap
  4. The Cluster Operator detects the annotated resource and instructs Cruise Control to rebalance the Kafka cluster.

  5. Wait for the status of the optimization proposal to change to Ready: 

    oc get kafkarebalance -o wide -w -n <namespace>
    Copy to Clipboard Toggle word wrap
    Rebalancing
    A Rebalancing status means the rebalancing is in progress.
    Ready
    A Ready status means the rebalance is complete.
    NotReady
    A NotReady status means an error occurred—​see Fixing problems with a KafkaRebalance resource.

    When the status changes to Ready, the rebalance is complete.

    To use the same KafkaRebalance custom resource to generate another optimization proposal, apply the refresh annotation to the custom resource. This moves the custom resource to the PendingProposal or ProposalReady state. You can then review the optimization proposal and approve it, if desired.

21.5. Stopping rebalances
링크 복사

Once started, a cluster rebalance operation might take some time to complete and affect the overall performance of the Kafka cluster.

If you want to stop a cluster rebalance operation that is in progress, apply the stop annotation to the KafkaRebalance custom resource. This instructs Cruise Control to finish the current batch of partition reassignments and then stop the rebalance. When the rebalance has stopped, completed partition reassignments have already been applied; therefore, the state of the Kafka cluster is different when compared to prior to the start of the rebalance operation. If further rebalancing is required, you should generate a new optimization proposal.

Note

The performance of the Kafka cluster in the intermediate (stopped) state might be worse than in the initial state.

Prerequisites

  • The status of the KafkaRebalance custom resource is Rebalancing.

Procedure

  1. Annotate the KafkaRebalance resource to stop the rebalance: 

    oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="stop"
    Copy to Clipboard Toggle word wrap

  2. Check the status of the KafkaRebalance resource: 

    oc describe kafkarebalance <kafka_rebalance_resource_name>
    Copy to Clipboard Toggle word wrap
  3. Wait until the status changes to Stopped.

21.6. Troubleshooting and refreshing rebalances
링크 복사

When creating a KafkaRebalance resource or interacting with Cruise Control, errors are reported in the resource status, along with guidance on how to fix them. In such cases, the resource transitions to the NotReady state.

To continue with a cluster rebalance operation, you must rectify any configuration issues in the KafkaRebalance resource or address any problems with the Cruise Control deployment.

Common issues include the following:

  • Misconfigured parameters in the KafkaRebalance resource.
  • The strimzi.io/cluster label for specifying the Kafka cluster in the KafkaRebalance resource is missing.
  • The Cruise Control server is not deployed as the cruiseControl property in the Kafka resource is missing.
  • The Cruise Control server is not reachable.

After fixing any issues, you need to add the refresh annotation to the KafkaRebalance resource. During a “refresh”, a new optimization proposal is requested from the Cruise Control server.

Prerequisites

Procedure

  1. Get information about the error from the KafkaRebalance status: 

    oc describe kafkarebalance <kafka_rebalance_resource_name>
    Copy to Clipboard Toggle word wrap

  2. Attempt to resolve the issue by annotating the KafkaRebalance resource to refresh the proposal: 

    oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"
    Copy to Clipboard Toggle word wrap

  3. Check the status of the KafkaRebalance resource: 

    oc describe kafkarebalance <kafka_rebalance_resource_name>
    Copy to Clipboard Toggle word wrap
  4. Wait until the status changes to PendingProposal, or directly to ProposalReady.
맨 위로 이동
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2025 Red Hat