Chapter 8. Cruise Control for cluster rebalancing
You can deploy Cruise Control to your AMQ Streams cluster and use it to rebalance the Kafka cluster.
Cruise Control is an open source system for automating Kafka operations, such as monitoring cluster workload, rebalancing a cluster based on predefined constraints, and detecting and fixing anomalies. It consists of four main components—the Load Monitor, the Analyzer, the Anomaly Detector, and the Executor—and a REST API for client interactions. AMQ Streams utilizes the REST API to support the following Cruise Control features:
- Generating optimization proposals from multiple optimization goals.
- Rebalancing a Kafka cluster based on an optimization proposal.
Other Cruise Control features are not currently supported, including anomaly detection, notifications, write-your-own goals, and changing the topic replication factor.
Example YAML files for Cruise Control are provided in examples/cruise-control/
.
8.1. Why use Cruise Control?
Cruise Control reduces the time and effort involved in running an efficient and balanced Kafka cluster.
A typical cluster can become unevenly loaded over time. Partitions that handle large amounts of message traffic might be unevenly distributed across the available brokers. To rebalance the cluster, administrators must monitor the load on brokers and manually reassign busy partitions to brokers with spare capacity.
Cruise Control automates the cluster rebalancing process. It constructs a workload model of resource utilization for the cluster—based on CPU, disk, and network load—and generates optimization proposals (that you can approve or reject) for more balanced partition assignments. A set of configurable optimization goals is used to calculate these proposals.
When you approve an optimization proposal, Cruise Control applies it to your Kafka cluster. When the cluster rebalancing operation is complete, the broker pods are used more effectively and the Kafka cluster is more evenly balanced.
Additional resources
8.2. Optimization goals overview
To rebalance a Kafka cluster, Cruise Control uses optimization goals to generate optimization proposals, which you can approve or reject.
Optimization goals are constraints on workload redistribution and resource utilization across a Kafka cluster. AMQ Streams supports most of the optimization goals developed in the Cruise Control project. The supported goals, in the default descending order of priority, are as follows:
- Rack-awareness
- Minimum number of leader replicas per broker for a set of topics
- Replica capacity
- Capacity: Disk capacity, Network inbound capacity, Network outbound capacity, CPU capacity
- Replica distribution
- Potential network output
Resource distribution: Disk utilization distribution, Network inbound utilization distribution, Network outbound utilization distribution, CPU utilization distribution
NoteThe resource distribution goals are controlled using capacity limits on broker resources.
- Leader bytes-in rate distribution
- Topic replica distribution
- Leader replica distribution
- Preferred leader election
For more information on each optimization goal, see Goals in the Cruise Control Wiki.
Intra-broker disk goals, "Write your own" goals, and Kafka assigner goals are not yet supported.
Goals configuration in AMQ Streams custom resources
You configure optimization goals in Kafka
and KafkaRebalance
custom resources. Cruise Control has configurations for hard optimization goals that must be satisfied, as well as master, default, and user-provided optimization goals. Optimization goals for resource distribution (disk, network inbound, network outbound, and CPU) are subject to capacity limits on broker resources.
The following sections describe each goal configuration in more detail.
Hard goals and soft goals
Hard goals are goals that must be satisfied in optimization proposals. Goals that are not configured as hard goals are known as soft goals. You can think of soft goals as best effort goals: they do not need to be satisfied in optimization proposals, but are included in optimization calculations. An optimization proposal that violates one or more soft goals, but satisfies all hard goals, is valid.
Cruise Control will calculate optimization proposals that satisfy all the hard goals and as many soft goals as possible (in their priority order). An optimization proposal that does not satisfy all the hard goals is rejected by Cruise Control and not sent to the user for approval.
For example, you might have a soft goal to distribute a topic’s replicas evenly across the cluster (the topic replica distribution goal). Cruise Control will ignore this goal if doing so enables all the configured hard goals to be met.
In Cruise Control, the following master optimization goals are preset as hard goals:
RackAwareGoal; MinTopicLeadersPerBrokerGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal
You configure hard goals in the Cruise Control deployment configuration, by editing the hard.goals
property in Kafka.spec.cruiseControl.config
.
-
To inherit the preset hard goals from Cruise Control, do not specify the
hard.goals
property inKafka.spec.cruiseControl.config
-
To change the preset hard goals, specify the desired goals in the
hard.goals
property, using their fully-qualified domain names.
Example Kafka
configuration for hard optimization goals
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... zookeeper: # ... entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal # ...
Increasing the number of configured hard goals will reduce the likelihood of Cruise Control generating valid optimization proposals.
If skipHardGoalCheck: true
is specified in the KafkaRebalance
custom resource, Cruise Control does not check that the list of user-provided optimization goals (in KafkaRebalance.spec.goals
) contains all the configured hard goals (hard.goals
). Therefore, if some, but not all, of the user-provided optimization goals are in the hard.goals
list, Cruise Control will still treat them as hard goals even if skipHardGoalCheck: true
is specified.
Master optimization goals
The master optimization goals are available to all users. Goals that are not listed in the master optimization goals are not available for use in Cruise Control operations.
Unless you change the Cruise Control deployment configuration, AMQ Streams will inherit the following master optimization goals from Cruise Control, in descending priority order:
RackAwareGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal; ReplicaDistributionGoal; PotentialNwOutGoal; DiskUsageDistributionGoal; NetworkInboundUsageDistributionGoal; NetworkOutboundUsageDistributionGoal; CpuUsageDistributionGoal; TopicReplicaDistributionGoal; LeaderReplicaDistributionGoal; LeaderBytesInDistributionGoal; PreferredLeaderElectionGoal
Six of these goals are preset as hard goals.
To reduce complexity, we recommend that you use the inherited master optimization goals, unless you need to completely exclude one or more goals from use in KafkaRebalance
resources. The priority order of the master optimization goals can be modified, if desired, in the configuration for default optimization goals.
You configure master optimization goals, if necessary, in the Cruise Control deployment configuration: Kafka.spec.cruiseControl.config.goals
-
To accept the inherited master optimization goals, do not specify the
goals
property inKafka.spec.cruiseControl.config
. -
If you need to modify the inherited master optimization goals, specify a list of goals, in descending priority order, in the
goals
configuration option.
If you change the inherited master optimization goals, you must ensure that the hard goals, if configured in the hard.goals
property in Kafka.spec.cruiseControl.config
, are a subset of the master optimization goals that you configured. Otherwise, errors will occur when generating optimization proposals.
Default optimization goals
Cruise Control uses the default optimization goals to generate the cached optimization proposal. For more information about the cached optimization proposal, see Section 8.3, “Optimization proposals overview”.
You can override the default optimization goals by setting user-provided optimization goals in a KafkaRebalance
custom resource.
Unless you specify default.goals
in the Cruise Control deployment configuration, the master optimization goals are used as the default optimization goals. In this case, the cached optimization proposal is generated using the master optimization goals.
-
To use the master optimization goals as the default goals, do not specify the
default.goals
property inKafka.spec.cruiseControl.config
. -
To modify the default optimization goals, edit the
default.goals
property inKafka.spec.cruiseControl.config
. You must use a subset of the master optimization goals.
Example Kafka
configuration for default optimization goals
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... zookeeper: # ... entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal # ...
If no default optimization goals are specified, the cached proposal is generated using the master optimization goals.
User-provided optimization goals
User-provided optimization goals narrow down the configured default goals for a particular optimization proposal. You can set them, as required, in spec.goals
in a KafkaRebalance
custom resource:
KafkaRebalance.spec.goals
User-provided optimization goals can generate optimization proposals for different scenarios. For example, you might want to optimize leader replica distribution across the Kafka cluster without considering disk capacity or disk utilization. So, you create a KafkaRebalance
custom resource containing a single user-provided goal for leader replica distribution.
User-provided optimization goals must:
- Include all configured hard goals, or an error occurs
- Be a subset of the master optimization goals
To ignore the configured hard goals when generating an optimization proposal, add the skipHardGoalCheck: true
property to the KafkaRebalance
custom resource. See Section 8.7, “Generating optimization proposals”.
Additional resources
- Section 8.5, “Cruise Control configuration”
- Configurations in the Cruise Control Wiki.
8.3. Optimization proposals overview
An optimization proposal is a summary of proposed changes that would produce a more balanced Kafka cluster, with partition workloads distributed more evenly among the brokers. Each optimization proposal is based on the set of optimization goals that was used to generate it, subject to any configured capacity limits on broker resources.
An optimization proposal is contained in the Status.Optimization Result
property of a KafkaRebalance
custom resource. The information provided is a summary of the full optimization proposal. Use the summary to decide whether to:
- Approve the optimization proposal. This instructs Cruise Control to apply the proposal to the Kafka cluster and start a cluster rebalance operation.
- Reject the optimization proposal. You can change the optimization goals and then generate another proposal.
All optimization proposals are dry runs: you cannot approve a cluster rebalance without first generating an optimization proposal. There is no limit to the number of optimization proposals that can be generated.
Cached optimization proposal
Cruise Control maintains a cached optimization proposal based on the configured default optimization goals. Generated from the workload model, the cached optimization proposal is updated every 15 minutes to reflect the current state of the Kafka cluster. If you generate an optimization proposal using the default optimization goals, Cruise Control returns the most recent cached proposal.
To change the cached optimization proposal refresh interval, edit the proposal.expiration.ms
setting in the Cruise Control deployment configuration. Consider a shorter interval for fast changing clusters, although this increases the load on the Cruise Control server.
Contents of optimization proposals
The following table describes the properties contained in an optimization proposal:
JSON property | Description |
---|---|
| The total number of partition replicas that will be transferred between the disks of the cluster’s brokers.
Performance impact during rebalance operation: Relatively high, but lower than |
| Not yet supported. An empty list is returned. |
| The number of partition replicas that will be moved between separate brokers. Performance impact during rebalance operation: Relatively high. |
| A measurement of the overall balancedness of a Kafka Cluster, before and after the optimization proposal was generated.
The score is calculated by subtracting the sum of the
The |
|
The sum of the size of each partition replica that will be moved between disks on the same broker (see also
Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see |
| The number of metrics windows upon which the optimization proposal is based. |
|
The sum of the size of each partition replica that will be moved to a separate broker (see also Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. |
|
The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of |
|
If you specified a regular expression in the |
| The number of partitions whose leaders will be switched to different replicas. This involves a change to ZooKeeper configuration. Performance impact during rebalance operation: Relatively low. |
| Not yet supported. An empty list is returned. |
8.4. Rebalance performance tuning overview
You can adjust several performance tuning options for cluster rebalances. These options control how partition replica and leadership movements in a rebalance are executed, as well as the bandwidth that is allocated to a rebalance operation.
Partition reassignment commands
Optimization proposals are comprised of separate partition reassignment commands. When you approve a proposal, the Cruise Control server applies these commands to the Kafka cluster.
A partition reassignment command consists of either of the following types of operations:
Partition movement: Involves transferring the partition replica and its data to a new location. Partition movements can take one of two forms:
- Inter-broker movement: The partition replica is moved to a log directory on a different broker.
- Intra-broker movement: The partition replica is moved to a different log directory on the same broker.
- Leadership movement: This involves switching the leader of the partition’s replicas.
Cruise Control issues partition reassignment commands to the Kafka cluster in batches. The performance of the cluster during the rebalance is affected by the number of each type of movement contained in each batch.
Replica movement strategies
Cluster rebalance performance is also influenced by the replica movement strategy that is applied to the batches of partition reassignment commands. By default, Cruise Control uses the BaseReplicaMovementStrategy
, which simply applies the commands in the order they were generated. However, if there are some very large partition reassignments early in the proposal, this strategy can slow down the application of the other reassignments.
Cruise Control provides three alternative replica movement strategies that can be applied to optimization proposals:
-
PrioritizeSmallReplicaMovementStrategy
: Order reassignments in order of ascending size. -
PrioritizeLargeReplicaMovementStrategy
: Order reassignments in order of descending size. -
PostponeUrpReplicaMovementStrategy
: Prioritize reassignments for replicas of partitions which have no out-of-sync replicas.
These strategies can be configured as a sequence. The first strategy attempts to compare two partition reassignments using its internal logic. If the reassignments are equivalent, then it passes them to the next strategy in the sequence to decide the order, and so on.
Rebalance tuning options
Cruise Control provides several configuration options for tuning the rebalance parameters discussed above. You can set these tuning options at either the Cruise Control server or optimization proposal levels:
-
The Cruise Control server setting can be set in the Kafka custom resource under
Kafka.spec.cruiseControl.config
. -
The individual rebalance performance configurations can be set under
KafkaRebalance.spec
.
The relevant configurations are summarized below:
Server and KafkaRebalance Configuration | Description | Default Value |
---|---|---|
| The maximum number of inter-broker partition movements in each partition reassignment batch | 5 |
| ||
| The maximum number of intra-broker partition movements in each partition reassignment batch | 2 |
| ||
| The maximum number of partition leadership changes in each partition reassignment batch | 1000 |
| ||
| The bandwidth (in bytes per second) to be assigned to the reassigning of partitions | No Limit |
| ||
| The list of strategies (in priority order) used to determine the order in which partition reassignment commands are executed for generated proposals.
For the server setting, use a comma separated string with the fully qualified names of the strategy class (add |
|
|
Changing the default settings affects the length of time that the rebalance takes to complete, as well as the load placed on the Kafka cluster during the rebalance. Using lower values reduces the load but increases the amount of time taken, and vice versa.
8.5. Cruise Control configuration
The config
property in Kafka.spec.cruiseControl
contains configuration options as keys with values as one of the following JSON types:
- String
- Number
- Boolean
You can specify and configure all the options listed in the "Configurations" section of the Cruise Control documentation, apart from those managed directly by AMQ Streams. Specifically, you cannot modify configuration options with keys equal to or starting with one of the keys mentioned here.
If restricted options are specified, they are ignored and a warning message is printed to the Cluster Operator log file. All the supported options are passed to Cruise Control.
An example Cruise Control configuration
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... cruiseControl: # ... config: default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal cpu.balance.threshold: 1.1 metadata.max.age.ms: 300000 send.buffer.bytes: 131072 # ...
Cross-Origin Resource Sharing configuration
Cross-Origin Resource Sharing (CORS) allows you to specify allowed methods and originating URLs for accessing REST APIs.
By default, CORS is disabled for the Cruise Control REST API. When enabled, only GET
requests for read-only access to the Kafka cluster state are allowed. This means that external applications, which are running in different origins than the AMQ Streams components, cannot make POST
requests to the Cruise Control API. However, those applications can make GET
requests to access read-only information about the Kafka cluster, such as the current cluster load or the most recent optimization proposal.
Enabling CORS for Cruise Control
You enable and configure CORS in Kafka.spec.cruiseControl.config
.
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... cruiseControl: # ... config: webserver.http.cors.enabled: true webserver.http.cors.origin: "*" webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type" # ...
For more information, see REST APIs in the Cruise Control Wiki.
Capacity configuration
Cruise Control uses capacity limits to determine if optimization goals for resource distribution are being broken. There are four goals of this type:
-
DiskUsageDistributionGoal
- Disk utilization distribution -
CpuUsageDistributionGoal
- CPU utilization distribution -
NetworkInboundUsageDistributionGoal
- Network inbound utilization distribution -
NetworkOutboundUsageDistributionGoal
- Network outbound utilization distribution
You specify capacity limits for Kafka broker resources in the brokerCapacity
property in Kafka.spec.cruiseControl
. They are enabled by default and you can change their default values. Capacity limits can be set for the following broker resources, using the standard OpenShift byte units (K, M, G and T) or their bibyte (power of two) equivalents (Ki, Mi, Gi and Ti):
-
disk
- Disk storage per broker (Default: 100000Mi) -
cpuUtilization
- CPU utilization as a percentage (Default: 100) -
inboundNetwork
- Inbound network throughput in byte units per second (Default: 10000KiB/s) -
outboundNetwork
- Outbound network throughput in byte units per second (Default: 10000KiB/s)
Because AMQ Streams Kafka brokers are homogeneous, Cruise Control applies the same capacity limits to every broker it is monitoring.
An example Cruise Control brokerCapacity configuration using bibyte units
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... cruiseControl: # ... brokerCapacity: disk: 100Gi cpuUtilization: 100 inboundNetwork: 10000KiB/s outboundNetwork: 10000KiB/s # ...
Additional resources
For more information, refer to the Section 13.2.73, “BrokerCapacity
schema reference”.
Logging configuration
Cruise Control has its own configurable logger:
-
rootLogger.level
Cruise Control uses the Apache log4j 2
logger implementation.
Use the logging
property to configure loggers and logger levels.
You can set the log levels by specifying the logger and level directly (inline) or use a custom (external) ConfigMap. If a ConfigMap is used, you set logging.valueFrom.configMapKeyRef.name
property to the name of the ConfigMap containing the external logging configuration. Inside the ConfigMap, the logging configuration is described using log4j.properties
. Both logging.valueFrom.configMapKeyRef.name
and logging.valueFrom.configMapKeyRef.key
properties are mandatory. A ConfigMap using the exact logging configuration specified is created with the custom resource when the Cluster Operator is running, then recreated after each reconciliation. If you do not specify a custom ConfigMap, default logging settings are used. If a specific logger value is not set, upper-level logger settings are inherited for that logger. Here we see examples of inline
and external
logging.
Inline logging
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka # ... spec: cruiseControl: # ... logging: type: inline loggers: rootLogger.level: "INFO" # ...
External logging
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka # ... spec: cruiseControl: # ... logging: type: external valueFrom: configMapKeyRef: name: customConfigMap key: cruise-control-log4j.properties # ...
8.6. Deploying Cruise Control
To deploy Cruise Control to your AMQ Streams cluster, define the configuration using the cruiseControl
property in the Kafka
resource, and then create or update the resource.
Deploy one instance of Cruise Control per Kafka cluster.
Prerequisites
- An OpenShift cluster
- A running Cluster Operator
Procedure
Edit the
Kafka
resource and add thecruiseControl
property.The properties you can configure are shown in this example configuration:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... cruiseControl: brokerCapacity: 1 inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s # ... config: 2 default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal # ... cpu.balance.threshold: 1.1 metadata.max.age.ms: 300000 send.buffer.bytes: 131072 # ... resources: 3 requests: cpu: 1 memory: 512Mi limits: cpu: 2 memory: 2Gi logging: 4 type: inline loggers: rootLogger.level: "INFO" template: 5 pod: metadata: labels: label1: value1 securityContext: runAsUser: 1000001 fsGroup: 0 terminationGracePeriodSeconds: 120 readinessProbe: 6 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: 7 initialDelaySeconds: 15 timeoutSeconds: 5 # ...
- 1
- Specifies capacity limits for broker resources. For more information, see Capacity configuration.
- 2
- Defines the Cruise Control configuration, including the default optimization goals (in
default.goals
) and any customizations to the master optimization goals (ingoals
) or the hard goals (inhard.goals
). You can provide any standard Cruise Control configuration option apart from those managed directly by AMQ Streams. For more information on configuring optimization goals, see Section 8.2, “Optimization goals overview”. - 3
- CPU and memory resources reserved for Cruise Control. For more information, see Section 13.1.5, “
resources
”. - 4
- Defined loggers and log levels added directly (inline) or indirectly (external) through a ConfigMap. A custom ConfigMap must be placed under the log4j.properties key. Cruise Control has a single logger named
rootLogger.level
. You can set the log level to INFO, ERROR, WARN, TRACE, DEBUG, FATAL or OFF. For more information, see Logging configuration. - 5
- 6
- 7
Create or update the resource:
oc apply -f kafka.yaml
Verify that Cruise Control was successfully deployed:
oc get deployments -l app.kubernetes.io/name=cruise-control
Auto-created topics
The following table shows the three topics that are automatically created when Cruise Control is deployed. These topics are required for Cruise Control to work properly and must not be deleted or changed.
Auto-created topic | Created by | Function |
---|---|---|
| AMQ Streams Metrics Reporter | Stores the raw metrics from the Metrics Reporter in each Kafka broker. |
| Cruise Control | Stores the derived metrics for each partition. These are created by the Metric Sample Aggregator. |
| Cruise Control | Stores the metrics samples used to create the Cluster Workload Model. |
To prevent the removal of records that are needed by Cruise Control, log compaction is disabled in the auto-created topics.
What to do next
After configuring and deploying Cruise Control, you can generate optimization proposals.
Additional resources
8.7. Generating optimization proposals
When you create or update a KafkaRebalance
resource, Cruise Control generates an optimization proposal for the Kafka cluster based on the configured optimization goals.
Analyze the information in the optimization proposal and decide whether to approve it.
Prerequisites
- You have deployed Cruise Control to your AMQ Streams cluster.
- You have configured optimization goals and, optionally, capacity limits on broker resources.
Procedure
Create a
KafkaRebalance
resource:To use the default optimization goals defined in the
Kafka
resource, leave thespec
property empty:apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: {}
To configure user-provided optimization goals instead of using the default goals, add the
goals
property and enter one or more goals.In the following example, rack awareness and replica capacity are configured as user-provided optimization goals:
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal
To ignore the configured hard goals, add the
skipHardGoalCheck: true
property:apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal skipHardGoalCheck: true
Create or update the resource:
oc apply -f your-file
The Cluster Operator requests the optimization proposal from Cruise Control. This might take a few minutes depending on the size of the Kafka cluster.
Check the status of the
KafkaRebalance
resource:oc describe kafkarebalance rebalance-cr-name
Cruise Control returns one of two statuses:
-
PendingProposal
: The rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready. -
ProposalReady
: The optimization proposal is ready for review and, if desired, approval. The optimization proposal is contained in theStatus.Optimization Result
property of theKafkaRebalance
resource.
-
Review the optimization proposal.
oc describe kafkarebalance rebalance-cr-name
Here is an example proposal:
Status: Conditions: Last Transition Time: 2020-05-19T13:50:12.533Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 0 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 0 Num Replica Movements: 26 On Demand Balancedness Score After: 81.8666802863978 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 1 Session Id: 05539377-ca7b-45ef-b359-e13564f1458c
The properties in the
Optimization Result
section describe the pending cluster rebalance operation. For descriptions of each property, see Contents of optimization proposals.
What to do next
Additional resources
8.8. Approving an optimization proposal
You can approve an optimization proposal generated by Cruise Control, if its status is ProposalReady
. Cruise Control will then apply the optimization proposal to the Kafka cluster, reassigning partitions to brokers and changing partition leadership.
This is not a dry run. Before you approve an optimization proposal, you must:
- Refresh the proposal in case it has become out of date.
- Carefully review the contents of the proposal.
Prerequisites
- You have generated an optimization proposal from Cruise Control.
-
The
KafkaRebalance
custom resource status isProposalReady
.
Procedure
Perform these steps for the optimization proposal that you want to approve:
Unless the optimization proposal is newly generated, check that it is based on current information about the state of the Kafka cluster. To do so, refresh the optimization proposal to make sure it uses the latest cluster metrics:
Annotate the
KafkaRebalance
resource in OpenShift withrefresh
:oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance=refresh
Check the status of the
KafkaRebalance
resource:oc describe kafkarebalance rebalance-cr-name
-
Wait until the status changes to
ProposalReady
.
Approve the optimization proposal that you want Cruise Control to apply.
Annotate the
KafkaRebalance
resource in OpenShift:oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance=approve
- The Cluster Operator detects the annotated resource and instructs Cruise Control to rebalance the Kafka cluster.
Check the status of the
KafkaRebalance
resource:oc describe kafkarebalance rebalance-cr-name
Cruise Control returns one of three statuses:
- Rebalancing: The cluster rebalance operation is in progress.
-
Ready: The cluster rebalancing operation completed successfully. The
KafkaRebalance
custom resource cannot be reused. -
NotReady: An error occurred—see Section 8.10, “Fixing problems with a
KafkaRebalance
resource”.
8.9. Stopping a cluster rebalance
Once started, a cluster rebalance operation might take some time to complete and affect the overall performance of the Kafka cluster.
If you want to stop a cluster rebalance operation that is in progress, apply the stop
annotation to the KafkaRebalance
custom resource. This instructs Cruise Control to finish the current batch of partition reassignments and then stop the rebalance. When the rebalance has stopped, completed partition reassignments have already been applied; therefore, the state of the Kafka cluster is different when compared to prior to the start of the rebalance operation. If further rebalancing is required, you should generate a new optimization proposal.
The performance of the Kafka cluster in the intermediate (stopped) state might be worse than in the initial state.
Prerequisites
-
You have approved the optimization proposal by annotating the
KafkaRebalance
custom resource withapprove
. -
The status of the
KafkaRebalance
custom resource isRebalancing
.
Procedure
Annotate the
KafkaRebalance
resource in OpenShift:oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance=stop
Check the status of the
KafkaRebalance
resource:oc describe kafkarebalance rebalance-cr-name
-
Wait until the status changes to
Stopped
.
Additional resources
8.10. Fixing problems with a KafkaRebalance
resource
If an issue occurs when creating a KafkaRebalance
resource or interacting with Cruise Control, the error is reported in the resource status, along with details of how to fix it. The resource also moves to the NotReady
state.
To continue with the cluster rebalance operation, you must fix the problem in the KafkaRebalance
resource itself or with the overall Cruise Control deployment. Problems might include the following:
-
A misconfigured parameter in the
KafkaRebalance
resource. -
The
strimzi.io/cluster
label for specifying the Kafka cluster in theKafkaRebalance
resource is missing. -
The Cruise Control server is not deployed as the
cruiseControl
property in theKafka
resource is missing. - The Cruise Control server is not reachable.
After fixing the issue, you need to add the refresh
annotation to the KafkaRebalance
resource. During a “refresh”, a new optimization proposal is requested from the Cruise Control server.
Prerequisites
- You have approved an optimization proposal.
-
The status of the
KafkaRebalance
custom resource for the rebalance operation isNotReady
.
Procedure
Get information about the error from the
KafkaRebalance
status:oc describe kafkarebalance rebalance-cr-name
-
Attempt to resolve the issue in the
KafkaRebalance
resource. Annotate the
KafkaRebalance
resource in OpenShift:oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance=refresh
Check the status of the
KafkaRebalance
resource:oc describe kafkarebalance rebalance-cr-name
-
Wait until the status changes to
PendingProposal
, or directly toProposalReady
.
Additional resources