Chapter 1. Kafka tuning overview
Use configuration properties to optimize the performance of Kafka brokers, producers and consumers. You can specify configuration properties for AMQ Streams on OCP and RHEL.
A minimum set of configuration properties is required, but you can add or adjust properties to change how producers and consumers interact with Kafka brokers. For example, you can tune latency and throughput of messages so that clients can respond to data in real time.
You might start by analyzing metrics to gauge where to make your initial configurations, then make incremental changes and further comparisons of metrics until you have the configuration you need.
For more information about Apache Kafka configuration properties, see the Apache Kafka documentation.
1.1. Mapping properties and values
How you specify configuration properties depends on the type of deployment. If you deployed AMQ Streams on OCP, you can use the Kafka resource to add configuration for Kafka brokers through the config property. With AMQ Streams on RHEL, you add the configuration to a properties file as environment variables.
When you add config properties to custom resources, you use a colon (':') to map the property and value.
Example configuration in a custom resource
num.partitions:1
When you add the properties as environment variables, you use an equal sign ('=') to map the property and value.
Example configuration as an environment variable
num.partitions=1
1.2. Tools that help with tuning
The following tools help with Kafka tuning:
- Cruise Control generates optimization proposals that you can use to assess and implement a cluster rebalance
- Kafka Static Quota plugin sets limits on brokers
- Rack configuration spreads broker partitions across racks and allows consumers to fetch data from the nearest replica
For more information on these tools, see the following guides:
