红帽全球峰会此内容没有您所选择的语言版本。
Chapter 13. Setting throughput and storage limits on brokers
This feature is a technology preview and not intended for a production environment. For more information see the release notes.
This procedure describes how to set throughput and storage limits on brokers in your Kafka cluster. Enable the Strimzi Quotas plugin and configure limits using quota properties
The plugin provides storage utilization quotas and dynamic distribution of throughput limits.
-
Storage quotas throttle Kafka producers based on disk storage utilization. Limits can be specified in bytes (
storage.per.volume.limit.min.available.bytes) or percentage (storage.per.volume.limit.min.available.ratio) of available disk space, applying to each disk individually. When any broker in the cluster exceeds the configured disk threshold, clients are throttled to prevent disks from filling up too quickly and exceeding capacity. - A total throughput limit is distributed dynamically across all clients. For example, if you set a 40 MBps producer byte-rate threshold, the distribution across two producers is not static. If one producer is using 10 MBps, the other can use up to 30 MBps.
- Specific users (clients) can be excluded from the restrictions.
With the plugin, you see only aggregated quota metrics, not per-client metrics.
Prerequisites
- Streams for Apache Kafka is installed on each host, and the configuration files are available.
Procedure
Edit the Kafka configuration properties file.
Example plugin configuration
# ... client.quota.callback.class=io.strimzi.kafka.quotas.StaticQuotaCallback 1 client.quota.callback.static.produce=1000000 2 client.quota.callback.static.fetch=1000000 3 client.quota.callback.static.storage.per.volume.limit.min.available.bytes=500000000000 4 client.quota.callback.static.storage.check-interval=5 5 client.quota.callback.static.kafka.admin.bootstrap.servers=localhost:9092 6 client.quota.callback.static.excluded.principal.name.list=User:my-user-1;User:my-user-2 7 # ...
- 1
- Loads the plugin.
- 2
- Sets the producer byte-rate threshold of 1 MBps.
- 3
- Sets the consumer byte-rate threshold. 1 MBps.
- 4
- Sets an available bytes limit of 500 GB.
- 5
- Sets the interval in seconds between checks on storage to 5 seconds. The default is 60 seconds. Set this property to 0 to disable the check.
- 6
- Kafka cluster bootstrap servers address. This property is required if
storage.check-intervalis >0. All configuration properties starting withclient.quota.callback.static.kafka.admin.prefix are passed to the Kafka Admin client configuration. - 7
- Excludes
my-user-1andmy-user-2from the restrictions. Each principal must be be prefixed withUser:.
storage.per.volume.limit.min.available.bytesandstorage.per.volume.limit.min.available.ratioare mutually exclusive. Only configure one of these parameters.NoteThe full list of supported configuration properties can be found in the plugin documentation.
Start the Kafka broker with the default configuration file.
./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties
Verify that the Kafka broker is running.
jcmd | grep Kafka
