Chapter 46. TieredStorageCustom schema reference
Used in: KafkaClusterSpec
Full list of TieredStorageCustom schema properties
Enables custom tiered storage for Kafka.
If you want to use custom tiered storage, you must first add a tiered storage for Kafka plugin to the Streams for Apache Kafka image by building a custom container image.
Custom tiered storage configuration enables the use of a custom RemoteStorageManager configuration. RemoteStorageManager is a Kafka interface for managing the interaction between Kafka and remote tiered storage.
If custom tiered storage is enabled, Streams for Apache Kafka uses the TopicBasedRemoteLogMetadataManager for Remote Log Metadata Management (RLMM).
Tiered storage is a production-ready feature in Kafka since version 3.9.0, and it is also supported in Streams for Apache Kafka. Before introducing tiered storage to your environment, review the known limitations of this feature.
Example custom tiered storage configuration
kafka:
tieredStorage:
type: custom
remoteStorageManager:
className: com.example.kafka.tiered.storage.s3.S3RemoteStorageManager
classPath: /opt/kafka/plugins/tiered-storage-s3/*
config:
# A map with String keys and String values.
# Key properties are automatically prefixed with `rsm.config.`
# and appended to Kafka broker config.
storage.bucket.name: my-bucket
config:
...
# Additional RLMM configuration can be added through the Kafka config
# under `spec.kafka.config` using the `rlmm.config.` prefix.
rlmm.config.remote.log.metadata.topic.replication.factor: 1
46.1. TieredStorageCustom schema properties Copy linkLink copied to clipboard!
The type property is a discriminator that distinguishes use of the TieredStorageCustom type from other subtypes which may be added in the future. It must have the value custom for the type TieredStorageCustom.
| Property | Property type | Description |
|---|---|---|
| type | string |
Must be |
| remoteStorageManager | Configuration for the Remote Storage Manager. |