Chapter 44. TieredStorageCustom schema reference
Used in: KafkaClusterSpec
Full list of TieredStorageCustom schema properties
Enables custom tiered storage for Kafka.
If you want to use custom tiered storage, you must first add a tiered storage for Kafka plugin to the Streams for Apache Kafka image by building a custom container image.
Custom tiered storage configuration enables the use of a custom RemoteStorageManager configuration. RemoteStorageManager is a Kafka interface for managing the interaction between Kafka and remote tiered storage.
If custom tiered storage is enabled, Streams for Apache Kafka uses the TopicBasedRemoteLogMetadataManager for Remote Log Metadata Management (RLMM).
Tiered storage is an early access Kafka feature, which is also available in Streams for Apache Kafka. Due to its current limitations, it is not recommended for production environments.
Example custom tiered storage configuration
kafka:
tieredStorage:
type: custom
remoteStorageManager:
className: com.example.kafka.tiered.storage.s3.S3RemoteStorageManager
classPath: /opt/kafka/plugins/tiered-storage-s3/*
config:
# A map with String keys and String values.
# Key properties are automatically prefixed with `rsm.config.`
# and appended to Kafka broker config.
storage.bucket.name: my-bucket
config:
...
# Additional RLMM configuration can be added through the Kafka config
# under `spec.kafka.config` using the `rlmm.config.` prefix.
rlmm.config.remote.log.metadata.topic.replication.factor: 1
44.1. TieredStorageCustom schema properties
The type property is a discriminator that distinguishes use of the TieredStorageCustom type from other subtypes which may be added in the future. It must have the value custom for the type TieredStorageCustom.
| Property | Property type | Description |
|---|---|---|
| remoteStorageManager | Configuration for the Remote Storage Manager. | |
| type | string |
Must be |
