Chapter 5. Using the Topic Operator
When you create, modify or delete a topic using the KafkaTopic
resource, the Topic Operator ensures those changes are reflected in the Kafka cluster.
5.1. Kafka topic resource
The KafkaTopic
resource is used to configure topics, including the number of partitions and replicas.
The full schema for KafkaTopic
is described in KafkaTopic
schema reference.
5.1.1. Kafka topic usage recommendations
When working with topics, be consistent. Always operate on either KafkaTopic
resources or topics directly in OpenShift. Avoid routinely switching between both methods for a given topic.
Use topic names that reflect the nature of the topic, and remember that names cannot be changed later.
If creating a topic in Kafka, use a name that is a valid OpenShift resource name, otherwise the Topic Operator will need to create the corresponding KafkaTopic
with a name that conforms to the OpenShift rules.
Recommendations for identifiers and names in OpenShift are outlined in Identifiers and Names in OpenShift community article.
5.1.2. Kafka topic naming conventions
Kafka and OpenShift impose their own validation rules for the naming of topics in Kafka and KafkaTopic.metadata.name
respectively. There are valid names for each which are invalid in the other.
Using the spec.topicName
property, it is possible to create a valid topic in Kafka with a name that would be invalid for the Kafka topic in OpenShift.
The spec.topicName
property inherits Kafka naming validation rules:
- The name must not be longer than 249 characters.
-
Valid characters for Kafka topics are ASCII alphanumerics,
.
,_
, and-
. -
The name cannot be
.
or..
, though.
can be used in a name, such asexampleTopic.
or.exampleTopic
.
spec.topicName
must not be changed.
For example:
kind: KafkaTopic
metadata:
name: topic-name-1
spec:
topicName: topicName-1 1
# ...
- 1
- Upper case is invalid in OpenShift.
cannot be changed to:
kind: KafkaTopic metadata: name: topic-name-1 spec: topicName: name-2 # ...
Some Kafka client applications, such as Kafka Streams, can create topics in Kafka programmatically. If those topics have names that are invalid OpenShift resource names, the Topic Operator gives them valid names based on the Kafka names. Invalid characters are replaced and a hash is appended to the name.
5.2. Configuring a Kafka topic
Use the properties of the KafkaTopic
resource to configure a Kafka topic.
You can use oc apply
to create or modify topics, and oc delete
to delete existing topics.
For example:
-
oc apply -f <topic-config-file>
-
oc delete KafkaTopic <topic-name>
This procedure shows how to create a topic with 10 partitions and 2 replicas.
Before you start
It is important that you consider the following before making your changes:
Kafka does not support making the following changes through the
KafkaTopic
resource:-
Changing topic names using
spec.topicName
-
Decreasing partition size using
spec.partitions
-
Changing topic names using
-
You cannot use
spec.replicas
to change the number of replicas that were initially specified. -
Increasing
spec.partitions
for topics with keys will change how records are partitioned, which can be particularly problematic when the topic uses semantic partitioning.
Prerequisites
- A running Kafka cluster configured with a Kafka broker listener using TLS authentication and encryption.
- A running Topic Operator (typically deployed with the Entity Operator).
-
For deleting a topic,
delete.topic.enable=true
(default) in thespec.kafka.config
of theKafka
resource.
Procedure
Prepare a file containing the
KafkaTopic
to be created.An example
KafkaTopic
apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaTopic metadata: name: orders labels: strimzi.io/cluster: my-cluster spec: partitions: 10 replicas: 2
TipWhen modifying a topic, you can get the current version of the resource using
oc get kafkatopic orders -o yaml
.Create the
KafkaTopic
resource in OpenShift.oc apply -f <topic-config-file>