红帽全球峰会此内容没有您所选择的语言版本。
Deploying and Managing Streams for Apache Kafka on OpenShift
Deploy and manage Streams for Apache Kafka 3.1 on OpenShift Container Platform
Abstract
Providing feedback on Red Hat documentation
We appreciate your feedback on our documentation.
To propose improvements, open a Jira issue and describe your suggested changes. Provide as much detail as possible to enable us to address your request quickly.
Prerequisite
- You have a Red Hat Customer Portal account. This account enables you to log in to the Red Hat Jira Software instance. If you do not have an account, you will be prompted to create one.
Procedure
- Click Create issue.
- In the Summary text box, enter a brief description of the issue.
In the Description text box, provide the following information:
- The URL of the page where you found the issue.
-
A detailed description of the issue.
You can leave the information in any other fields at their default values.
- Add a reporter name.
- Click Create to submit the Jira issue to the documentation team.
Thank you for taking the time to provide feedback.
Chapter 1. Deployment overview
Streams for Apache Kafka simplifies the process of running Apache Kafka within an OpenShift cluster.
This guide provides instructions for deploying and managing Streams for Apache Kafka. Deployment options and steps are covered using the example installation files included with Streams for Apache Kafka. While the guide highlights important configuration considerations, it does not cover all available options. For a deeper understanding of the Kafka component configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.
In addition to deployment instructions, the guide offers pre- and post-deployment guidance. It covers setting up and securing client access to your Kafka cluster. Furthermore, it explores additional deployment options such as metrics integration, distributed tracing, and cluster management tools like Cruise Control and the Streams for Apache Kafka Drain Cleaner. You’ll also find recommendations on managing Streams for Apache Kafka and fine-tuning Kafka configuration for optimal performance.
Upgrade instructions are provided for both Streams for Apache Kafka and Kafka, to help keep your deployment up to date.
Streams for Apache Kafka is designed to be compatible with all types of OpenShift clusters, irrespective of their distribution. Whether your deployment involves public or private clouds, or if you are setting up a local development environment, the instructions in this guide are applicable in all cases.
1.1. Streams for Apache Kafka custom resources
The deployment of Kafka components onto an OpenShift cluster using Streams for Apache Kafka is highly configurable through the use of custom resources. These resources are created as instances of APIs introduced by Custom Resource Definitions (CRDs), which extend OpenShift resources.
CRDs act as configuration instructions to describe the custom resources in an OpenShift cluster, and are provided with Streams for Apache Kafka for each Kafka component used in a deployment, as well as users and topics. CRDs and custom resources are defined as YAML files. Example YAML files are provided with the Streams for Apache Kafka distribution.
CRDs also allow Streams for Apache Kafka resources to benefit from native OpenShift features like CLI accessibility and configuration validation.
1.1.1. Streams for Apache Kafka custom resource example
CRDs require a one-time installation in a cluster to define the schemas used to instantiate and manage Streams for Apache Kafka-specific resources.
After a new custom resource type is added to your cluster by installing a CRD, you can create instances of the resource based on its specification.
Depending on the cluster setup, installation typically requires cluster admin privileges.
Access to manage custom resources is limited to Streams for Apache Kafka administrators. For more information, see Section 6.5, “Designating Streams for Apache Kafka administrators”.
A CRD defines a new kind of resource, such as kind:Kafka, within an OpenShift cluster.
The Kubernetes API server allows custom resources to be created based on the kind and understands from the CRD how to validate and store the custom resource when it is added to the OpenShift cluster.
Each Streams for Apache Kafka-specific custom resource conforms to the schema defined by the CRD for the resource’s kind. The custom resources for Streams for Apache Kafka components have common configuration properties, which are defined under spec.
To understand the relationship between a CRD and a custom resource, let’s look at a sample of the CRD for a Kafka topic.
Kafka topic CRD
- 1
- The metadata for the topic CRD, its name and a label to identify the CRD.
- 2
- The specification for this CRD, including the group (domain) name, the plural name and the supported schema version, which are used in the URL to access the API of the topic. The other names are used to identify instance resources in the CLI. For example,
oc get kafkatopic my-topicoroc get kafkatopics. - 3
- The shortname can be used in CLI commands. For example,
oc get ktcan be used as an abbreviation instead ofoc get kafkatopic. - 4
- The information presented when using a
getcommand on the custom resource. - 5
- The current status of the CRD as described in the schema reference for the resource.
- 6
- openAPIV3Schema validation provides validation for the creation of topic custom resources. For example, a topic requires at least one partition and one replica.
You can identify the CRD YAML files supplied with the Streams for Apache Kafka installation files, because the file names contain an index number followed by ‘Crd’.
Here is a corresponding example of a KafkaTopic custom resource.
Kafka topic custom resource
- 1
- The
kindandapiVersionidentify the CRD of which the custom resource is an instance. - 2
- A label, applicable only to
KafkaTopicandKafkaUserresources, that defines the name of the Kafka cluster (which is same as the name of theKafkaresource) to which a topic or user belongs. - 3
- The spec shows the number of partitions and replicas for the topic as well as the configuration parameters for the topic itself. In this example, the retention period for a message to remain in the topic and the segment file size for the log are specified.
- 4
- Status conditions for the
KafkaTopicresource. Thetypecondition changed toReadyat thelastTransitionTime.
Custom resources can be applied to a cluster through the platform CLI. When the custom resource is created, it uses the same validation as the built-in resources of the Kubernetes API.
After a KafkaTopic custom resource is created, the Topic Operator is notified and corresponding Kafka topics are created in Streams for Apache Kafka.
1.1.2. Performing oc operations on custom resources
You can use oc commands to retrieve information and perform other operations on Streams for Apache Kafka custom resources. Use oc commands, such as get, describe, edit, or delete, to perform operations on resource types. For example, oc get kafkatopics retrieves a list of all Kafka topics and oc get kafkas retrieves all deployed Kafka clusters.
When referencing resource types, you can use both singular and plural names: oc get kafkas gets the same results as oc get kafka.
You can also use the short name of the resource. Learning short names can save you time when managing Streams for Apache Kafka. The short name for Kafka is k, so you can also run oc get k to list all Kafka clusters.
Listing Kafka clusters
oc get k NAME READY METADATA STATE WARNINGS my-cluster True KRaft
oc get k
NAME READY METADATA STATE WARNINGS
my-cluster True KRaft| Streams for Apache Kafka resource | Long name | Short name |
|---|---|---|
| Kafka | kafka | k |
| Kafka Node Pool | kafkanodepool | knp |
| Kafka Topic | kafkatopic | kt |
| Kafka User | kafkauser | ku |
| Kafka Connect | kafkaconnect | kc |
| Kafka Connector | kafkaconnector | kctr |
| Kafka MirrorMaker 2 | kafkamirrormaker2 | kmm2 |
| Kafka Bridge | kafkabridge | kb |
| Kafka Rebalance | kafkarebalance | kr |
| Streams for Apache Kafka Pod Set | strimzipodset | sps |
1.1.2.1. Resource categories
Categories of custom resources can also be used in oc commands.
All Streams for Apache Kafka custom resources belong to the category strimzi, so you can use strimzi to get all the Streams for Apache Kafka resources with one command.
For example, running oc get strimzi lists all Streams for Apache Kafka custom resources in a given namespace.
Listing all custom resources
The oc get strimzi -o name command returns all resource types and resource names. The -o name option fetches the output in the type/name format
Listing all resource types and names
You can combine this strimzi command with other commands. For example, you can pass it into a oc delete command to delete all resources in a single command.
Deleting all custom resources
Deleting all resources in a single operation might be useful, for example, when you are testing new Streams for Apache Kafka features.
1.1.2.2. Querying the status of sub-resources
There are other values you can pass to the -o option. For example, by using -o yaml you get the output in YAML format. Using -o json will return it as JSON.
You can see all the options in oc get --help.
One of the most useful options is the JSONPath support, which allows you to pass JSONPath expressions to query the Kubernetes API. A JSONPath expression can extract or navigate specific parts of any resource.
For example, you can use the JSONPath expression {.status.listeners[?(@.name=="tls")].bootstrapServers} to get the bootstrap address from the status of the Kafka custom resource and use it in your Kafka clients.
Here, the command retrieves the bootstrapServers value of the listener named tls:
Retrieving the bootstrap address
oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="tls")].bootstrapServers}{"\n"}'
my-cluster-kafka-bootstrap.myproject.svc:9093
oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="tls")].bootstrapServers}{"\n"}'
my-cluster-kafka-bootstrap.myproject.svc:9093By changing the name condition you can also get the address of the other Kafka listeners.
You can use jsonpath to extract any other property or group of properties from any custom resource.
Status properties provide status information for certain custom resources.
The following table lists the custom resources that provide status information (when deployed) and the schemas that define the status properties.
For more information on the schemas, see the Streams for Apache Kafka Custom Resource API Reference.
| Streams for Apache Kafka resource | Schema reference | Publishes status information on… |
|---|---|---|
|
|
| The Kafka cluster, its listeners, node pools, and any auto-rebalances on scaling |
|
|
| The nodes in the node pool, their roles, and the associated Kafka cluster |
|
|
| Kafka topics in the Kafka cluster |
|
|
| Kafka users in the Kafka cluster |
|
|
| The Kafka Connect cluster and connector plugins |
|
|
| KafkaConnector resources |
|
|
| The Kafka MirrorMaker 2 cluster and internal connectors |
|
|
| The Kafka Bridge |
|
|
| The status and results of a rebalance |
|
|
| The number of pods: being managed, using the current version, and in a ready state |
The status property of a resource provides information on the state of the resource. The status.conditions and status.observedGeneration properties are common to all resources.
status.conditions-
Status conditions describe the current state of a resource. Status condition properties are useful for tracking progress related to the resource achieving its desired state, as defined by the configuration specified in its
spec. Status condition properties provide the time and reason the state of the resource changed, and details of events preventing or delaying the operator from realizing the desired state. status.observedGeneration-
Last observed generation denotes the latest reconciliation of the resource by the Cluster Operator. If the value of
observedGenerationis different from the value ofmetadata.generation(the current version of the deployment), the operator has not yet processed the latest update to the resource. If these values are the same, the status information reflects the most recent changes to the resource.
The status properties also provide resource-specific information. For example, KafkaStatus provides information on listener addresses, and the ID of the Kafka cluster.
KafkaStatus also provides information on the Kafka and Streams for Apache Kafka versions being used. You can check the values of operatorLastSuccessfulVersion and kafkaVersion to determine whether an upgrade of Streams for Apache Kafka or Kafka has completed
Streams for Apache Kafka creates and maintains the status of custom resources, periodically evaluating the current state of the custom resource and updating its status accordingly. When performing an update on a custom resource using oc edit, for example, its status is not editable. Moreover, changing the status would not affect the configuration of the Kafka cluster.
Here we see the status properties for a Kafka custom resource.
Kafka custom resource status
- 1
- The Kafka cluster ID.
- 2
- Status
conditionsdescribe the current state of the Kafka cluster. - 3
- The
Readycondition indicates that the Cluster Operator considers the Kafka cluster able to handle traffic. - 4
- Kafka metadata state that shows KRaft is managing Kafka metadata and coordinating operations.
- 5
- The version of Kafka being used by the Kafka cluster.
- 6
- The node pools belonging to the Kafka cluster.
- 7
- The
listenersdescribe Kafka bootstrap addresses by type. - 8
- The
observedGenerationvalue indicates the last reconciliation of theKafkacustom resource by the Cluster Operator. - 9
- The version of the operator that successfully completed the last reconciliation.
The Kafka bootstrap addresses listed in the status do not signify that those endpoints or the Kafka cluster is in a Ready state.
1.1.4. Finding the status of a custom resource
Use oc with the status subresource of a custom resource to retrieve information about the resource.
Prerequisites
- An OpenShift cluster.
- The Cluster Operator is running.
Procedure
Specify the custom resource and use the
-o jsonpathoption to apply a standard JSONPath expression to select thestatusproperty:oc get kafka <kafka_resource_name> -o jsonpath='{.status}' | jqoc get kafka <kafka_resource_name> -o jsonpath='{.status}' | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow This expression returns all the status information for the specified custom resource. You can use dot notation, such as
status.listenersorstatus.observedGeneration, to fine-tune the status information you wish to see.Using the
jqcommand line JSON parser tool makes it easier to read the output.
1.2. Streams for Apache Kafka operators
Streams for Apache Kafka uses operators to deploy and manage Kafka components. The operators continuously monitor Streams for Apache Kafka custom resources (like Kafka, KafkaTopic, and KafkaUser) and reconcile the state of Kafka components to match their configuration.
This reconciliation process involves three main operations:
- Creation
-
When you create a Streams for Apache Kafka custom resource, the responsible operator detects it and takes the necessary actions to create the component. This might involve creating OpenShift resources, such as
Deployment,Pod,Service, andConfigMap, or configuring items inside the Kafka cluster itself, such as topics and users. - Update
- Each time you update a custom resource, the operator detects the change and applies a corresponding update if the changes are valid. This could trigger a rolling update of pods or reconfigure a resource within Kafka. Rolling updates maintain the availability of the Kafka cluster, but can lead to service disruption in the Kafka clients.
- Deletion
- When you delete a custom resource, the operator detects the deletion and acts to remove the component. Most dependent resources are deleted automatically by OpenShift garbage collection. The exact behavior depends on the resource type. For a Kafka cluster, PVCs are retained by default to prevent data loss. For a Kafka topic, the topic is fully deleted from the Apache Kafka cluster.
Streams for Apache Kafka provides the following operators, each responsible for different aspects of a Kafka deployment:
- Cluster Operator (required)
The Cluster Operator is the core operator and must be deployed first. It handles the deployment and lifecycle of Apache Kafka clusters on OpenShift, automating the setup of Kafka nodes and related resources.
Additionally, Streams for Apache Kafka provides Drain Cleaner, which is deployed separately. Drain Cleaner supports the Cluster Operator in managing pod evictions for Kafka clusters.
- Entity Operator (recommended)
The Entity Operator can be deployed by the Cluster Operator. It runs in a single pod and includes one or both of the following operators:
- Topic Operator to manage Kafka topics.
- User Operator to manage Kafka users.
Each operator runs in a separate container within the Entity Operator pod.
The Topic Operator and User Operator can also be deployed standalone (without the Entity Operator) to manage topics and users for a Kafka cluster that is not managed by Streams for Apache Kafka.
1.2.1. Operator-watched Kafka resources
Operators watch and manage Kafka resources within defined OpenShift namespaces. The namespace scope of where each operator can watch these resources differs.
You can choose the namespace scope for the Cluster Operator. The Topic Operator and the User Operator can only watch a single Kafka cluster in a namespace. And they can only be connected to a single Kafka cluster.
While the operator can be configured to watch multiple namespaces, each watched namespace should contain only one instance of a specific component type, such as one Kafka cluster, to avoid conflicts.
| Operator | Watched resources | Namespace scope |
|---|---|---|
| Cluster Operator |
| Single, multiple, or all |
| Topic Operator |
| Single namespace (one Kafka cluster only) |
| User Operator |
| Single namespace (one Kafka cluster only) |
For a standalone deployment of the Topic Operator or User Operator, you specify a namespace and connection to the Kafka cluster to watch in the configuration.
1.2.2. Managing RBAC resources
The Cluster Operator creates and manages role-based access control (RBAC) resources for Streams for Apache Kafka components that need access to OpenShift resources.
For the Cluster Operator to function, it needs permission within the OpenShift cluster to interact with Kafka resources, such as Kafka and KafkaConnect, as well as managed resources like ConfigMap, Pod, Deployment, and Service.
Permission is specified through the following OpenShift RBAC resources:
-
ServiceAccount -
RoleandClusterRole -
RoleBindingandClusterRoleBinding
The Cluster Operator runs under a service account called strimzi-cluster-operator, which is assigned cluster roles that give it permission to create the necessary RBAC resources for Streams for Apache Kafka components. Role bindings associate the cluster roles with the service account.
OpenShift enforces privilege escalation prevention, meaning the Cluster Operator cannot grant privileges it does not possess, nor can it grant such privileges in a namespace it cannot access. Consequently, the Cluster Operator must have the necessary privileges for all the components it orchestrates.
The Cluster Operator must be able to do the following:
-
Enable the Topic Operator to manage
KafkaTopicresources by creatingRoleandRoleBindingresources in the relevant namespace. -
Enable the User Operator to manage
KafkaUserresources by creatingRoleandRoleBindingresources in the relevant namespace. -
Allow Streams for Apache Kafka to discover the failure domain of a
Nodeby creating aClusterRoleBinding.
When using rack-aware partition assignment, broker pods need to access information about the Node they are running on, such as the Availability Zone in Amazon AWS. Similarly, when using NodePort type listeners, broker pods need to advertise the address of the Node they are running on. Since a Node is a cluster-scoped resource, this access must be granted through a ClusterRoleBinding, not a namespace-scoped RoleBinding.
The following sections describe the RBAC resources required by the Cluster Operator.
1.2.2.2. ClusterRole resources
The Cluster Operator uses ClusterRole resources to provide the necessary access to resources. Depending on the OpenShift cluster setup, a cluster administrator might be needed to create the cluster roles.
Cluster administrator rights are only needed for the creation of ClusterRole resources. The Cluster Operator will not run under a cluster admin account.
The RBAC resources follow the principle of least privilege and contain only those privileges needed by the Cluster Operator to operate the cluster of the Kafka component.
All cluster roles are required by the Cluster Operator in order to delegate privileges.
| Name | Description |
|---|---|
|
| Access rights for namespace-scoped resources used by the Cluster Operator to deploy and manage the operands. |
|
| Access rights for cluster-scoped resources used by the Cluster Operator to deploy and manage the operands. |
|
| Access rights used by the Cluster Operator for leader election. |
|
| Access rights used by the Cluster Operator to watch and manage the Streams for Apache Kafka custom resources. |
|
| Access rights to allow Kafka brokers to get the topology labels from OpenShift worker nodes when rack-awareness is used. |
|
| Access rights used by the Topic and User Operators to manage Kafka users and topics. |
|
| Access rights to allow Kafka Connect, MirrorMaker (1 and 2), and Kafka Bridge to get the topology labels from OpenShift worker nodes when rack-awareness is used. |
1.2.2.3. ClusterRoleBinding resources
The Cluster Operator uses ClusterRoleBinding and RoleBinding resources to associate its ClusterRole with its ServiceAccount. Cluster role bindings are required by cluster roles containing cluster-scoped resources.
| Name | Description |
|---|---|
|
|
Grants the Cluster Operator the rights from the |
|
|
Grants the Cluster Operator the rights from the |
|
|
Grants the Cluster Operator the rights from the |
| Name | Description |
|---|---|
|
|
Grants the Cluster Operator the rights from the |
|
|
Grants the Cluster Operator the rights from the |
|
|
Grants the Cluster Operator the rights from the |
|
|
Grants the Cluster Operator the rights from the |
1.2.2.4. ServiceAccount resources
The Cluster Operator runs using the strimzi-cluster-operator ServiceAccount. This service account grants it the privileges it requires to manage the operands. The Cluster Operator creates additional ClusterRoleBinding and RoleBinding resources to delegate some of these RBAC rights to the operands.
Each of the operands uses its own service account created by the Cluster Operator. This allows the Cluster Operator to follow the principle of least privilege and give the operands only the access rights that are really need.
| Name | Used by |
|---|---|
|
| Kafka broker pods |
|
| Entity Operator |
|
| Cruise Control pods |
|
| Kafka Exporter pods |
|
| Kafka Connect pods |
|
| MirrorMaker pods |
|
| MirrorMaker 2 pods |
|
| Kafka Bridge pods |
1.2.3. Managing pod resources
The StrimziPodSet custom resource is used by Streams for Apache Kafka to create and manage Kafka, Kafka Connect, and MirrorMaker 2 pods.
You must not create, update, or delete StrimziPodSet resources. The StrimziPodSet custom resource is used internally and resources are managed solely by the Cluster Operator. As a consequence, the Cluster Operator must be running properly to avoid the possibility of pods not starting and Kafka clusters not being available.
OpenShift Deployment resources are used for creating and managing the pods of other components.
1.2.4. Lock acquisition warnings for cluster operations
The Cluster Operator ensures that only one operation runs at a time for each cluster by using locks. If another operation attempts to start while a lock is held, it waits until the current operation completes.
Operations such as cluster creation, rolling updates, scaling down, and scaling up are managed by the Cluster Operator.
If acquiring a lock takes longer than the configured timeout (STRIMZI_OPERATION_TIMEOUT_MS), a DEBUG message is logged:
Example DEBUG message for lock acquisition
DEBUG AbstractOperator:406 - Reconciliation #55(timer) Kafka(myproject/my-cluster): Failed to acquire lock lock::myproject::Kafka::my-cluster within 10000ms.
DEBUG AbstractOperator:406 - Reconciliation #55(timer) Kafka(myproject/my-cluster): Failed to acquire lock lock::myproject::Kafka::my-cluster within 10000ms.
Timed-out operations are retried during the next periodic reconciliation in intervals defined by STRIMZI_FULL_RECONCILIATION_INTERVAL_MS (by default 120 seconds).
If an INFO message continues to appear with the same same reconciliation number, it might indicate a lock release error:
Example INFO message for reconciliation
INFO AbstractOperator:399 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Reconciliation is in progress
INFO AbstractOperator:399 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Reconciliation is in progressRestarting the Cluster Operator can resolve such issues.
You can use the Kafka Bridge API to create and manage consumers and send and receive records over HTTP rather than the native Kafka protocol.
When you set up the Kafka Bridge you configure HTTP access to the Kafka cluster. You can then use the Kafka Bridge to produce and consume messages from the cluster, as well as performing other operations through its REST interface.
1.4. Seamless FIPS support
Federal Information Processing Standards (FIPS) are standards for computer security and interoperability. When running Streams for Apache Kafka on a FIPS-enabled OpenShift cluster, the OpenJDK used in Streams for Apache Kafka container images automatically switches to FIPS mode. From version 2.3, Streams for Apache Kafka can run on FIPS-enabled OpenShift clusters without any changes or special configuration. It uses only the FIPS-compliant security libraries from the OpenJDK.
If you are using FIPS-enabled OpenShift clusters, you may experience higher memory consumption compared to regular OpenShift clusters. To avoid any issues, we suggest increasing the memory request to at least 512Mi.
For more information about the NIST validation program and validated modules, see Cryptographic Module Validation Program on the NIST website.
Compatibility with Streams for Apache Kafka Proxy and Streams for Apache Kafka Console has not been tested regarding FIPS support. While they are expected to function properly, we cannot guarantee full support at this time.
1.4.1. Minimum password length
When running in the FIPS mode, SCRAM-SHA-512 passwords need to be at least 32 characters long. From Streams for Apache Kafka 2.3, the default password length in Streams for Apache Kafka User Operator is set to 32 characters as well. If you have a Kafka cluster with custom configuration that uses a password length that is less than 32 characters, you need to update your configuration. If you have any users with passwords shorter than 32 characters, you need to regenerate a password with the required length. You can do that, for example, by deleting the user secret and waiting for the User Operator to create a new password with the appropriate length.
1.5. Document conventions
User-replaced values, also known as replaceables, are shown in with angle brackets (< >). Underscores ( _ ) are used for multi-word values. If the value refers to code or commands, monospace is also used.
For example, the following code shows that <my_namespace> must be replaced by the correct namespace name:
sed -i 's/namespace: .*/namespace: <my_namespace>/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: <my_namespace>/' install/cluster-operator/*RoleBinding*.yamlChapter 2. Using Kafka in KRaft mode
KRaft (Kafka Raft metadata) mode replaces Kafka’s dependency on ZooKeeper for cluster management. KRaft mode simplifies the deployment and management of Kafka clusters by bringing metadata management and coordination of clusters into Kafka.
Kafka in KRaft mode is designed to offer enhanced reliability, scalability, and throughput. Metadata operations become more efficient as they are directly integrated. And by removing the need to maintain a ZooKeeper cluster, there’s also a reduction in the operational and security overhead.
To deploy a Kafka cluster in KRaft mode, you must use Kafka and KafkaNodePool custom resources. For more details and examples, see Section 8.2.1, “Deploying a Kafka cluster”.
Through node pool configuration using KafkaNodePool resources, nodes are assigned the role of broker, controller, or both:
- Controller nodes operate in the control plane to manage cluster metadata and the state of the cluster using a Raft-based consensus protocol.
- Broker nodes operate in the data plane to manage the streaming of messages, receiving and storing data in topic partitions.
- Dual-role nodes fulfill the responsibilities of controllers and brokers.
Controllers use a metadata log, stored as a single-partition topic (__cluster_metadata) on every node, which records the state of the cluster. When requests are made to change the cluster configuration, an active (lead) controller manages updates to the metadata log, and follower controllers replicate these updates. The metadata log stores information on brokers, replicas, topics, and partitions, including the state of in-sync replicas and partition leadership. Kafka uses this metadata to coordinate changes and manage the cluster effectively.
Broker nodes act as observers, storing the metadata log passively to stay up-to-date with the cluster’s state. Each node fetches updates to the log independently. If you are using JBOD storage, you can change the volume that stores the metadata log.
The KRaft metadata version used in the Kafka cluster must be supported by the Kafka version in use. Both versions are managed through the Kafka resource configuration. For more information, see Section 10.2, “Configuring Kafka”.
In the following example, a Kafka cluster comprises a quorum of controller and broker nodes for fault tolerance and high availability.
Figure 2.1. Example cluster with separate broker and controller nodes
In a typical production environment, use dedicated broker and controller nodes. However, you might want to use nodes in a dual-role configuration for development or testing.
You can use a combination of nodes that combine roles with nodes that perform a single role. In the following example, three nodes perform a dual role and two nodes act only as brokers.
Figure 2.2. Example cluster with dual-role nodes and dedicated broker nodes
2.1. KRaft limitations
KRaft limitations primarily relate to controller scaling, which impacts cluster operations.
2.1.1. Controller scaling
KRaft mode supports two types of controller quorums:
-
Static controller quorums
In this mode, the number of controllers is fixed, and scaling requires downtime. -
Dynamic controller quorums
This mode enables dynamic scaling of controllers without downtime. New controllers join as observers, replicate the metadata log, and eventually become eligible to join the quorum. If a controller being removed is the active controller, it will step down from the quorum only after the new quorum is confirmed.
Scaling is useful not only for adding or removing controllers, but supports the following operations:
- Renaming a node pool, which involves adding a new node pool with the desired name and deleting the old one.
- Changing non-JBOD storage, which requires creating a new node pool with the updated storage configuration and removing the old one.
Dynamic controller quorums provide the flexibility to make these operations significantly easier to perform.
2.1.2. Limitations with static controller quorums
Migration between static and dynamic controller quorums is not currently supported by Apache Kafka, though it is expected to be introduced in a future release. As a result, Streams for Apache Kafka uses static controller quorums for all deployments, including new installations. All pre-existing KRaft-based Apache Kafka clusters that use static controller quorums must continue using them. To ensure compatibility with existing KRaft-based clusters, Streams for Apache Kafka continues to use static controller quorums as well.
This limitation means dynamic scaling of controller quorums cannot be used to support the following:
- Adding or removing node pools with controller roles
- Adding the controller role to an existing node pool
- Removing the controller role from an existing node pool
- Scaling a node pool with the controller role
- Renaming a node pool with the controller role
Static controller quorums also limit operations that require scaling. For example, changing the storage type for a node pool with a controller role is not possible because it involves scaling the controller quorum. For non-JBOD storage, creating a new node pool with the desired storage type, adding it to the cluster, and removing the old one would require scaling operations, which are not supported. In some cases, workarounds are possible. For instance, when modifying node pool roles to combine controller and broker functions, you can add broker roles to controller nodes instead of adding controller roles to broker nodes to avoid controller scaling. However, this approach would require reassigning more data, which may temporarily affect cluster performance.
Once migration is possible, Streams for Apache Kafka plans to assess introducing support for dynamic quorums.
2.2. Migrating ZooKeeper-based Kafka clusters
Kafka 4.0 runs exclusively in KRaft mode, with no ZooKeeper integration. As a result of this change, Streams for Apache Kafka removed support for ZooKeeper-based Kafka clusters starting with version 3.0.
To upgrade to Streams for Apache Kafka 3.0 or later, first migrate any ZooKeeper-based Kafka clusters to KRaft mode.
NOTE: To perform the migration before upgrading, follow the procedure outlined in the Streams for Apache Kafka 2.9.x documentation. For more information, see Migrating to KRaft mode.
Chapter 3. Deployment methods
You can deploy Streams for Apache Kafka on OpenShift 4.16 to 4.20 using one of the following methods:
| Installation method | Description |
|---|---|
|
Download the deployment files to manually deploy Streams for Apache Kafka components. The installation files bundle all the necessary OpenShift resources, including the Custom Resource Definitions (CRDs) that define resources like | |
| Install Streams for Apache Kafka using the Operator Lifecycle Manager (OLM) from OperatorHub. Once the Cluster Operator is running, you can deploy Streams for Apache Kafka components using custom resources. This method provides a standard configuration and supports automatic updates for streamlined lifecycle management. |
All deployment methods assume that you have access to a running OpenShift cluster with appropriate permissions. Some methods may also require additional setup, such as access to container registries.
Chapter 4. Deployment path
You can configure a deployment where Streams for Apache Kafka manages a single Kafka cluster in the same namespace, suitable for development or testing. Alternatively, Streams for Apache Kafka can manage multiple Kafka clusters across different namespaces in a production environment.
The basic deployment path includes the following steps:
- Create an OpenShift namespace for the Cluster Operator.
- Deploy the Cluster Operator based on your chosen deployment method.
- Deploy the Kafka cluster, including the Topic Operator and User Operator if desired.
Optionally, deploy additional components:
- The Topic Operator and User Operator as standalone components, if not deployed with the Kafka cluster
- Kafka Connect
- Kafka MirrorMaker
- Kafka Bridge
- Metrics monitoring components
The Cluster Operator creates OpenShift resources such as Deployment, Service, and Pod for each component. The resource names are appended with the name of the deployed component. For example, a Kafka cluster named my-kafka-cluster will have a service named my-kafka-cluster-kafka.
Chapter 5. Downloading deployment files
To deploy Streams for Apache Kafka components using YAML files, download and extract the latest release archive (streams-3.1-ocp-install-examples.zip) from the Streams for Apache Kafka software downloads page.
The release archive contains sample YAML files for deploying Streams for Apache Kafka components to OpenShift using oc.
Begin by deploying the Cluster Operator from the install/cluster-operator directory to watch a single namespace, multiple namespaces, or all namespaces.
In the install folder, you can also deploy other Streams for Apache Kafka components, including:
-
Streams for Apache Kafka administrator roles (
strimzi-admin) -
Standalone Topic Operator (
topic-operator) -
Standalone User Operator (
user-operator) -
Streams for Apache Kafka Drain Cleaner (
drain-cleaner)
The examples folder provides examples of Streams for Apache Kafka custom resources to help you develop your own Kafka configurations.
Streams for Apache Kafka container images are available through the Red Hat Ecosystem Catalog, but we recommend using the provided YAML files for deployment.
5.1. Deploying the Streams for Apache Kafka Proxy
Streams for Apache Kafka Proxy is an Apache Kafka protocol-aware proxy designed to enhance Kafka-based systems. Through its filter mechanism it allows additional behavior to be introduced into a Kafka-based system without requiring changes to either your applications or the Kafka cluster itself.
For more information on connecting to and using the Streams for Apache Kafka Proxy, see the proxy guide in the Streams for Apache Kafka documentation.
5.2. Deploying the Streams for Apache Kafka Console
After you have deployed a Kafka cluster that’s managed by Streams for Apache Kafka, you can deploy and connect the Streams for Apache Kafka Console to the cluster. The console facilitates the administration of Kafka clusters, providing real-time insights for monitoring, managing, and optimizing each cluster from its user interface.
For more information on connecting to and using the Streams for Apache Kafka Console, see the console guide in the Streams for Apache Kafka documentation.
Prepare for a deployment of Streams for Apache Kafka by completing any necessary pre-deployment tasks. Take the necessary preparatory steps according to your specific requirements, such as the following:
- Ensuring you have the necessary prerequisites before deploying Streams for Apache Kafka
- Considering operator deployment best practices
- Pushing the Streams for Apache Kafka container images into your own registry (if required)
- Creating a pull secret for authentication to the container image registry
- Setting up admin roles to enable configuration of custom resources used in the deployment
To run the commands in this guide, your cluster user must have the rights to manage role-based access control (RBAC) and CRDs.
6.1. Deployment prerequisites
To deploy Streams for Apache Kafka, you will need the following:
An OpenShift 4.16–4.20 (tested); 4.12, 4.14 (supported) cluster.
Streams for Apache Kafka is based on Strimzi 0.48.x.
-
The
occommand-line tool is installed and configured to connect to the running cluster.
6.2. Planning your Cluster Operator deployment
To support a stable and reliable Streams for Apache Kafka deployment, follow the best practices in this section. Run a single Cluster Operator per OpenShift cluster, choose an appropriate watch strategy, and isolate components within watched namespaces to reduce the risk of conflicts and unexpected behavior.
6.2.1. Avoiding deployment conflicts
A single operator is capable of managing multiple Kafka clusters across different namespaces. Deploying multiple instances of the Cluster Operator, particularly with different versions, introduces the following risks:
- Resource conflicts
- Conflicts over cluster-scoped resources like Custom Resource Definitions (CRDs) and ClusterRoles, leading to unpredictable behavior. This conflict occurs even when the operators are deployed in separate namespaces.
- Version incompatibility
- Different operator versions can create compatibility issues with the Kafka clusters they manage. New Streams for Apache Kafka releases may introduce features, bug fixes, or other changes that are not backward-compatible.
Approach to avoid risks
To avoid these risks, the recommended approach to deploying the Cluster Operator is as follows:
- Run a single Cluster Operator
- Deploy only one Cluster Operator per OpenShift cluster.
- Consider a dedicated namespace
- Install the Cluster Operator in its own namespace, separate from the Kafka components it manages. This separation is most useful when the operator is configured to watch multiple namespaces, but it can also help prevent uncontrolled growth of resources in a single namespace.
- Keep everything updated
- Regularly update Streams for Apache Kafka and the version of Kafka it manages so that you have the latest features, bug fixes, and enhancements.
6.2.2. Choosing namespace watch options
You configure the Cluster Operator to watch for changes to Kafka resources in specific namespaces.
You can configure the operator to watch:
Choosing to watch a specific list of multiple namespaces can have the biggest impact on performance due to increased processing overhead. To optimize performance, the recommended modes are to either watch a single namespace for focused monitoring or all namespaces for a comprehensive view of the entire cluster.
6.2.3. Isolating components in watched namespaces
After deploying the Cluster Operator, it begins watching specified namespaces for changes to Kafka resources. To reduce risks and maintain reliability, isolate component types within each watched namespace. Each namespace should contain only one instance of a given component type, such as one Kafka cluster, to avoid the following types of issues:
- Conflicting resource names
- Ambiguity in access management
- Topic and user name collisions
- Unpredictable behavior during upgrades or recovery
As Streams for Apache Kafka is based on Strimzi, the same issues can also arise when combining Streams for Apache Kafka operators with Strimzi operators in an OpenShift cluster.
6.3. Pushing container images to your own registry
Container images for Streams for Apache Kafka are available in the Red Hat Ecosystem Catalog. The installation YAML files provided by Streams for Apache Kafka pull the images directly from the Red Hat Ecosystem Catalog.
If you do not have access to the Red Hat Ecosystem Catalog, or want to use your own container repository, do the following:
- Pull all container images listed here.
- Push them into your own registry.
- Update the image names in the installation YAML files.
Each Kafka version supported for the release has a separate image.
6.3.1. Streams for Apache Kafka container images
| Container image | Namespace/Repository | Description |
|---|---|---|
| Kafka |
| Images for running Kafka components, including:
|
| Operator |
| Images for running the Streams for Apache Kafka operators:
|
| Kafka Bridge |
| Image for running the Streams for Apache Kafka Bridge |
| Streams for Apache Kafka Drain Cleaner |
| Image for running the Streams for Apache Kafka Drain Cleaner |
| Streams for Apache Kafka Proxy |
| Images for running the Streams for Apache Kafka Proxy |
| Streams for Apache Kafka Console |
| Images for running the Streams for Apache Kafka Console |
The installation YAML files provided by Streams for Apache Kafka pull container images directly from the Red Hat Ecosystem Catalog. If a Streams for Apache Kafka deployment requires authentication, configure authentication credentials in a secret and add it to the installation YAML.
Authentication is not usually required, but might be requested on certain platforms.
Prerequisites
- You need your Red Hat username and password or the login details from your Red Hat registry service account.
You can use your Red Hat subscription to create a registry service account from the Red Hat Customer Portal.
Procedure
Create a pull secret containing your login details and the container registry where the Streams for Apache Kafka image is pulled from:
oc create secret docker-registry <pull_secret_name> \ --docker-server=registry.redhat.io \ --docker-username=<user_name> \ --docker-password=<password> \ --docker-email=<email>oc create secret docker-registry <pull_secret_name> \ --docker-server=registry.redhat.io \ --docker-username=<user_name> \ --docker-password=<password> \ --docker-email=<email>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add your user name and password. The email address is optional.
Edit the
install/cluster-operator/060-Deployment-strimzi-cluster-operator.yamldeployment file to specify the pull secret using theSTRIMZI_IMAGE_PULL_SECRETSenvironment variable:Copy to Clipboard Copied! Toggle word wrap Toggle overflow The secret applies to all pods created by the Cluster Operator.
6.5. Designating Streams for Apache Kafka administrators
Streams for Apache Kafka provides custom resources for configuration of your deployment. By default, permission to view, create, edit, and delete these resources is limited to OpenShift cluster administrators. Streams for Apache Kafka provides two cluster roles that you can use to assign these rights to other users:
-
strimzi-viewallows users to view and list Streams for Apache Kafka resources. -
strimzi-adminallows users to also create, edit or delete Streams for Apache Kafka resources.
When you install these roles, they will automatically aggregate (add) these rights to the default OpenShift cluster roles. strimzi-view aggregates to the view role, and strimzi-admin aggregates to the edit and admin roles. Because of the aggregation, you might not need to assign these roles to users who already have similar rights.
The following procedure shows how to assign a strimzi-admin role that allows non-cluster administrators to manage Streams for Apache Kafka resources.
A system administrator can designate Streams for Apache Kafka administrators after the Cluster Operator is deployed.
Prerequisites
- The Streams for Apache Kafka admin deployment files, which are included in the Streams for Apache Kafka deployment files.
- The Streams for Apache Kafka Custom Resource Definitions (CRDs) and role-based access control (RBAC) resources to manage the CRDs have been deployed with the Cluster Operator.
Procedure
Create the
strimzi-viewandstrimzi-admincluster roles in OpenShift.oc create -f install/strimzi-admin
oc create -f install/strimzi-adminCopy to Clipboard Copied! Toggle word wrap Toggle overflow If needed, assign the roles that provide access rights to users that require them.
oc create clusterrolebinding strimzi-admin --clusterrole=strimzi-admin --user=user1 --user=user2
oc create clusterrolebinding strimzi-admin --clusterrole=strimzi-admin --user=user1 --user=user2Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Deploy Streams for Apache Kafka by installing the operator from the OperatorHub in the OpenShift Container Platform web console.
The procedures in this section show how to:
You can install and subscribe to the Streams for Apache Kafka operator using the OperatorHub in the OpenShift Container Platform web console.
This procedure describes how to create a project and install the Streams for Apache Kafka operator to that project. A project is a representation of a namespace. For manageability, it is a good practice to use namespaces to separate functions.
Make sure you use the appropriate update channel. If you are on a supported version of OpenShift, installing Streams for Apache Kafka from the default stable channel is generally safe. However, we do not recommend enabling automatic updates on the stable channel. An automatic upgrade will skip any necessary steps prior to upgrade. Use automatic upgrades only on version-specific channels.
Prerequisites
-
Access to an OpenShift Container Platform web console using an account with
cluster-adminorstrimzi-adminpermissions.
Procedure
Navigate in the OpenShift web console to the Home > Projects page and create a project (namespace) for the installation.
We use a project named
streams-kafkain this example.- Navigate to the Operators > OperatorHub page.
Scroll or type a keyword into the Filter by keyword box to find the Streams for Apache Kafka operator.
The operator is located in the Streaming & Messaging category.
- Click Streams for Apache Kafka to display the operator information.
- Read the information about the operator and click Install.
On the Install Operator page, choose from the following installation and update options:
Update Channel: Choose the update channel for the operator.
- The (default) stable channel contains all the latest updates and releases, including major, minor, and micro releases, which are assumed to be well tested and stable.
- An amq-streams-X.x channel contains the minor and micro release updates for a major release, where X is the major release version number.
- An amq-streams-X.Y.x channel contains the micro release updates for a minor release, where X is the major release version number and Y is the minor release version number.
Installation Mode: Choose the project you created to install the operator on a specific namespace.
You can install the Streams for Apache Kafka operator to all namespaces in the cluster (the default option) or a specific namespace. We recommend that you dedicate a specific namespace to the Kafka cluster and other Streams for Apache Kafka components.
- Update approval: By default, the Streams for Apache Kafka operator is automatically upgraded to the latest Streams for Apache Kafka version by the Operator Lifecycle Manager (OLM). Optionally, select Manual if you want to manually approve future upgrades. For more information on operators, see the OpenShift documentation.
Click Install to install the operator to your selected namespace.
The Streams for Apache Kafka operator deploys the Cluster Operator, CRDs, and role-based access control (RBAC) resources to the selected namespace.
After the operator is ready for use, navigate to Operators > Installed Operators to verify that the operator has installed to the selected namespace.
The status will show as Succeeded.
You can now use the Streams for Apache Kafka operator to deploy Kafka components, starting with a Kafka cluster and node pools.
If you navigate to Workloads > Deployments, you can see the deployment details for the Cluster Operator and Entity Operator. The name of the Cluster Operator includes a version number: amq-streams-cluster-operator-<version>. The name is different when deploying the Cluster Operator using the Streams for Apache Kafka installation artifacts. In this case, the name is strimzi-cluster-operator.
When installed on Openshift, the Streams for Apache Kafka operator makes Kafka components available for installation from the user interface.
The following Kafka components are available for installation:
- Kafka
- Kafka Node Pool
- Kafka Connect
- Kafka MirrorMaker 2
- Kafka Topic
- Kafka User
- Kafka Bridge
- Kafka Connector
- Kafka Rebalance
You select the component and create an instance. As a minimum, you create a Kafka instance and node pool. This procedure describes how to create a Kafka instance with separate node pools for brokers and controllers. You can configure the default installation specification before you perform the installation.
The process is the same for creating instances of other Kafka components.
Prerequisites
- The Streams for Apache Kafka operator is installed on the OpenShift cluster.
Procedure
Navigate in the web console to the Operators > Installed Operators page and click Streams for Apache Kafka to display the operator details.
From Provided APIs, you can create instances of Kafka components.
Click Create instance under Kafka to create a Kafka instance.
By default, you’ll create a Kafka cluster called
my-cluster:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Click Create to start the installation of Kafka.
The
Kafkaresource remains in a pending state until at least one node pool is created.Click Create instance under KafkaNodePool to create a node pool instance.
Switch to YAML view and paste a minimal broker pool configuration with ephemeral storage:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Click Create instance under KafkaNodePool to create a second node pool instance.
Switch to YAML view and paste a minimal controller pool configuration with ephemeral storage:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Select the Kafka page to show the installed Kafka clusters. Wait until the status of the Kafka cluster changes to Ready.
These examples use ephemeral storage for evaluation only. For production deployments, configure persistent volumes.
Download and use the Streams for Apache Kafka deployment files to deploy Streams for Apache Kafka components to an OpenShift cluster.
Streams for Apache Kafka is based on Strimzi 0.48.x. You can deploy Streams for Apache Kafka 3.1 on OpenShift 4.16–4.20 (tested); 4.12, 4.14 (supported).
The steps to deploy Streams for Apache Kafka using the installation files are as follows:
- Deploy the Cluster Operator
Use the Cluster Operator to deploy the following:
Optionally, deploy the following Kafka components according to your requirements:
To run the commands in this guide, an OpenShift user must have the rights to manage role-based access control (RBAC) and CRDs.
8.1. Deploying the Cluster Operator
The first step for any deployment of Streams for Apache Kafka is to install the Cluster Operator, which is responsible for deploying and managing Kafka clusters within an OpenShift cluster.
A single command applies all the installation files in the install/cluster-operator folder: oc apply -f ./install/cluster-operator.
The command sets up everything you need to be able to create and manage a Kafka deployment, including the following resources:
-
Cluster Operator (
Deployment,ConfigMap) -
Streams for Apache Kafka CRDs (
CustomResourceDefinition) -
RBAC resources (
ClusterRole,ClusterRoleBinding,RoleBinding) -
Service account (
ServiceAccount)
Cluster-scoped resources like CustomResourceDefinition, ClusterRole, and ClusterRoleBinding require administrator privileges for installation. Prior to installation, it’s advisable to review the ClusterRole specifications to ensure they do not grant unnecessary privileges.
After installation, the Cluster Operator runs as a regular Deployment to watch for updates of Kafka resources. For more information, see Operator-watched resources.
Any standard (non-admin) OpenShift user with privileges to access the Deployment can configure it. A cluster administrator can also grant standard users the privileges necessary to manage Streams for Apache Kafka custom resources.
By default, a single replica of the Cluster Operator is deployed. You can add replicas with leader election so that additional Cluster Operators are on standby in case of disruption. For more information, see Section 10.6.4, “Running multiple Cluster Operator replicas with leader election”.
While the operator can be configured to watch multiple namespaces, each watched namespace should contain only one instance of a specific component type, such as one Kafka cluster, to avoid conflicts.
This procedure shows how to deploy the Cluster Operator to watch Streams for Apache Kafka resources in a single namespace in your OpenShift cluster.
Prerequisites
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Procedure
Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.
For example, in this procedure the Cluster Operator is installed into the namespace
my-cluster-operator-namespace.On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Cluster Operator:
oc create -f install/cluster-operator -n my-cluster-operator-namespace
oc create -f install/cluster-operator -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments -n my-cluster-operator-namespace
oc get deployments -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
This procedure shows how to deploy the Cluster Operator to watch Streams for Apache Kafka resources across multiple namespaces in your OpenShift cluster.
Prerequisites
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Procedure
Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.
For example, in this procedure the Cluster Operator is installed into the namespace
my-cluster-operator-namespace.On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
install/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlfile to add a list of all the namespaces the Cluster Operator will watch to theSTRIMZI_NAMESPACEenvironment variable.For example, in this procedure the Cluster Operator will watch the namespaces
watched-namespace-1,watched-namespace-2,watched-namespace-3.Copy to Clipboard Copied! Toggle word wrap Toggle overflow For each namespace listed, install the
RoleBindings.In this example, we replace
watched-namespacein these commands with the namespaces listed in the previous step, repeating them forwatched-namespace-1,watched-namespace-2,watched-namespace-3:oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>
oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Cluster Operator:
oc create -f install/cluster-operator -n my-cluster-operator-namespace
oc create -f install/cluster-operator -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments -n my-cluster-operator-namespace
oc get deployments -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
This procedure shows how to deploy the Cluster Operator to watch Streams for Apache Kafka resources across all namespaces in your OpenShift cluster.
When running in this mode, the Cluster Operator automatically manages clusters in any new namespaces that are created.
Prerequisites
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Procedure
Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.
For example, in this procedure the Cluster Operator is installed into the namespace
my-cluster-operator-namespace.On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
install/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlfile to set the value of theSTRIMZI_NAMESPACEenvironment variable to*.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create
ClusterRoleBindingsthat grant cluster-wide access for all namespaces to the Cluster Operator.oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Cluster Operator to your OpenShift cluster.
oc create -f install/cluster-operator -n my-cluster-operator-namespace
oc create -f install/cluster-operator -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments -n my-cluster-operator-namespace
oc get deployments -n my-cluster-operator-namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
8.2. Deploying Kafka
To be able to manage a Kafka cluster with the Cluster Operator, you must deploy it as a Kafka resource. Streams for Apache Kafka provides example deployment files to do this. You can use these files to deploy the Topic Operator and User Operator at the same time.
After you have deployed the Cluster Operator, use a Kafka resource to deploy the following components:
Node pools are used in the deployment of a Kafka cluster. Node pools represent a distinct group of Kafka nodes within the Kafka cluster that share the same configuration. For each Kafka node in the node pool, any configuration not defined in node pool is inherited from the cluster configuration in the Kafka resource.
If you haven’t deployed a Kafka cluster as a Kafka resource, you can’t use the Cluster Operator to manage it. This applies, for example, to a Kafka cluster running outside of OpenShift. However, you can use the Topic Operator and User Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, by deploying them as standalone components. You can also deploy and use other Kafka components with a Kafka cluster not managed by Streams for Apache Kafka.
8.2.1. Deploying a Kafka cluster
This procedure shows how to deploy a Kafka cluster and associated node pools using the Cluster Operator.
The deployment uses a YAML file to provide the specification to create a Kafka resource and KafkaNodePool resources.
Streams for Apache Kafka provides the following example deployment files that you can use to create a Kafka cluster that uses node pools:
kafka/kafka-with-dual-role-nodes.yaml- Deploys a Kafka cluster with one pool of nodes that share the broker and controller roles.
kafka/kafka-persistent.yaml- Deploys a persistent Kafka cluster with one pool of controller nodes and one pool of broker nodes.
kafka/kafka-ephemeral.yaml- Deploys an ephemeral Kafka cluster with one pool of controller nodes and one pool of broker nodes.
kafka/kafka-single-node.yaml- Deploys a Kafka cluster with a single node.
kafka/kafka-jbod.yaml- Deploys a Kafka cluster with multiple volumes in each broker node.
In this procedure, we use the example deployment file that deploys a Kafka cluster with one pool of nodes that share the broker and controller roles.
The example YAML files specify the latest supported Kafka version and KRaft metadata version used by the Kafka cluster.
When deploying multiple Kafka clusters managed by the Cluster Operator, deploy each cluster into a separate namespace. Deploying multiple clusters in the same namespace can lead to naming conflicts and resource collisions.
Prerequisites
Before you begin
By default, the example deployment files specify my-cluster as the Kafka cluster name. The name cannot be changed after the cluster has been deployed. To change the cluster name before you deploy the cluster, edit the Kafka.metadata.name property of the Kafka resource in the relevant YAML file.
Procedure
Deploy a Kafka cluster.
To deploy a Kafka cluster with a single node pool that uses dual-role nodes:
oc apply -f examples/kafka/kafka-with-dual-role-nodes.yaml
oc apply -f examples/kafka/kafka-with-dual-role-nodes.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the node pool names and readiness
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-4 1/1 Running 0
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-4 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
my-clusteris the name of the Kafka cluster. pool-ais the name of the node pool.A sequential index number starting with
0identifies each Kafka pod created.READYshows the number of replicas that are ready/expected. The deployment is successful when theSTATUSdisplays asRunning.Information on the deployment is also shown in the status of the
KafkaNodePoolresource, including a list of IDs for nodes in the pool.NoteNode IDs are assigned sequentially starting at 0 (zero) across all node pools within a cluster. This means that node IDs might not run sequentially within a specific node pool. If there are gaps in the sequence of node IDs across the cluster, the next node to be added is assigned an ID that fills the gap. When scaling down, the node with the highest node ID within a pool is removed.
-
This procedure describes how to deploy the Topic Operator using the Cluster Operator.
You configure the entityOperator property of the Kafka resource to include the topicOperator. By default, the Topic Operator watches for KafkaTopic resources in the namespace of the Kafka cluster deployed by the Cluster Operator. You can also specify a namespace using watchedNamespace in the Topic Operator spec. A single Topic Operator can watch a single namespace. One namespace should be watched by only one Topic Operator.
Do not deploy more than one Kafka cluster into the same namespace. This causes the Topic Operator connected to each cluster to compete for the same topic resources using the same names, leading to name collisions and unpredictable behavior.
If you want to use the Topic Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, you must deploy the Topic Operator as a standalone component.
For more information about configuring the entityOperator and topicOperator properties, see Configuring the Entity Operator.
Prerequisites
Procedure
Edit the
entityOperatorproperties of theKafkaresource to includetopicOperator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the Topic Operator
specusing the properties described in theEntityTopicOperatorSpecschema reference.Use an empty object (
{}) if you want all properties to use their default values.Create or update the resource:
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the pod name and readiness
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 # ...
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 # ...Copy to Clipboard Copied! Toggle word wrap Toggle overflow my-clusteris the name of the Kafka cluster.READYshows the number of replicas that are ready/expected. The deployment is successful when theSTATUSdisplays asRunning.
This procedure describes how to deploy the User Operator using the Cluster Operator.
You configure the entityOperator property of the Kafka resource to include the userOperator. By default, the User Operator watches for KafkaUser resources in the namespace of the Kafka cluster deployment. You can also specify a namespace using watchedNamespace in the User Operator spec. A single User Operator can watch a single namespace. One namespace should be watched by only one User Operator.
Do not deploy more than one Kafka cluster into the same namespace. This can cause the User Operator deployed with each cluster to compete for the same KafkaUser resources, leading to name collisions and unpredictable behavior.
If you want to use the User Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, you must deploy the User Operator as a standalone component.
For more information about configuring the entityOperator and userOperator properties, see Configuring the Entity Operator.
Prerequisites
Procedure
Edit the
entityOperatorproperties of theKafkaresource to includeuserOperator:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the User Operator
specusing the properties described inEntityUserOperatorSpecschema reference.Use an empty object (
{}) if you want all properties to use their default values.Create or update the resource:
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the pod name and readiness
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 # ...
NAME READY STATUS RESTARTS my-cluster-entity-operator 3/3 Running 0 # ...Copy to Clipboard Copied! Toggle word wrap Toggle overflow my-clusteris the name of the Kafka cluster.READYshows the number of replicas that are ready/expected. The deployment is successful when theSTATUSdisplays asRunning.
8.2.4. List of Kafka cluster resources
The following resources are created by the Cluster Operator in the OpenShift cluster.
Shared resources
<kafka_cluster_name>-cluster-ca- Secret with the Cluster CA private key used to encrypt the cluster communication.
<kafka_cluster_name>-cluster-ca-cert- Secret with the Cluster CA public key. This key can be used to verify the identity of the Kafka brokers.
<kafka_cluster_name>-clients-ca- Secret with the Clients CA private key used to sign user certificates
<kafka_cluster_name>-clients-ca-cert- Secret with the Clients CA public key. This key can be used to verify the identity of the Kafka users.
<kafka_cluster_name>-cluster-operator-certs- Secret with Cluster operators keys for communication with Kafka.
Kafka brokers
<kafka_cluster_name>-kafkaName given to the following Kafka resources:
- StrimziPodSet for managing the Kafka pods.
- Service account used by the Kafka pods.
- PodDisruptionBudget that applies to all Kafka cluster node pool pods.
<kafka_cluster_name>-kafka-<pod_id>Name given to the following Kafka resources:
- Pods created by the StrimziPodSet.
- ConfigMaps with Kafka broker configuration.
<kafka_cluster_name>-kafka-brokers- Service needed to have DNS resolve the Kafka broker pods IP addresses directly.
<kafka_cluster_name>-kafka-bootstrap- Service can be used as bootstrap servers for Kafka clients connecting from within the OpenShift cluster.
<kafka_cluster_name>-kafka-external-bootstrap-
Bootstrap service for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled. The old service name will be used for backwards compatibility when the listener name is
externaland port is9094. <kafka_cluster_name>-kafka-<pod_id>-
Service used to route traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled. The old service name will be used for backwards compatibility when the listener name is
externaland port is9094. <kafka_cluster_name>-kafka-external-bootstrap-
Bootstrap route for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled and set to type
route. The old route name will be used for backwards compatibility when the listener name isexternaland port is9094. <kafka_cluster_name>-kafka-<pod_id>-
Route for traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled and set to type
route. The old route name will be used for backwards compatibility when the listener name isexternaland port is9094. <kafka_cluster_name>-kafka-<listener_name>-bootstrap- Bootstrap service for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled. The new service name will be used for all other external listeners.
<kafka_cluster_name>-kafka-<listener_name>-<pod_id>- Service used to route traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled. The new service name will be used for all other external listeners.
<kafka_cluster_name>-kafka-<listener_name>-bootstrap-
Bootstrap route for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled and set to type
route. The new route name will be used for all other external listeners. <kafka_cluster_name>-kafka-<listener_name>-<pod_id>-
Route for traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled and set to type
route. The new route name will be used for all other external listeners. <kafka_cluster_name>-<pool_name>-<pod_id>_- Secret with Kafka node public and private keys.
<kafka_cluster_name>-network-policy-kafka- Network policy managing access to the Kafka services.
strimzi-namespace-name-<kafka_cluster_name>-kafka-init- Cluster role binding used by the Kafka brokers.
<kafka_cluster_name>-jmx- Secret with JMX username and password used to secure the Kafka broker port. This resource is created only when JMX is enabled in Kafka.
data-<kafka_cluster_name>-kafka-<pod_id>- Persistent Volume Claim for the volume used for storing data for a specific Kafka broker. This resource is created only if persistent storage is selected for provisioning persistent volumes to store data.
data-<id>-<kafka_cluster_name>-kafka-<pod_id>-
Persistent Volume Claim for the volume
idused for storing data for a specific Kafka broker. This resource is created only if persistent storage is selected for JBOD volumes when provisioning persistent volumes to store data.
Kafka node pools
If you are using Kafka node pools, the resources created apply to the nodes managed in the node pools whether they are operating as brokers, controllers, or both. The naming convention includes the name of the Kafka cluster and the node pool: <kafka_cluster_name>-<pool_name>.
<kafka_cluster_name>-<pool_name>- Name given to the StrimziPodSet for managing the Kafka node pool.
<kafka_cluster_name>-<pool_name>-<pod_id>Name given to the following Kafka node pool resources:
- Pods created by the StrimziPodSet.
- ConfigMaps with Kafka node configuration.
data-<kafka_cluster_name>-<pool_name>-<pod_id>- Persistent Volume Claim for the volume used for storing data for a specific node. This resource is created only if persistent storage is selected for provisioning persistent volumes to store data.
data-<id>-<kafka_cluster_name>-<pool_name>-<pod_id>-
Persistent Volume Claim for the volume
idused for storing data for a specific node. This resource is created only if persistent storage is selected for JBOD volumes when provisioning persistent volumes to store data.
Entity Operator
These resources are only created if the Entity Operator is deployed using the Cluster Operator.
<kafka_cluster_name>-entity-operatorName given to the following Entity Operator resources:
- Deployment with Topic and User Operators.
- Service account used by the Entity Operator.
- Network policy managing access to the Entity Operator metrics.
<kafka_cluster_name>-entity-operator-<random_string>- Pod created by the Entity Operator deployment.
<kafka_cluster_name>-entity-topic-operator-config- ConfigMap with ancillary configuration for Topic Operators.
<kafka_cluster_name>-entity-user-operator-config- ConfigMap with ancillary configuration for User Operators.
<kafka_cluster_name>-entity-topic-operator-certs- Secret with Topic Operator keys for communication with Kafka.
<kafka_cluster_name>-entity-user-operator-certs- Secret with User Operator keys for communication with Kafka.
strimzi-<kafka_cluster_name>-entity-topic-operator- Role binding used by the Entity Topic Operator.
strimzi-<kafka_cluster_name>-entity-user-operator- Role binding used by the Entity User Operator.
Kafka Exporter
These resources are only created if the Kafka Exporter is deployed using the Cluster Operator.
<kafka_cluster_name>-kafka-exporterName given to the following Kafka Exporter resources:
- Deployment with Kafka Exporter.
- Service used to collect consumer lag metrics.
- Service account used by the Kafka Exporter.
- Network policy managing access to the Kafka Exporter metrics.
<kafka_cluster_name>-kafka-exporter-<random_string>- Pod created by the Kafka Exporter deployment.
Cruise Control
These resources are only created if Cruise Control was deployed using the Cluster Operator.
<kafka_cluster_name>-cruise-controlName given to the following Cruise Control resources:
- Deployment with Cruise Control.
- Service used to communicate with Cruise Control.
- Service account used by the Cruise Control.
<kafka_cluster_name>-cruise-control-<random_string>- Pod created by the Cruise Control deployment.
<kafka_cluster_name>-cruise-control-config- ConfigMap that contains the Cruise Control ancillary configuration, and is mounted as a volume by the Cruise Control pods.
<kafka_cluster_name>-cruise-control-certs- Secret with Cruise Control keys for communication with Kafka.
<kafka_cluster_name>-network-policy-cruise-control- Network policy managing access to the Cruise Control service.
8.3. Deploying Kafka Connect
Kafka Connect is an integration toolkit for streaming data between Kafka brokers and other systems using connector plugins. Kafka Connect provides a framework for integrating Kafka with an external data source or target, such as a database or messaging system, for import or export of data using connectors. Connectors are plugins that provide the connection configuration needed.
In Streams for Apache Kafka, Kafka Connect is deployed in distributed mode. Kafka Connect can also work in standalone mode, but this is not supported by Streams for Apache Kafka.
Using the concept of connectors, Kafka Connect provides a framework for moving large amounts of data into and out of your Kafka cluster while maintaining scalability and reliability.
The Cluster Operator manages Kafka Connect clusters deployed using the KafkaConnect resource and connectors created using the KafkaConnector resource.
In order to use Kafka Connect, you need to do the following.
The term connector is used interchangeably to mean a connector instance running within a Kafka Connect cluster, or a connector class. In this guide, the term connector is used when the meaning is clear from the context.
8.3.1. Deploying Kafka Connect
This procedure shows how to deploy a Kafka Connect cluster to your OpenShift cluster using the Cluster Operator.
A Kafka Connect cluster deployment is implemented with a configurable number of nodes (also called workers) that distribute the workload of connectors as tasks, ensuring a scalable and reliable message flow.
The deployment uses a YAML file to provide the specification to create a KafkaConnect resource.
Streams for Apache Kafka provides example configuration files. In this procedure, we use the following example file:
-
examples/connect/kafka-connect.yaml
When deploying multiple Kafka Connect clusters managed by the Cluster Operator, deploy each cluster into a separate namespace. Deploying multiple clusters in the same namespace can lead to naming conflicts and resource collisions.
Prerequisites
- Cluster Operator is deployed.
Kafka cluster is running.
This procedure assumes that the Kafka cluster was deployed using Streams for Apache Kafka.
Procedure
Edit the deployment file to configure connection details (if required).
In
examples/connect/kafka-connect.yaml, add or update the following properties as needed:-
spec.bootstrapServersto specify the Kafka bootstrap address. -
spec.authenticationto specify the authentication type astls,scram-sha-256,scram-sha-512,plain, oroauth.
See theKafkaConnectSpecschema properties for configuration details. -
spec.tls.trustedCertificatesto configure the TLS certificate.
Use[](an empty array) to trust the default Java CAs, or specify secrets containing trusted certificates.
See thetrustedCertificatesproperties for configuration details.
-
Configure the deployment for multiple Kafka Connect clusters (if required).
If you plan to run more than one Kafka Connect cluster in the same environment, each instance must use unique internal topic names and a unique group ID.
Update the
spec.configproperties inkafka-connect.yamlto replace the default values.See Configuring multiple Kafka Connect clusters for details.
Deploy Kafka Connect to your OpenShift cluster. Use the
examples/connect/kafka-connect.yamlfile to deploy Kafka Connect.oc apply -f examples/connect/kafka-connect.yaml
oc apply -f examples/connect/kafka-connect.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY STATUS RESTARTS my-connect-cluster-connect-<pod_id> 1/1 Running 0
NAME READY STATUS RESTARTS my-connect-cluster-connect-<pod_id> 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
my-connect-clusteris the name of the Kafka Connect cluster. A pod ID identifies each created pod. By default, the deployment creates a single Kafka Connect pod.READYshows the number of ready versus expected replicas. The deployment is successful when theSTATUSisRunning.
8.3.2. List of Kafka Connect cluster resources
The following resources are created by the Cluster Operator in the OpenShift cluster:
- <connect_cluster_name>-connect
Name given to the following Kafka Connect resources:
- StrimziPodSet that creates the Kafka Connect worker node pods.
- Headless service that provides stable DNS names to the Kafka Connect pods.
- Service account used by the Kafka Connect pods.
- Pod disruption budget configured for the Kafka Connect worker nodes.
- Network policy managing access to the Kafka Connect REST API.
- <connect_cluster_name>-connect-<pod_id>
- Pods created by the Kafka Connect StrimziPodSet.
- <connect_cluster_name>-connect-api
- Service which exposes the REST interface for managing the Kafka Connect cluster.
- <connect_cluster_name>-connect-config
- ConfigMap which contains the Kafka Connect ancillary configuration and is mounted as a volume by the Kafka Connect pods.
- strimzi-<namespace-name>-<connect_cluster_name>-connect-init
- Cluster role binding used by the Kafka Connect cluster.
- <connect_cluster_name>-connect-build
- Pod used to build a new container image with additional connector plugins (only when Kafka Connect Build feature is used).
- <connect_cluster_name>-connect-dockerfile
- ConfigMap with the Dockerfile generated to build the new container image with additional connector plugins (only when the Kafka Connect build feature is used).
8.4. Adding Kafka Connect connectors
Kafka Connect uses connectors to integrate with other systems to stream data. A connector is an instance of a Kafka Connector class, which can be one of the following type:
- Source connector
- A source connector is a runtime entity that fetches data from an external system and feeds it to Kafka as messages.
- Sink connector
- A sink connector is a runtime entity that fetches messages from Kafka topics and feeds them to an external system.
Kafka Connect uses a plugin architecture to provide the implementation artifacts for connectors. Plugins allow connections to other systems and provide additional configuration to manipulate data. Plugins include connectors and other components, such as data converters and transforms. A connector operates with a specific type of external system. Each connector defines a schema for its configuration. You supply the configuration to Kafka Connect to create a connector instance within Kafka Connect. Connector instances then define a set of tasks for moving data between systems.
Plugins provide a set of one or more artifacts that define a connector and task implementation for connecting to a given kind of data source. The configuration describes the source input data and target output data to feed into and out of Kafka Connect. The plugins might also contain the libraries and files needed to transform the data.
A Kafka Connect deployment can have one or more plugins, but only one version of each plugin. Plugins for many external systems are available for use with Kafka Connect. You can also create your own plugins.
Add connector plugins to Kafka Connect in one of the following ways:
- Configure Kafka Connect to build a new container image with plugins automatically
- Create a Docker image from the base Kafka Connect image (manually or using continuous integration)
After plugins have been added to the container image, you can start, stop, and manage connector instances in the following ways:
You can also create new connector instances using these options.
Configure Kafka Connect so that Streams for Apache Kafka automatically builds a new container image with additional connectors. You define the connector plugins using the .spec.build.plugins property of the KafkaConnect custom resource.
Streams for Apache Kafka automatically downloads and adds the connector plugins into a new container image. The container is pushed into the container repository specified in .spec.build.output and automatically used in the Kafka Connect deployment.
Prerequisites
- The Cluster Operator must be deployed.
- A container registry.
You need to provide your own container registry where images can be pushed to, stored, and pulled from. Streams for Apache Kafka supports private container registries as well as public registries such as Quay or Docker Hub.
Procedure
Configure the
KafkaConnectcustom resource by specifying the container registry in.spec.build.output, and additional connectors in.spec.build.plugins:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource:
oc apply -f <kafka_connect_configuration_file>
$ oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Wait for the new container image to build, and for the Kafka Connect cluster to be deployed.
-
Use the Kafka Connect REST API or
KafkaConnectorcustom resources to use the connector plugins you added.
Rebuilding the container image with new artifacts
A new container image is built automatically when you change the base image (.spec.image) or change the connector plugin artifacts configuration (.spec.build.plugins).
To pull an upgraded base image or to download the latest connector plugin artifacts without changing the KafkaConnect resource, you can trigger a rebuild of the container image associated with the Kafka Connect cluster by applying the annotation strimzi.io/force-rebuild=true to the Kafka Connect StrimziPodSet resource.
The annotation triggers the rebuilding process, fetching any new artifacts for plugins specified in the KafkaConnect custom resource and incorporating them into the container image. The rebuild includes downloads of new plugin artifacts without versions.
Configure Kafka Connect so that connectors are automatically mounted as OpenShift Image Volumes. You define the connector plugins using the .spec.plugins property of the KafkaConnect custom resource. Streams for Apache Kafka automatically mounts them into the Kafka Connect deployment.
This feature requires that the OpenShift Image Volume feature is enabled and supported by your OpenShift cluster.
Prerequisites
- The Cluster Operator must be deployed.
- Your connector plugins must be available in your container registry as container images (OCI artifacts).
Procedure
Configure the
KafkaConnectcustom resource by specifying the additional connector plugins in`.spec.plugins`:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource:
oc apply -f <kafka_connect_configuration_file>
$ oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Wait for the Kafka Connect cluster to be deployed.
-
Use the Kafka Connect REST API or
KafkaConnectorcustom resources to configure and run connectors based on the plugins you added.
Create a custom Docker image with connector plugins from the Kafka Connect base image. Add the custom image to the /opt/kafka/plugins directory.
You can use the Kafka container image on Red Hat Ecosystem Catalog as a base image for creating your own custom image with additional connector plugins. At startup, the Streams for Apache Kafka version of Kafka Connect loads any third-party connector plugins contained in /opt/kafka/plugins.
Use the KafkaConnect.spec.image property to scope the custom image to a single Kafka Connect cluster. Editing the STRIMZI_KAFKA_CONNECT_IMAGES environment variable in the Cluster Operator instead applies the image globally to all Connect clusters.
This procedure shows how to do the following:
- Build a custom image from the Kafka Connect base.
- Push the image to your registry.
-
Reference the image by setting
KafkaConnect.spec.image.
When you specify a custom image, coordinate upgrades to keep the image and the Kafka version aligned.
Prerequisites
Procedure
Create a new
Dockerfileusingregistry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0as the base image:FROM registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 USER root:root COPY ./my-plugins/ /opt/kafka/plugins/ USER 1001
FROM registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 USER root:root COPY ./my-plugins/ /opt/kafka/plugins/ USER 1001Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example plugins file
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The COPY command points to the plugin files to copy to the container image.
This example adds plugins for Debezium connectors (MongoDB, MySQL, and PostgreSQL), though not all files are listed for brevity. Debezium running in Kafka Connect looks the same as any other Kafka Connect task.
- Build the container image.
- Push your custom image to your container registry.
Point the
KafkaConnectcustom resource to the new container image.Edit the
KafkaConnect.spec.imageproperty of the KafkaConnect custom resource to specify your custom image.Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Check the Kafka version you need for your custom image. If spec.version is unspecified, Streams for Apache Kafka defaults to Kafka 4.1.0, which may cause compatibility issues if your Kafka cluster runs an earlier version. Version mismatches may lead to connector failures and runtime errors.
8.4.4. Deploying KafkaConnector resources
Deploy KafkaConnector resources to manage connectors. The KafkaConnector custom resource offers an OpenShift-native approach to management of connectors by the Cluster Operator. You don’t need to send HTTP requests to manage connectors, as with the Kafka Connect REST API. You manage a running connector instance by updating its corresponding KafkaConnector resource, and then applying the updates. The Cluster Operator updates the configurations of the running connector instances. You remove a connector by deleting its corresponding KafkaConnector.
KafkaConnector resources must be deployed to the same namespace as the Kafka Connect cluster they link to.
In the configuration shown in this procedure, the autoRestart feature is enabled (enabled: true) for automatic restarts of failed connectors and tasks. You can also annotate the KafkaConnector resource to restart a connector or restart a connector task manually.
Example connectors
You can use your own connectors or try the examples provided by Streams for Apache Kafka. Up until Apache Kafka 3.1.0, example file connector plugins were included with Apache Kafka. Starting from the 3.1.1 and 3.2.0 releases of Apache Kafka, the examples need to be added to the plugin path as any other connector.
Streams for Apache Kafka provides an example KafkaConnector configuration file (examples/connect/source-connector.yaml) for the example file connector plugins, which creates the following connector instances as KafkaConnector resources:
-
A
FileStreamSourceConnectorinstance that reads each line from the Kafka license file (the source) and writes the data as messages to a single Kafka topic. -
A
FileStreamSinkConnectorinstance that reads messages from the Kafka topic and writes the messages to a temporary file (the sink).
We use the example file to create connectors in this procedure.
The example connectors are not intended for use in a production environment.
Prerequisites
- A Kafka Connect deployment
- The Cluster Operator is running
Procedure
Add the
FileStreamSourceConnectorandFileStreamSinkConnectorplugins to Kafka Connect in one of the following ways:- Configure Kafka Connect to build a new container image with plugins automatically
- Create a Docker image from the base Kafka Connect image (manually or using continuous integration)
Set the
strimzi.io/use-connector-resources annotationtotruein the Kafka Connect configuration.Copy to Clipboard Copied! Toggle word wrap Toggle overflow With the
KafkaConnectorresources enabled, the Cluster Operator watches for them.Edit the
examples/connect/source-connector.yamlfile:Example source connector configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Name of the
KafkaConnectorresource, which is used as the name of the connector. Use any name that is valid for an OpenShift resource. - 2
- Name of the Kafka Connect cluster to create the connector instance in. Connectors must be deployed to the same namespace as the Kafka Connect cluster they link to.
- 3
- Full name of the connector class. This should be present in the image being used by the Kafka Connect cluster.
- 4
- Maximum number of Kafka Connect tasks that the connector can create.
- 5
- Enables automatic restarts of failed connectors and tasks. By default, the number of restarts is indefinite, but you can set a maximum on the number of automatic restarts using the
maxRestartsproperty. - 6
- Connector configuration as key-value pairs.
- 7
- Location of the external data file. In this example, we’re configuring the
FileStreamSourceConnectorto read from the/opt/kafka/LICENSEfile. - 8
- Kafka topic to publish the source data to.
Create the source
KafkaConnectorin your OpenShift cluster:oc apply -f examples/connect/source-connector.yaml
oc apply -f examples/connect/source-connector.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create an
examples/connect/sink-connector.yamlfile:touch examples/connect/sink-connector.yaml
touch examples/connect/sink-connector.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Paste the following YAML into the
sink-connector.yamlfile:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Full name or alias of the connector class. This should be present in the image being used by the Kafka Connect cluster.
- 2
- Connector configuration as key-value pairs.
- 3
- Temporary file to publish the source data to.
- 4
- Kafka topic to read the source data from.
Create the sink
KafkaConnectorin your OpenShift cluster:oc apply -f examples/connect/sink-connector.yaml
oc apply -f examples/connect/sink-connector.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the connector resources were created:
oc get kctr --selector strimzi.io/cluster=<my_connect_cluster> -o name my-source-connector my-sink-connector
oc get kctr --selector strimzi.io/cluster=<my_connect_cluster> -o name my-source-connector my-sink-connectorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <my_connect_cluster> with the name of your Kafka Connect cluster.
In the container, execute
kafka-console-consumer.shto read the messages that were written to the topic by the source connector:oc exec <my_kafka_cluster>-kafka-0 -i -t -- bin/kafka-console-consumer.sh --bootstrap-server <my_kafka_cluster>-kafka-bootstrap.NAMESPACE.svc:9092 --topic my-topic --from-beginning
oc exec <my_kafka_cluster>-kafka-0 -i -t -- bin/kafka-console-consumer.sh --bootstrap-server <my_kafka_cluster>-kafka-bootstrap.NAMESPACE.svc:9092 --topic my-topic --from-beginningCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <my_kafka_cluster> with the name of your Kafka cluster.
Source and sink connector configuration options
The connector configuration is defined in the spec.config property of the KafkaConnector resource.
The FileStreamSourceConnector and FileStreamSinkConnector classes support the same configuration options as the Kafka Connect REST API. Other connectors support different configuration options.
| Name | Type | Default value | Description |
|---|---|---|---|
|
| String | Null | Source file to write messages to. If not specified, the standard input is used. |
|
| List | Null | The Kafka topic to publish data to. |
| Name | Type | Default value | Description |
|---|---|---|---|
|
| String | Null | Destination file to write messages to. If not specified, the standard output is used. |
|
| List | Null | One or more Kafka topics to read data from. |
|
| String | Null | A regular expression matching one or more Kafka topics to read data from. |
8.4.5. Exposing the Kafka Connect API
Use the Kafka Connect REST API as an alternative to using KafkaConnector resources to manage connectors. The Kafka Connect REST API is available as a service running on <connect_cluster_name>-connect-api:8083, where <connect_cluster_name> is the name of your Kafka Connect cluster. The service is created when you create a Kafka Connect instance.
The operations supported by the Kafka Connect REST API are described in the Apache Kafka Connect API documentation.
The strimzi.io/use-connector-resources annotation enables KafkaConnectors. If you applied the annotation to your KafkaConnect resource configuration, you need to remove it to use the Kafka Connect API. Otherwise, manual changes made directly using the Kafka Connect REST API are reverted by the Cluster Operator.
You can add the connector configuration as a JSON object.
Example curl request to add connector configuration
The API is only accessible within the OpenShift cluster. If you want to make the Kafka Connect API accessible to applications running outside of the OpenShift cluster, you can expose it manually by creating one of the following features:
-
LoadBalancerorNodePorttype services -
Ingressresources (Kubernetes only) - OpenShift routes (OpenShift only)
The connection is insecure, so allow external access advisedly.
If you decide to create services, use the labels from the selector of the <connect_cluster_name>-connect-api service to configure the pods to which the service will route the traffic:
Selector configuration for the service
You must also create a NetworkPolicy that allows HTTP requests from external clients.
Example NetworkPolicy to allow requests to the Kafka Connect API
- 1
- The label of the pod that is allowed to connect to the API.
To add the connector configuration outside the cluster, use the URL of the resource that exposes the API in the curl command.
8.4.6. Limiting access to the Kafka Connect API
It is crucial to restrict access to the Kafka Connect API only to trusted users to prevent unauthorized actions and potential security issues. The Kafka Connect API provides extensive capabilities for altering connector configurations, which makes it all the more important to take security precautions. Someone with access to the Kafka Connect API could potentially obtain sensitive information that an administrator may assume is secure.
The Kafka Connect REST API can be accessed by anyone who has authenticated access to the OpenShift cluster and knows the endpoint URL, which includes the hostname/IP address and port number.
For example, suppose an organization uses a Kafka Connect cluster and connectors to stream sensitive data from a customer database to a central database. The administrator uses a configuration provider plugin to store sensitive information related to connecting to the customer database and the central database, such as database connection details and authentication credentials. The configuration provider protects this sensitive information from being exposed to unauthorized users. However, someone who has access to the Kafka Connect API can still obtain access to the customer database without the consent of the administrator. They can do this by setting up a fake database and configuring a connector to connect to it. They then modify the connector configuration to point to the customer database, but instead of sending the data to the central database, they send it to the fake database. By configuring the connector to connect to the fake database, the login details and credentials for connecting to the customer database are intercepted, even though they are stored securely in the configuration provider.
If you are using the KafkaConnector custom resources, then by default the OpenShift RBAC rules permit only OpenShift cluster administrators to make changes to connectors. You can also designate non-cluster administrators to manage Streams for Apache Kafka resources. With KafkaConnector resources enabled in your Kafka Connect configuration, changes made directly using the Kafka Connect REST API are reverted by the Cluster Operator. If you are not using the KafkaConnector resource, the default RBAC rules do not limit access to the Kafka Connect API. If you want to limit direct access to the Kafka Connect REST API using OpenShift RBAC, you need to enable and use the KafkaConnector resources.
For improved security, we recommend configuring the following properties for the Kafka Connect API:
org.apache.kafka.disallowed.login.modules(Kafka 3.4 or later) Set the
org.apache.kafka.disallowed.login.modulesJava system property to prevent the use of insecure login modules. For example, specifyingcom.sun.security.auth.module.JndiLoginModuleprevents the use of the KafkaJndiLoginModule.Example configuration for disallowing login modules
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Only allow trusted login modules and follow the latest advice from Kafka for the version you are using. As a best practice, you should explicitly disallow insecure login modules in your Kafka Connect configuration by using the
org.apache.kafka.disallowed.login.modulessystem property.connector.client.config.override.policySet the
connector.client.config.override.policyproperty toNoneto prevent connector configurations from overriding the Kafka Connect configuration and the consumers and producers it uses.Example configuration to specify connector override policy
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
You can switch from using the Kafka Connect API to using KafkaConnector custom resources to manage your connectors. To make the switch, do the following in the order shown:
-
Deploy
KafkaConnectorresources with the configuration to create your connector instances. -
Enable
KafkaConnectorresources in your Kafka Connect configuration by setting thestrimzi.io/use-connector-resourcesannotation totrue.
If you enable KafkaConnector resources before creating them, you delete all connectors.
To switch from using KafkaConnector resources to using the Kafka Connect API, first remove the annotation that enables the KafkaConnector resources from your Kafka Connect configuration. Otherwise, manual changes made directly using the Kafka Connect REST API are reverted by the Cluster Operator.
When making the switch, check the status of the KafkaConnect resource. The value of metadata.generation (the current version of the deployment) must match status.observedGeneration (the latest reconciliation of the resource). When the Kafka Connect cluster is Ready, you can delete the KafkaConnector resources.
8.5. Deploying Kafka MirrorMaker
Kafka MirrorMaker replicates data between two or more Kafka clusters, within or across data centers. This process is called mirroring to avoid confusion with the concept of Kafka partition replication. MirrorMaker consumes messages from a source cluster and republishes those messages to a target cluster.
Data replication across clusters supports scenarios that require the following:
- Recovery of data in the event of a system failure
- Consolidation of data from multiple source clusters for centralized analysis
- Restriction of data access to a specific cluster
- Provision of data at a specific location to improve latency
8.5.1. Deploying Kafka MirrorMaker
This procedure shows how to deploy a Kafka MirrorMaker 2 cluster to your OpenShift cluster using the Cluster Operator.
The deployment uses a YAML file to provide the specification to create a KafkaMirrorMaker2 resource. MirrorMaker 2 is based on Kafka Connect and uses its configuration properties.
Streams for Apache Kafka provides example configuration files. In this procedure, we use the following example file:
-
examples/mirror-maker/kafka-mirror-maker-2.yaml
When deploying multiple Kafka MirrorMaker 2 clusters managed by the Cluster Operator, deploy each cluster into a separate namespace. Deploying multiple clusters in the same namespace can lead to naming conflicts and resource collisions.
Prerequisites
- Cluster Operator is deployed.
Kafka cluster is running.
This procedure assumes that the Kafka cluster was deployed using Streams for Apache Kafka.
Procedure
Edit the deployment file to configure connection details (if required).
In
examples/mirror-maker/kafka-mirror-maker-2.yaml, add or update the following properties as needed:-
spec.clusters[].bootstrapServersto specify the Kafka bootstrap address for the source and target clusters. -
spec.clusters[].aliasto specify a unique identifier for each cluster. -
spec.clusters[].authenticationto specify the authentication type for each cluster astls,scram-sha-256,scram-sha-512,plain, oroauth.
See theKafkaMirrorMaker2Specschema properties for configuration details. -
spec.clusters[].tls.trustedCertificatesto configure the TLS certificate for each cluster.
Use[](an empty array) to trust the default Java CAs, or specify secrets containing trusted certificates.
See thetrustedCertificatesproperties for configuration details.
-
Configure the deployment for multiple MirrorMaker 2 clusters (if required).
If you plan to run more than one MirrorMaker 2 cluster in the same environment, each instance must use unique internal topic names and a unique group ID.
Update the
spec.clusters[].configproperties inkafka-mirror-maker-2.yamlto replace the default values.See Configuring multiple MirrorMaker 2 clusters for details.
Deploy Kafka MirrorMaker to your OpenShift cluster:
oc apply -f examples/mirror-maker/kafka-mirror-maker-2.yaml
oc apply -f examples/mirror-maker/kafka-mirror-maker-2.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY STATUS RESTARTS my-mm2-cluster-mirrormaker2-<pod_id> 1/1 Running 1
NAME READY STATUS RESTARTS my-mm2-cluster-mirrormaker2-<pod_id> 1/1 Running 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
my-mm2-clusteris the name of the Kafka MirrorMaker 2 cluster. A pod ID identifies each created pod. By default, the deployment creates a single MirrorMaker 2 pod.READYshows the number of ready versus expected replicas. The deployment is successful when theSTATUSisRunning.
8.5.2. List of Kafka MirrorMaker 2 cluster resources
The following resources are created by the Cluster Operator in the OpenShift cluster:
- <mirrormaker2_cluster_name>-mirrormaker2
Name given to the following MirrorMaker 2 resources:
- StrimziPodSet that creates the MirrorMaker 2 worker node pods.
- Headless service that provides stable DNS names to the MirrorMaker 2 pods.
- Service account used by the MirrorMaker 2 pods.
- Pod disruption budget configured for the MirrorMaker 2 worker nodes.
- Network Policy managing access to the MirrorMaker 2 REST API.
- <mirrormaker2_cluster_name>-mirrormaker2-<pod_id>
- Pods created by the MirrorMaker 2 StrimziPodSet.
- <mirrormaker2_cluster_name>-mirrormaker2-api
- Service which exposes the REST interface for managing the MirrorMaker 2 cluster.
- <mirrormaker2_cluster_name>-mirrormaker2-config
- ConfigMap which contains the MirrorMaker 2 ancillary configuration and is mounted as a volume by the MirrorMaker 2 pods.
- strimzi-<namespace-name>-<mirrormaker2_cluster_name>-mirrormaker2-init
- Cluster role binding used by the MirrorMaker 2 cluster.
8.6. Deploying Kafka Bridge
Kafka Bridge provides an API for integrating HTTP-based clients with a Kafka cluster.
8.6.1. Deploying Kafka Bridge
This procedure shows how to deploy a Kafka Bridge cluster to your OpenShift cluster using the Cluster Operator.
The deployment uses a YAML file to provide the specification to create a KafkaBridge resource.
Streams for Apache Kafka provides example configuration files. In this procedure, we use the following example file:
-
examples/bridge/kafka-bridge.yaml
Prerequisites
- Cluster Operator is deployed.
Kafka cluster is running.
This procedure assumes that the Kafka cluster was deployed using Streams for Apache Kafka.
Procedure
Edit the deployment file to configure connection details (if required).
In
examples/bridge/kafka-bridge.yaml, add or update the following properties as needed:-
spec.bootstrapServersto specify the Kafka bootstrap address. -
spec.authenticationto specify the authentication type astls,scram-sha-256,scram-sha-512,plain, oroauth.
See theKafkaBridgeSpecschema properties for configuration details. -
spec.tls.trustedCertificatesto configure the TLS certificate.
Use[](an empty array) to trust the default Java CAs, or specify secrets containing trusted certificates.
See thetrustedCertificatesproperties for configuration details.
-
Deploy Kafka Bridge to your OpenShift cluster:
oc apply -f examples/bridge/kafka-bridge.yaml
oc apply -f examples/bridge/kafka-bridge.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY STATUS RESTARTS my-bridge-bridge-<pod_id> 1/1 Running 0
NAME READY STATUS RESTARTS my-bridge-bridge-<pod_id> 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
my-bridgeis the name of the Kafka Bridge cluster. A pod ID identifies each created pod. By default, the deployment creates a single Kafka Bridge pod.READYshows the number of ready versus expected replicas. The deployment is successful when theSTATUSisRunning.
Use port forwarding to expose the Kafka Bridge service to your local machine on http://localhost:8080.
Port forwarding is only suitable for development and testing purposes.
Procedure
List the names of the pods in your OpenShift cluster:
oc get pods -o name pod/kafka-consumer # ... pod/my-bridge-bridge-<pod_id>
oc get pods -o name pod/kafka-consumer # ... pod/my-bridge-bridge-<pod_id>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Connect to the Kafka Bridge pod on port
8080:oc port-forward pod/my-bridge-bridge-<pod_id> 8080:8080 &
oc port-forward pod/my-bridge-bridge-<pod_id> 8080:8080 &Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf port 8080 on your local machine is already in use, use an alternative HTTP port, such as
8008.
API requests are now forwarded from port 8080 on your local machine to port 8080 in the Kafka Bridge pod.
8.6.3. Accessing the Kafka Bridge outside of OpenShift
After deployment, the Kafka Bridge can only be accessed by applications running in the same OpenShift cluster. These applications use the <kafka_bridge_name>-bridge-service service to access the API.
If you want to make the Kafka Bridge accessible to applications running outside of the OpenShift cluster, you can expose it manually by creating one of the following features:
-
LoadBalancerorNodePorttype services -
Ingressresources (Kubernetes only) - OpenShift routes (OpenShift only)
If you decide to create Services, use the labels from the selector of the <kafka_bridge_name>-bridge-service service to configure the pods to which the service will route the traffic:
# ...
selector:
strimzi.io/cluster: kafka-bridge-name
strimzi.io/kind: KafkaBridge
#...
# ...
selector:
strimzi.io/cluster: kafka-bridge-name
strimzi.io/kind: KafkaBridge
#...- 1
- Name of the Kafka Bridge custom resource in your OpenShift cluster.
8.6.4. List of Kafka Bridge cluster resources
The following resources are created by the Cluster Operator in the OpenShift cluster:
- <bridge_cluster_name>-bridge
- Deployment which is in charge to create the Kafka Bridge worker node pods.
- <bridge_cluster_name>-bridge-service
- Service which exposes the REST interface of the Kafka Bridge cluster.
- <bridge_cluster_name>-bridge-config
- ConfigMap which contains the Kafka Bridge ancillary configuration and is mounted as a volume by the Kafka broker pods.
- <bridge_cluster_name>-bridge
- Pod Disruption Budget configured for the Kafka Bridge worker nodes.
You can perform a standalone deployment of the Topic Operator and User Operator. Consider a standalone deployment of these operators if you are using a Kafka cluster that is not managed by the Cluster Operator.
You deploy the operators to OpenShift, Kafka can be running outside of OpenShift. For example, you might be using a managed Kafka service.
To connect the standalone operators to your Kafka cluster, you must set environment variables that specify the cluster address and authentication details. These variables are automatically configured when deploying the operators with the Cluster Operator.
8.7.1. Deploying the standalone Topic Operator
This procedure shows how to deploy the Topic Operator as a standalone component for topic management. You can use a standalone Topic Operator with a Kafka cluster that is not managed by the Cluster Operator.
Standalone deployment files are provided with Streams for Apache Kafka. Use the 05-Deployment-strimzi-topic-operator.yaml deployment file to deploy the Topic Operator. Add or set the environment variables needed to make a connection to a Kafka cluster.
The Topic Operator watches for KafkaTopic resources in a single namespace. You specify the namespace to watch, and the connection to the Kafka cluster, in the Topic Operator configuration. A single Topic Operator can watch a single namespace. One namespace should be watched by only one Topic Operator. If you want to use more than one Topic Operator, configure each of them to watch different namespaces. In this way, you can use Topic Operators with multiple Kafka clusters.
Do not deploy more than one Kafka cluster into the same namespace. This causes the Topic Operator connected to each cluster to compete for the same topic resources using the same names, leading to name collisions and unpredictable behavior.
Prerequisites
- The standalone Topic Operator deployment files, which are included in the Streams for Apache Kafka deployment files.
You are running a Kafka cluster for the Topic Operator to connect to.
As long as the standalone Topic Operator is correctly configured for connection, the Kafka cluster can be running on a bare-metal environment, a virtual machine, or as a managed cloud application service.
Procedure
Edit the
envproperties in theinstall/topic-operator/05-Deployment-strimzi-topic-operator.yamlstandalone deployment file.Example standalone Topic Operator deployment configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The OpenShift namespace for the Topic Operator to watch for
KafkaTopicresources. Specify the namespace of the Kafka cluster. - 2
- The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.
- 3
- The label to identify the
KafkaTopicresources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to theKafkaTopicresource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources. - 4
- The interval between periodic reconciliations, in milliseconds. The default is
120000(2 minutes). - 5
- The level for printing logging messages. You can set the level to
ERROR,WARNING,INFO,DEBUG, orTRACE. - 6
- Enables TLS support for encrypted communication with the Kafka brokers.
- 7
- (Optional) The Java options used by the JVM running the Topic Operator.
- 8
- (Optional) The debugging (
-D) options set for the Topic Operator. - 9
- (Optional) Skips the generation of trust store certificates if TLS is enabled through
STRIMZI_TLS_ENABLED. If this environment variable is enabled, the brokers must use a public trusted certificate authority for their TLS certificates. The default isfalse. - 10
- (Optional) Generates key store certificates for mTLS authentication. Setting this to
falsedisables client authentication with mTLS to the Kafka brokers. The default istrue. - 11
- (Optional) Enables SASL support for client authentication when connecting to Kafka brokers. The default is
false. - 12
- (Optional) The SASL username for client authentication. Mandatory only if SASL is enabled through
STRIMZI_SASL_ENABLED. - 13
- (Optional) The SASL password for client authentication. Mandatory only if SASL is enabled through
STRIMZI_SASL_ENABLED. - 14
- (Optional) The SASL mechanism for client authentication. Mandatory only if SASL is enabled through
STRIMZI_SASL_ENABLED. You can set the value toplain,scram-sha-256, orscram-sha-512. - 15
- (Optional) The security protocol used for communication with Kafka brokers. The default value is "PLAINTEXT". You can set the value to
PLAINTEXT,SSL,SASL_PLAINTEXT, orSASL_SSL. - 16
-
If you want to connect to Kafka brokers that are using certificates from a public certificate authority, set
STRIMZI_PUBLIC_CAtotrue. Set this property totrue, for example, if you are using Amazon AWS MSK service. If you enabled mTLS with the
STRIMZI_TLS_ENABLEDenvironment variable, specify the keystore and truststore used to authenticate connection to the Kafka cluster.Example mTLS configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you need to configure custom SASL authentication, you can define the necessary authentication properties using the
STRIMZI_SASL_CUSTOM_CONFIG_JSONenvironment variable for the standalone operator. For example, this configuration may be used for accessing a Kafka cluster in a cloud provider with a custom login module like the Amazon MSK Library for AWS Identity and Access Management (aws-msk_iam-auth).The property
STRIMZI_ALTERABLE_TOPIC_CONFIGdefaults toALL, allowing all.spec.configproperties to be set in theKafkaTopicresource. If this setting is not suitable for a managed Kafka service, do as follows:- If only a subset of properties is configurable, list them as comma-separated values.
-
If no properties are to be configured, use
NONE, which is equivalent to an empty property list.
NoteOnly Kafka configuration properties starting with
sasl.can be set with theSTRIMZI_SASL_CUSTOM_CONFIG_JSONenvironment variable.Example custom SASL configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Disables cluster configuration lookup for managed Kafka services that don’t allow topic configuration changes.
- 2
- Defines the topic configuration properties that can be updated based on the limitations set by managed Kafka services.
- 3
- Specifies the SASL properties to be set in JSON format. Only properties starting with
sasl.are allowed.
Example Dockerfile with external jars
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Deploymentconfiguration to deploy the Topic Operator. Check the status of the deployment:
oc get deployments
oc get deploymentsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-topic-operator 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE strimzi-topic-operator 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
8.7.2. Deploying the standalone User Operator
This procedure shows how to deploy the User Operator as a standalone component for user management. You can use a standalone User Operator with a Kafka cluster that is not managed by the Cluster Operator.
A standalone deployment can operate with any Kafka cluster.
Standalone deployment files are provided with Streams for Apache Kafka. Use the 05-Deployment-strimzi-user-operator.yaml deployment file to deploy the User Operator. Add or set the environment variables needed to make a connection to a Kafka cluster.
The User Operator watches for KafkaUser resources in a single namespace. You specify the namespace to watch, and the connection to the Kafka cluster, in the User Operator configuration. A single User Operator can watch a single namespace. One namespace should be watched by only one User Operator. If you want to use more than one User Operator, configure each of them to watch different namespaces. In this way, you can use the User Operator with multiple Kafka clusters.
Do not configure standalone User Operators that connect to different Kafka clusters to watch the same namespace. This can cause the operators to compete for the same KafkaUser resources, leading to name collisions and unpredictable behavior.
Prerequisites
- The standalone User Operator deployment files, which are included in the Streams for Apache Kafka deployment files.
You are running a Kafka cluster for the User Operator to connect to.
As long as the standalone User Operator is correctly configured for connection, the Kafka cluster can be running on a bare-metal environment, a virtual machine, or as a managed cloud application service.
Procedure
Edit the following
envproperties in theinstall/user-operator/05-Deployment-strimzi-user-operator.yamlstandalone deployment file.Example standalone User Operator deployment configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The OpenShift namespace for the User Operator to watch for
KafkaUserresources. Only one namespace can be specified. - 2
- The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.
- 3
- The OpenShift
Secretthat contains the public key (ca.crt) value of the CA (certificate authority) that signs new user certificates for mTLS authentication. - 4
- The OpenShift
Secretthat contains the private key (ca.key) value of the CA that signs new user certificates for mTLS authentication. - 5
- The label to identify the
KafkaUserresources managed by the User Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to theKafkaUserresource. If you deploy more than one User Operator, the labels must be unique for each. That is, the operators cannot manage the same resources. - 6
- The interval between periodic reconciliations, in milliseconds. The default is
120000(2 minutes). - 7
- The size of the controller event queue. The size of the queue should be at least as big as the maximal amount of users you expect the User Operator to operate. The default is
1024. - 8
- The size of the worker pool for reconciling the users. Bigger pool might require more resources, but it will also handle more
KafkaUserresources The default is50. - 9
- The size of the worker pool for Kafka Admin API and OpenShift operations. Bigger pool might require more resources, but it will also handle more
KafkaUserresources The default is4. - 10
- The level for printing logging messages. You can set the level to
ERROR,WARNING,INFO,DEBUG, orTRACE. - 11
- Enables garbage collection (GC) logging. The default is
true. - 12
- The validity period for the CA. The default is
365days. - 13
- The renewal period for the CA. The renewal period is measured backwards from the expiry date of the current certificate. The default is
30days to initiate certificate renewal before the old certificates expire. - 14
- (Optional) The Java options used by the JVM running the User Operator
- 15
- (Optional) The debugging (
-D) options set for the User Operator - 16
- (Optional) Prefix for the names of OpenShift secrets created by the User Operator.
- 17
- (Optional) Indicates whether the Kafka cluster supports management of authorization ACL rules using the Kafka Admin API. When set to
false, the User Operator will reject all resources withsimpleauthorization ACL rules. This helps to avoid unnecessary exceptions in the Kafka cluster logs. The default istrue. - 18
- (Optional) Semi-colon separated list of Cron Expressions defining the maintenance time windows during which the expiring user certificates will be renewed.
- 19
- (Optional) Configuration options for configuring the Kafka Admin client used by the User Operator in the properties format.
- 20
- (Optional) Regular expression to configure users for which any existing ACLs, Quotas and SCRAM-SHa credentials will be ignored. This option is useful when you want to configure the User Operator to ignore users that are managed through another mechanism.
If you are using mTLS to connect to the Kafka cluster, specify the secrets used to authenticate connection. Otherwise, go to the next step.
Example mTLS configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The OpenShift
Secretthat contains the public key (ca.crt) value of the CA that signs Kafka broker certificates. - 2
- The OpenShift
Secretthat contains the certificate public key (entity-operator.crt) and private key (entity-operator.key) that is used for mTLS authentication against the Kafka cluster. - 3
- The name of the private key that is used for mTLS authentication against the Kafka cluster. The default is
entity-operator.key. - 4
- The name of the public key that is used for mTLS authentication against the Kafka cluster. The default is
entity-operator.crt.
Deploy the User Operator.
oc create -f install/user-operator
oc create -f install/user-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments
oc get deploymentsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-user-operator 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE strimzi-user-operator 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
Chapter 9. Feature gates
Streams for Apache Kafka operators use feature gates to enable or disable specific features and functions. Enabling a feature gate alters the behavior of the associated operator, introducing the corresponding feature to your Streams for Apache Kafka deployment.
The purpose of feature gates is to facilitate the trial and testing of a feature before it is fully adopted. The state (enabled or disabled) of a feature gate may vary by default, depending on its maturity level.
As a feature gate graduates and reaches General Availability (GA), it transitions to an enabled state by default and becomes a permanent part of the Streams for Apache Kafka deployment. A feature gate at the GA stage cannot be disabled.
The supported feature gates are applicable to all Streams for Apache Kafka operators. While a particular feature gate might be used by one operator and ignored by the others, it can still be configured in all operators. When deploying the User Operator and Topic Operator within the context of the Kafka custom resource, the Cluster Operator automatically propagates the feature gates configuration to them. When the User Operator and Topic Operator are deployed standalone, without a Cluster Operator available to configure the feature gates, they must be directly configured within their deployments.
9.1. Feature gate releases
Feature gates have three stages of maturity:
- Alpha
-
Alpha stage features are disabled by default.
They might be experimental or unstable, subject to change, or not sufficiently tested for production use. - Beta
-
Beta stage features are enabled by default, but they can be disabled if needed.
They are well tested and their functionality is not likely to change. - General Availability (GA)
-
GA features are always enabled and cannot be disabled.
They are stable and should not change in the future. Feature gates might be removed when they reach GA, which means that the feature was incorporated into the Streams for Apache Kafka core features.
Alpha and beta stage features are removed if they do not prove to be useful. When a feature gate reaches GA, it is permanently enabled and can no longer be disabled. The following table shows the maturity of the feature gates introduced across Streams for Apache Kafka versions.
| Feature gate | Alpha | Beta | GA |
|---|---|---|---|
|
| 1.8 | 2.0 | 2.3 |
|
| 1.8 | 2.0 | 2.3 |
|
| 2.1 | 2.3 | 2.5 |
|
| 2.2 | 2.7 | 2.8 |
|
| 2.4 | 2.6 | 2.7 |
|
| 2.5 | 2.7 | 2.8 |
|
| 2.5 | 2.7 | 2.8 |
|
| 2.8 | 2.9 | 3.0 |
|
| 3.1 | - | - |
9.2. Graduated feature gates (GA)
For information on feature gates that have reached GA, see the documentation for the Streams for Apache Kafka version in which they were introduced.
9.3. Stable feature gates (Beta)
Stable feature gates have reached a beta level of maturity, and are generally enabled by default for all users. Stable feature gates are production-ready, but they can still be disabled. Currently, there are no beta level feature gates.
9.4. Early access feature gates (Alpha)
Early access feature gates have not yet reached the beta stage, and are disabled by default. An early access feature gate provides an opportunity for assessment before its functionality is permanently incorporated into Streams for Apache Kafka. Currently, there are no alpha level feature gates.
9.4.1. ServerSideApplyPhase1 feature gate
The ServerSideApplyPhase1 feature gate has a default state of disabled.
When enabled, the Cluster Operator uses Server-Side Apply (SSA) for creating and updating certain resources. SSA is an OpenShift feature that provides better tracking of field ownership and more controlled resource updates. In Phase 1, Server-Side Apply is used for the following resources:
-
PersistentVolumeClaim -
ConfigMap -
Ingress -
ServiceAccount -
Service
All other resources continue to be reconciled using the existing mechanism - client-side apply. If users change the fields of operator-managed resources, the operator reverts these changes.
With SSA, the Cluster Operator modifies only the fields it manages. User changes to the other fields are preserved as long as they don’t conflict. On the first update attempt, the Cluster Operator sets force = false when applying changes, respecting user changes. If a conflict occurs in a managed field, the update is retried with force = true. The conflict and the switch to forced apply are logged by the Cluster Operator.
Ownership of individual fields is tracked through the .metadata.managedFields property of the resource being updated.
To enable the ServerSideApplyPhase1 feature gate, specify +ServerSideApplyPhase1 in the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.
9.5. Enabling feature gates
To modify a feature gate’s default state, use the STRIMZI_FEATURE_GATES environment variable in the operator’s configuration. You can modify multiple feature gates using this single environment variable. Specify a comma-separated list of feature gate names and prefixes. A + prefix enables the feature gate and a - prefix disables it.
Example feature gate configuration that enables FeatureGate1 and disables FeatureGate2
env:
- name: STRIMZI_FEATURE_GATES
value: +FeatureGate1,-FeatureGate2
env:
- name: STRIMZI_FEATURE_GATES
value: +FeatureGate1,-FeatureGate2Chapter 10. Configuring a deployment
Configure and manage a Streams for Apache Kafka deployment to your precise needs using Streams for Apache Kafka custom resources. Streams for Apache Kafka provides example custom resources with each release, allowing you to configure and create instances of supported Kafka components.
Use custom resources to configure and create instances of the following components:
- Kafka clusters
- Kafka Connect clusters
- Kafka MirrorMaker
- Kafka Bridge
- Cruise Control
You can use configuration to manage your instances or modify your deployment to introduce additional features. New features are sometimes introduced through feature gates, which are controlled through operator configuration.
The Streams for Apache Kafka Custom Resource API Reference describes the properties you can use in your configuration.
Centralizing configuration
For key configuration areas, such as metrics, logging, and external Kafka Connect connector settings, you can centralize management as follows:
We recommend configuration providers for securely supplying Kafka Connect connector credentials.
TLS certificate management
When deploying Kafka, the Cluster Operator automatically sets up and renews TLS certificates to enable encryption and authentication within your cluster. If required, you can manually renew the cluster and clients CA certificates before their renewal period starts. You can also replace the keys used by the cluster and clients CA certificates.
For more information, see Renewing CA certificates manually and Replacing private keys.
Labels applied to a custom resource are also applied to the OpenShift resources making up its cluster. This provides a convenient mechanism for resources to be labeled as required.
10.1. Using example configuration files
Further enhance your deployment by incorporating additional supported configuration. Example configuration files are included in the Streams for Apache Kafka deployment files.
The example files include only the essential properties and values for custom resources by default. You can download and apply the examples using the oc command-line tool. The examples can serve as a starting point when building your own Kafka component configuration for deployment.
If you installed Streams for Apache Kafka using the Operator, you can still download the example files and use them to upload configuration.
The release artifacts include an examples directory that contains the configuration examples.
Example configuration files provided with Streams for Apache Kafka
- 1
KafkaUsercustom resource configuration, which is managed by the User Operator.- 2
KafkaTopiccustom resource configuration, which is managed by Topic Operator.- 3
- Authentication and authorization configuration for Kafka components. Includes example configuration for TLS and SCRAM-SHA-512 authentication. The Red Hat build of Keycloak example includes
Kafkacustom resource configuration and a Red Hat build of Keycloak realm specification. You can use the example to try Red Hat build of Keycloak authorization services. There is also an example with enabledoauthauthentication andkeycloakauthorization metrics. - 4
KafkaMirrorMaker2custom resource configurations for a deployment of MirrorMaker 2. Includes example configuration for replication policy and synchronization frequency.- 5
- Metrics configuration, including Prometheus installation and Grafana dashboard files.
- 6
KafkaandKafkaNodePoolcustom resource configurations for a deployment of Kafka clusters that use KRaft mode. Includes example configuration for an ephemeral or persistent single or multi-node deployment.- 7
KafkaandKafkaRebalanceconfigurations for deploying and using Cruise Control to manage clusters.Kafkaconfiguration examples enable auto-rebalancing on scaling events and set default optimization goals.KakaRebalanceconfiguration examples set proposal-specific optimization goals and generate optimization proposals in various supported modes.- 8
KafkaConnectandKafkaConnectorcustom resource configuration for a deployment of Kafka Connect. Includes example configurations for a single or multi-node deployment.- 9
KafkaBridgecustom resource configuration for a deployment of Kafka Bridge.
10.2. Configuring Kafka
Configure your Kafka deployment by updating the spec properties of the Kafka custom resource.
You can also configure Streams for Apache Kafka components and features that support the deployment:
- Topic Operator and User Operator for managing topics and clients
- Cruise Control for cluster rebalancing
- Kafka Exporter for lag metrics
- Listeners for authenticated client access
- Data storage
- Rack awareness
Specify the metadata version for KRaft using .spec.kafka.metadataVersion. It must be compatible with .spec.kafka.version. If not set, the Cluster Operator applies the default for the Kafka version.
Kafka clusters use node pools. Use KafkaNodePool resources to configure distinct groups of nodes within a Kafka cluster.
For a deeper understanding of the Kafka cluster configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.
The oldest supported metadata version is 3.3. Older versions may disable certain features.
Example Kafka custom resource configuration
- 1
- Listeners configure how clients connect to the Kafka cluster via bootstrap addresses. Listeners are configured as internal or external listeners for connection from inside or outside the OpenShift cluster.
- 2
- Name to identify the listener. Must be unique within the Kafka cluster.
- 3
- Port number used by the listener inside Kafka. The port number has to be unique within a given Kafka cluster. Allowed port numbers are 9092 and higher with the exception of ports 9404 and 9999, which are already used for Prometheus and JMX. Depending on the listener type, the port number might not be the same as the port number that connects Kafka clients.
- 4
- Listener type specified as
internalorcluster-ip(to expose Kafka using per-brokerClusterIPservices), or for external listeners, asroute(OpenShift only),loadbalancer,nodeportoringress(Kubernetes only). - 5
- Enables or disables TLS encryption for each listener. For
routeandingresstype listeners, TLS encryption must always be enabled by setting it totrue. - 6
- Defines whether the fully-qualified DNS names including the cluster service suffix (usually
.cluster.local) are assigned. - 7
- Listener authentication mechanism specified as mTLS, SCRAM-SHA-512, or token-based OAuth 2.0.
- 8
- External listener configuration specifies how the Kafka cluster is exposed outside OpenShift, such as through a
route,loadbalancerornodeport. - 9
- Optional configuration for a Kafka listener certificate managed by an external CA (certificate authority). The
brokerCertChainAndKeyspecifies aSecretthat contains a server certificate and a private key. You can configure Kafka listener certificates on any listener with enabled TLS encryption. - 10
- Kafka version, which can be changed to a supported version by following the upgrade procedure.
- 11
- Kafka metadata version, which can be changed to a supported version by following the upgrade procedure.
- 12
- Broker configuration. Standard Apache Kafka configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka.
- 13
- Requests for reservation of supported resources, currently
cpuandmemory, and limits to specify the maximum resources that can be consumed. - 14
- Kafka loggers and log levels added directly (
inline) or indirectly (external) through aConfigMap. Custom Log4j configuration must be placed under thelog4j2.propertieskey in theConfigMap. You can set log levels toINFO,ERROR,WARN,TRACE,DEBUG,FATALorOFF. - 15
- Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).
- 16
- JVM configuration options to optimize performance for the Virtual Machine (VM) running Kafka.
- 17
- ADVANCED OPTION: Container image configuration, which is recommended only in special situations.
- 18
- Authorization enables simple, OAuth 2.0, custom, or OPA (deprecated) authorization on the Kafka broker. Simple authorization uses the
StandardAuthorizerKafka plugin. - 19
- Rack awareness configuration to spread replicas across different racks, data centers, or availability zones. The
topologyKeymust match a node label containing the rack ID. The example used in this configuration specifies a zone using the standardtopology.kubernetes.io/zonelabel. - 20
- Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter (the default metrics exporter).
- 21
- Rules for exporting metrics in Prometheus format to a Grafana dashboard through the Prometheus JMX Exporter, which are enabled by referencing a ConfigMap containing configuration for the Prometheus JMX exporter. You can enable metrics without further configuration using a reference to a ConfigMap containing an empty file under
metricsConfig.valueFrom.configMapKeyRef.key. - 22
- Entity Operator configuration, which specifies the configuration for the Topic Operator and User Operator.
- 23
- Specified Topic Operator loggers and log levels. This example uses
inlinelogging. - 24
- Specified User Operator loggers and log levels.
- 25
- Kafka Exporter configuration. Kafka Exporter is an optional component for extracting metrics data from Kafka brokers, in particular consumer lag data. For Kafka Exporter to be able to work properly, consumer groups need to be in use.
- 26
- Optional configuration for Cruise Control, which is used to rebalance the Kafka cluster.
If you remove the min.insync.replicas property from .spec.kafka.config in the Kafka resource, the Cluster Operator forces Kafka to fall back to the default value (1), regardless of whether ELR (Eligible Leader Replicas) is enabled or disabled. To ensure durability of the cluster, explicitly define min.insync.replicas with a value higher than 1.
10.2.1. Setting throughput and storage limits on brokers
This procedure describes how to set throughput and storage limits on brokers in your Kafka cluster. Enable a quota plugin and configure limits using quotas properties in the Kafka resource.
There are two types of quota plugins available:
-
The
strimzitype enables the Strimzi Quotas plugin. -
The
kafkatype enables the built-in Kafka plugin.
Only one quota plugin can be enabled at a time. The built-in kafka plugin is enabled by default. Enabling the strimzi plugin automatically disables the built-in plugin.
strimzi plugin
The strimzi plugin provides storage utilization quotas and dynamic distribution of throughput limits.
-
Storage quotas throttle Kafka producers based on disk storage utilization. Limits can be specified in bytes (
minAvailableBytesPerVolume) or percentage (minAvailableRatioPerVolume) of available disk space, applying to each disk individually. When any broker in the cluster exceeds the configured disk threshold, clients are throttled to prevent disks from filling up too quickly and exceeding capacity. - A total throughput limit is distributed dynamically across all clients. For example, if you set a 40 MBps producer byte-rate threshold, the distribution across two producers is not static. If one producer is using 10 MBps, the other can use up to 30 MBps.
- Specific users (clients) can be excluded from the restrictions.
With the strimzi plugin, you see only aggregated quota metrics, not per-client metrics.
kafka plugin
The kafka plugin applies throughput limits on a per-user, per-broker basis and includes additional CPU and operation rate limits.
-
Limits are applied per user and per broker. For example, setting a 20 MBps producer byte-rate threshold limits each user to 20 MBps on a per-broker basis across all producer connections for that user. There is no total throughput limit as there is in the
strimziplugin. Limits can be overridden by user-specific quota configurations. - CPU utilization limits for each client can be set as a percentage of the network threads and I/O threads on a per-broker basis.
- The number of concurrent partition creation and deletion operations (mutations) allowed per second can be set on a per-broker basis.
When using the default Kafka quotas plugin, the default quotas (if set) are applied to all users. This includes internal users such as the Topic Operator and Cruise Control, which may impact their operations. To avoid unduly limiting internal users, consider tuning the quotas effectively.
For example, a quota automatically applied to the Topic Operator by the Kafka quotas plugin could constrain the controller mutation rate, potentially throttling topic creation or deletion operations. Therefore, it is important to understand the minimal quotas required by the Topic Operator to function correctly and explicitly set appropriate quotas to avoid such issues. Monitoring relevant controller and broker metrics can help track and optimize the rate of operations on topics. Cruise Control and its metrics reporter also require sufficient produce and fetch rates to conduct rebalances, depending on the scale and configuration of the Kafka cluster. To prevent issues for Cruise Control, you might start with a rate of at least 1 KB/s for its producers and consumers in small clusters, such as three brokers with moderate traffic, and adjust as needed for larger or more active clusters.
Prerequisites
- The Cluster Operator that manages the Kafka cluster is running.
Procedure
Add the plugin configuration to the
quotassection of theKafkaresource.Example
strimziplugin configurationCopy to Clipboard Copied! Toggle word wrap Toggle overflow minAvailableBytesPerVolumeandminAvailableRatioPerVolumeare mutually exclusive. Only configure one of these parameters.Example
kafkaplugin configurationCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Kafkaconfiguration.
Additional options can be configured in the spec.kafka.config section. The full list of supported options can be found in the plugin documentation.
10.2.2. Deleting Kafka nodes using annotations
This procedure describes how to delete an existing Kafka node by using an OpenShift annotation. Deleting a Kafka node consists of deleting both the Pod on which the Kafka broker is running and the related PersistentVolumeClaim (if the cluster was deployed with persistent storage). After deletion, the Pod and its related PersistentVolumeClaim are recreated automatically.
Deleting a PersistentVolumeClaim can cause permanent data loss and the availability of your cluster cannot be guaranteed. The following procedure should only be performed if you have encountered storage issues.
Prerequisites
- A running Cluster Operator
Procedure
Find the name of the
Podthat you want to delete.Kafka broker pods are named
<cluster_name>-kafka-<index_number>, where<index_number>starts at zero and ends at the total number of replicas minus one. For example,my-cluster-kafka-0.Use
oc annotateto annotate thePodresource in OpenShift:oc annotate pod <cluster_name>-kafka-<index_number> strimzi.io/delete-pod-and-pvc="true"
oc annotate pod <cluster_name>-kafka-<index_number> strimzi.io/delete-pod-and-pvc="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Wait for the next reconciliation, when the annotated pod with the underlying persistent volume claim will be deleted and then recreated.
10.3. Configuring node pools
Update the spec properties of the KafkaNodePool custom resource to configure a node pool deployment. A node pool is a distinct group of Kafka nodes within a Kafka cluster. The strimzi.io/cluster metadata label identifies the name of the Kafka custom resource the pool belongs to.
Node pool configuration must define:
- Node roles within the cluster
- Number of replica nodes
- Storage specifications
The .spec.roles property defines whether the nodes in the pool act as controllers, brokers, or both.
Other optional properties may also be set in node pools:
-
resourcesto specify memory and cpu requests and limits -
templateto specify custom configuration for pods and other OpenShift resources -
jvmOptionsto specify custom JVM configuration for heap size, runtime and other options
The relationship between Kafka and KafkaNodePool resources is as follows:
-
Kafkaresources represent the configuration for all nodes in a Kafka cluster. -
KafkaNodePoolresources represent the configuration for nodes only in the node pool.
If a configuration property is not specified in KafkaNodePool, it is inherited from the Kafka resource. Configuration specified in the KafkaNodePool resource takes precedence if set in both resources. For example, if both the node pool and Kafka configuration includes jvmOptions, the values specified in the node pool configuration are used. When -Xmx: 1024m is set in KafkaNodePool.spec.jvmOptions and -Xms: 512m is set in Kafka.spec.kafka.jvmOptions, the node uses the value from its node pool configuration.
Properties from Kafka and KafkaNodePool schemas are not combined. To clarify, if KafkaNodePool.spec.template includes only podSet.metadata.labels, and Kafka.spec.kafka.template includes podSet.metadata.annotations and pod.metadata.labels, the template values from the Kafka configuration are ignored since there is a template value in the node pool configuration.
For a deeper understanding of the node pool configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.
Example configuration for a node pool in a cluster using KRaft mode
- 1
- Unique name for the node pool.
- 2
- The Kafka cluster the node pool belongs to. A node pool can only belong to a single cluster.
- 3
- Number of replicas for the nodes.
- 4
- Roles for the nodes in the node pool. In this example, the nodes have dual roles as controllers and brokers.
- 5
- Storage specification for the nodes.
- 6
- Requests for reservation of supported resources, currently
cpuandmemory, and limits to specify the maximum resources that can be consumed.
The configuration for the Kafka resource must be suitable for KRaft mode. Currently, KRaft mode has a number of limitations.
This procedure describes how to use annotations for advanced node ID handling by the Cluster Operator when performing scaling operations on node pools. You specify the node IDs to use, rather than the Cluster Operator using the next ID in sequence. Management of node IDs in this way gives greater control.
To add a range of IDs, you assign the following annotations to the KafkaNodePool resource:
-
strimzi.io/next-node-idsto add a range of IDs that are used for new brokers -
strimzi.io/remove-node-idsto add a range of IDs for removing existing brokers
You can specify an array of individual node IDs, ID ranges, or a combination of both. For example, you can specify the following range of IDs: [0, 1, 2, 10-20, 30] for scaling up the Kafka node pool. This format allows you to specify a combination of individual node IDs (0, 1, 2, 30) as well as a range of IDs (10-20).
In a typical scenario, you might specify a range of IDs for scaling up and a single node ID to remove a specific node when scaling down.
In this procedure, we add the scaling annotations to node pools as follows:
-
pool-ais assigned a range of IDs for scaling up -
pool-bis assigned a range of IDs for scaling down
During the scaling operation, IDs are used as follows:
- Scale up picks up the lowest available ID in the range for the new node.
- Scale down removes the node with the highest available ID in the range.
If there are gaps in the sequence of node IDs assigned in the node pool, the next node to be added is assigned an ID that fills the gap.
The annotations don’t need to be updated after every scaling operation. Any unused IDs are still valid for the next scaling event.
The Cluster Operator allows you to specify a range of IDs in either ascending or descending order, so you can define them in the order the nodes are scaled. For example, when scaling up, you can specify a range such as [1000-1999], and the new nodes are assigned the next lowest IDs: 1000, 1001, 1002, 1003, and so on. Conversely, when scaling down, you can specify a range like [1999-1000], ensuring that nodes with the next highest IDs are removed: 1003, 1002, 1001, 1000, and so on.
If you don’t specify an ID range using the annotations, the Cluster Operator follows its default behavior for handling IDs during scaling operations. Node IDs start at 0 (zero) and run sequentially across the Kafka cluster. The next lowest ID is assigned to a new node. Gaps to node IDs are filled across the cluster. This means that they might not run sequentially within a node pool. The default behavior for scaling up is to add the next lowest available node ID across the cluster; and for scaling down, it is to remove the node in the node pool with the highest available node ID. The default approach is also applied if the assigned range of IDs is misformatted, the scaling up range runs out of IDs, or the scaling down range does not apply to any in-use nodes.
Prerequisites
- The Cluster Operator must be deployed.
-
(Optional) Use the
reserved.broker-max.idconfiguration property to extend the allowable range for node IDs within your node pools.
By default, Apache Kafka restricts node IDs to numbers ranging from 0 to 999. To use node ID values greater than 999, add the reserved.broker-max.id configuration property to the Kafka custom resource and specify the required maximum node ID value.
In this example, the maximum node ID is set at 10000. Node IDs can then be assigned up to that value.
Example configuration for the maximum node ID number
Procedure
Annotate the node pool with the IDs to use when scaling up or scaling down, as shown in the following examples.
IDs for scaling up are assigned to node pool
pool-a:Assigning IDs for scaling up
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids="[0,1,2,10-20,30]"
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids="[0,1,2,10-20,30]"Copy to Clipboard Copied! Toggle word wrap Toggle overflow The lowest available ID from this range is used when adding a node to
pool-a.IDs for scaling down are assigned to node pool
pool-b:Assigning IDs for scaling down
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[60-50,9,8,7]"
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[60-50,9,8,7]"Copy to Clipboard Copied! Toggle word wrap Toggle overflow The highest available ID from this range is removed when scaling down
pool-b.NoteIf you want to remove a specific node, you can assign a single node ID to the scaling down annotation:
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[3]".You can now scale the node pool.
For more information, see the following:
On reconciliation, a warning is given if the annotations are misformatted.
After you have performed the scaling operation, you can remove the annotation if it’s no longer needed.
Removing the annotation for scaling up
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids-
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids-Copy to Clipboard Copied! Toggle word wrap Toggle overflow Removing the annotation for scaling down
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids-
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids-Copy to Clipboard Copied! Toggle word wrap Toggle overflow
If rack awareness is enabled on a Kafka cluster, replicas can be spread across different racks, data centers, or availability zones. When moving nodes from node pools, consider the implications on the cluster topology, particularly regarding rack awareness. Removing specific pods from node pools, especially out of order, may break the cluster topology or cause an imbalance in distribution across racks. An imbalance can impact both the distribution of nodes themselves and the partition replicas within the cluster. An uneven distribution of nodes and partitions across racks can affect the performance and resilience of the Kafka cluster.
Plan the removal of nodes strategically to maintain the required balance and resilience across racks. Use the strimzi.io/remove-node-ids annotation to move nodes with specific IDs with caution. Ensure that configuration to spread partition replicas across racks and for clients to consume from the closest replicas is not broken.
Use Cruise Control and the KafkaRebalance resource with the RackAwareGoal to make sure that replicas remain distributed across different racks.
10.3.3. Adding nodes to a node pool
This procedure describes how to scale up a node pool to add new nodes. Currently, scale up is only possible for broker-only node pools containing nodes that run as dedicated brokers.
In this procedure, we start with three nodes for node pool pool-a:
Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0
NAME READY STATUS RESTARTS
my-cluster-pool-a-0 1/1 Running 0
my-cluster-pool-a-1 1/1 Running 0
my-cluster-pool-a-2 1/1 Running 0
Node IDs are appended to the name of the node on creation. We add node my-cluster-pool-a-3, which has a node ID of 3.
During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
-
(Optional) Auto-rebalancing is enabled.
If auto-rebalancing is enabled, partition reassignment happens automatically during the node scaling process, so you don’t need to manually initiate the reassignment through Cruise Control. -
(Optional) For scale up operations, you can specify the node IDs to use in the operation.
If you have assigned a range of node IDs for the operation, the ID of the node being added is determined by the sequence of nodes given. If you have assigned a single node ID, a node is added with the specified ID. Otherwise, the lowest available node ID across the cluster is used.
Procedure
Create a new node in the node pool.
For example, node pool
pool-ahas three replicas. We add a node by increasing the number of replicas:oc scale kafkanodepool pool-a --replicas=4
oc scale kafkanodepool pool-a --replicas=4Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment and wait for the pods in the node pool to be created and ready (
1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows four Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0 my-cluster-pool-a-3 1/1 Running 0
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0 my-cluster-pool-a-3 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reassign the partitions after increasing the number of nodes in the node pool.
- If auto-rebalancing is enabled, partitions are reassigned to new nodes automatically, so you can skip this step.
If auto-rebalancing is not enabled, use the Cruise Control
add-brokersmode to move partition replicas from existing brokers to the newly added brokers.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow We are reassigning partitions to node
my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.
10.3.4. Removing nodes from a node pool
This procedure describes how to scale down a node pool to remove nodes. Currently, scale down is only possible for broker-only node pools containing nodes that run as dedicated brokers.
In this procedure, we start with four nodes for node pool pool-a:
Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0 my-cluster-pool-a-3 1/1 Running 0
NAME READY STATUS RESTARTS
my-cluster-pool-a-0 1/1 Running 0
my-cluster-pool-a-1 1/1 Running 0
my-cluster-pool-a-2 1/1 Running 0
my-cluster-pool-a-3 1/1 Running 0
Node IDs are appended to the name of the node on creation. We remove node my-cluster-pool-a-3, which has a node ID of 3.
During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
-
(Optional) Auto-rebalancing is enabled.
If auto-rebalancing is enabled, partition reassignment happens automatically during the node scaling process, so you don’t need to manually initiate the reassignment through Cruise Control. -
(Optional) For scale down operations, you can specify the node IDs to use in the operation.
If you have assigned a range of node IDs for the operation, the ID of the node being removed is determined by the sequence of nodes given. If you have assigned a single node ID, the node with the specified ID is removed. Otherwise, the node with the highest available ID in the node pool is removed.
Procedure
Reassign the partitions before decreasing the number of nodes in the node pool.
- If auto-rebalancing is enabled, partitions are moved off brokers that are going to be removed automatically, so you can skip this step.
If auto-rebalancing is not enabled, use the Cruise Control
remove-brokersmode to move partition replicas off the brokers that are going to be removed.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow We are reassigning partitions from node
my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.
After the reassignment process is complete, and the node being removed has no live partitions, reduce the number of Kafka nodes in the node pool.
For example, node pool
pool-ahas four replicas. We remove a node by decreasing the number of replicas:oc scale kafkanodepool pool-a --replicas=3
oc scale kafkanodepool pool-a --replicas=3Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows three Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-b-kafka-0 1/1 Running 0 my-cluster-pool-b-kafka-1 1/1 Running 0 my-cluster-pool-b-kafka-2 1/1 Running 0
NAME READY STATUS RESTARTS my-cluster-pool-b-kafka-0 1/1 Running 0 my-cluster-pool-b-kafka-1 1/1 Running 0 my-cluster-pool-b-kafka-2 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.3.5. Moving nodes between node pools
This procedure describes how to move nodes between source and target Kafka node pools without downtime. You create a new node on the target node pool and reassign partitions to move data from the old node on the source node pool. When the replicas on the new node are in-sync, you can delete the old node.
In this procedure, we start with two node pools:
-
pool-awith three replicas is the target node pool -
pool-bwith four replicas is the source node pool
We scale up pool-a, and reassign partitions and scale down pool-b, which results in the following:
-
pool-awith four replicas -
pool-bwith three replicas
Currently, scaling is only possible for broker-only node pools containing nodes that run as dedicated brokers.
During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
-
(Optional) Auto-rebalancing is enabled.
If auto-rebalancing is enabled, partition reassignment happens automatically during the node scaling process, so you don’t need to manually initiate the reassignment through Cruise Control. -
(Optional) For scale up and scale down operations, you can specify the range of node IDs to use.
If you have assigned node IDs for the operation, the ID of the node being added or removed is determined by the sequence of nodes given. Otherwise, the lowest available node ID across the cluster is used when adding nodes; and the node with the highest available ID in the node pool is removed.
Procedure
Create a new node in the target node pool.
For example, node pool
pool-ahas three replicas. We add a node by increasing the number of replicas:oc scale kafkanodepool pool-a --replicas=4
oc scale kafkanodepool pool-a --replicas=4Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment and wait for the pods in the node pool to be created and ready (
1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows four Kafka nodes in the source and target node pools
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Node IDs are appended to the name of the node on creation. We add node
my-cluster-pool-a-7, which has a node ID of7.If auto-rebalancing is enabled, partitions are reassigned to new nodes and moved off brokers that are going to be removed automatically, so you can skip the next step.
If auto-rebalancing is not enabled, reassign partitions before decreasing the number of nodes in the source node pool.
Use the Cruise Control
remove-brokersmode to move partition replicas off the brokers that are going to be removed.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow We are reassigning partitions from node
my-cluster-pool-b-6. The reassignment can take some time depending on the number of topics and partitions in the cluster.After the reassignment process is complete, reduce the number of Kafka nodes in the source node pool.
For example, node pool
pool-bhas four replicas. We remove a node by decreasing the number of replicas:oc scale kafkanodepool pool-b --replicas=3
oc scale kafkanodepool pool-b --replicas=3Copy to Clipboard Copied! Toggle word wrap Toggle overflow The node with the highest ID (
6) within the pool is removed.Output shows three Kafka nodes in the source node pool
NAME READY STATUS RESTARTS my-cluster-pool-b-kafka-2 1/1 Running 0 my-cluster-pool-b-kafka-3 1/1 Running 0 my-cluster-pool-b-kafka-5 1/1 Running 0
NAME READY STATUS RESTARTS my-cluster-pool-b-kafka-2 1/1 Running 0 my-cluster-pool-b-kafka-3 1/1 Running 0 my-cluster-pool-b-kafka-5 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.3.6. Changing node pool roles
Node pools are used with Kafka clusters that operate in KRaft mode (using Kafka Raft metadata). If you are using KRaft mode, you can specify roles for all nodes in the node pool to operate as brokers, controllers, or both.
In certain circumstances you might want to change the roles assigned to a node pool. For example, you may have a node pool that contains nodes that perform dual broker and controller roles, and then decide to split the roles between two node pools. In this case, you create a new node pool with nodes that act only as brokers, and then reassign partitions from the dual-role nodes to the new brokers. You can then switch the old node pool to a controller-only role.
You can also perform the reverse operation by moving from node pools with controller-only and broker-only roles to a node pool that contains nodes that perform dual broker and controller roles. In this case, you add the broker role to the existing controller-only node pool, reassign partitions from the broker-only nodes to the dual-role nodes, and then delete the broker-only node pool.
When removing broker roles in the node pool configuration, keep in mind that Kafka does not automatically reassign partitions. Before removing the broker role, ensure that nodes changing to controller-only roles do not have any assigned partitions. If partitions are assigned, the change is prevented. No replicas must be left on the node before removing the broker role. The best way to reassign partitions before changing roles is to apply a Cruise Control optimization proposal in remove-brokers mode. For more information, see Section 21.3, “Generating optimization proposals”.
Scaling controller nodes in node pools is currently not supported because the related Kafka feature is still under development. For more information, see KAFKA-16538.
This procedure describes how to transition to using node pools with separate roles. If your Kafka cluster is using a node pool with combined controller and broker roles, you can transition to using two node pools with separate roles. To do this, rebalance the cluster to move partition replicas to a node pool with a broker-only role, and then switch the old node pool to a controller-only role.
In this procedure, we start with node pool pool-a, which has controller and broker roles:
Dual-role node pool
The node pool has three nodes:
Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0
NAME READY STATUS RESTARTS
my-cluster-pool-a-0 1/1 Running 0
my-cluster-pool-a-1 1/1 Running 0
my-cluster-pool-a-2 1/1 Running 0
Each node performs a combined role of broker and controller. We create a second node pool called pool-b, with three nodes that act as brokers only.
During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.
Procedure
Create a node pool with a
brokerrole.Example node pool configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The new node pool also has three nodes. If you already have a broker-only node pool, you can skip this step.
-
Apply the new
KafkaNodePoolresource to create the brokers. Check the status of the deployment and wait for the pods in the node pool to be created and ready (
1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows pods running in two node pools
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Node IDs are appended to the name of the node on creation.
Use the Cruise Control
remove-brokersmode to reassign partition replicas from the dual-role nodes to the newly added brokers.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The reassignment can take some time depending on the number of topics and partitions in the cluster.
NoteIf nodes changing to controller-only roles have any assigned partitions, the change is prevented. The
status.conditionsof theKafkaresource provide details of events preventing the change.Remove the
brokerrole from the node pool that originally had a combined role.Dual-role nodes switched to controllers
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Apply the configuration change so that the node pool switches to a controller-only role.
10.3.8. Transitioning to dual-role nodes
This procedure describes how to transition from separate node pools with broker-only and controller-only roles to using a dual-role node pool. If your Kafka cluster is using node pools with dedicated controller and broker nodes, you can transition to using a single node pool with both roles. To do this, add the broker role to the controller-only node pool, rebalance the cluster to move partition replicas to the dual-role node pool, and then delete the old broker-only node pool.
In this procedure, we start with two node pools pool-a, which has only the controller role and pool-b which has only the broker role:
Single role node pools
The Kafka cluster has six nodes:
Kafka nodes in the node pools
The pool-a nodes perform the role of controller. The pool-b nodes perform the role of broker.
During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.
Procedure
Edit the node pool
pool-aand add thebrokerrole to it.Example node pool configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status and wait for the pods in the node pool to be restarted and ready (
1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows pods running in two node pools
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Node IDs are appended to the name of the node on creation.
Use the Cruise Control
remove-brokersmode to reassign partition replicas from the broker-only nodes to the dual-role nodes.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The reassignment can take some time depending on the number of topics and partitions in the cluster.
Remove the
pool-bnode pool that has the old broker-only nodes.oc delete kafkanodepool pool-b -n <my_cluster_operator_namespace>
oc delete kafkanodepool pool-b -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.4. Configuring Kafka storage
Streams for Apache Kafka supports different Kafka storage options. You can choose between the following basic types:
- Ephemeral storage
- Ephemeral storage is temporary and only persists while a pod is running. When a pod is deleted, the data is lost, though data can be recovered in a highly available environment. Due to its transient nature, ephemeral storage is only recommended for development and testing environments.
- Persistent storage
- Persistent storage retains data across pod restarts and system disruptions, making it ideal for production environments.
JBOD (Just a Bunch of Disks) storage allows you to configure your Kafka cluster to use multiple disks or volumes as ephemeral or persistent storage.
JBOD storage (multiple volumes)
When specifying JBOD storage, you must still decide between using ephemeral or persistent volumes for each disk. Even if you start with only one volume, using JBOD allows for future scaling by adding more volumes as needed, and that is why it is always recommended.
Persistent, ephemeral, and JBOD storage types cannot be changed after a Kafka cluster is deployed. However, you can add or remove volumes of different types from the JBOD storage. You can also create and migrate to node pools with new storage specifications.
Tiered storage (advanced)
Tiered storage provides additional flexibility for managing Kafka data by combining different storage types with varying performance and cost characteristics. It allows Kafka to offload older data to cheaper, long-term storage (such as object storage) while keeping recent, frequently accessed data on faster, more expensive storage (such as block storage).
Tiered storage is an add-on capability. After configuring storage (ephemeral, persistent, or JBOD) for Kafka nodes, you can configure tiered storage at the cluster level and enable it for specific topics using the remote.storage.enable topic-level configuration.
10.4.1. Storage considerations
Efficient data storage is essential for Streams for Apache Kafka to operate effectively. Streams for Apache Kafka has been tested with block storage as the primary storage type for Kafka brokers, and block storage is strongly recommended. File system-based storage (such as NFS) is not guaranteed to work for primary broker storage and may cause stability or performance issues.
Common block storage types supported by OpenShift include:
Cloud-based block storage solutions:
- Amazon EBS (for AWS)
- Azure Disk Storage (for Microsoft Azure)
- Persistent Disk (for Google Cloud)
- Persistent storage (for bare metal deployments) using local persistent volumes
- Storage Area Network (SAN) volumes accessed by protocols like Fibre Channel or iSCSI
Streams for Apache Kafka does not require OpenShift raw block volumes.
10.4.1.1. File systems
Kafka uses a file system for storing messages. Streams for Apache Kafka is compatible with the XFS and ext4 file systems, which are commonly used with Kafka. Consider the underlying architecture and requirements of your deployment when choosing and setting up your file system.
For more information, refer to Filesystem Selection in the Kafka documentation.
10.4.1.2. Tiered storage
Kafka’s tiered storage feature is supported in Streams for Apache Kafka as an optional capability.
With tiered storage enabled:
- Primary broker storage,such as persistent volumes or JBOD, handles recent data
- Remote tiered storage, such as object storage, is used for historical data
Streams for Apache Kafka allows users to integrate their own tiered storage plugins to support specific remote storage backends. If using a custom plugin, ensure that it meets performance and compatibility requirements before deploying to production.
10.4.1.3. Disk usage
Solid-state drives (SSDs), though not essential, can improve the performance of Kafka in large clusters where data is sent to and received from multiple topics asynchronously.
Replicated storage is not required, as Kafka provides built-in data replication.
10.4.2. Configuring storage types
Use the storage properties of the KafkaNodePool custom resource to configure storage for a deployment of Kafka in KRaft mode.
10.4.2.1. Configuring ephemeral storage
To use ephemeral storage, specify ephemeral as the storage type.
Example configuration for ephemeral storage
Ephemeral storage uses emptyDir volumes, which are created when a pod is assigned to a node. You can limit the size of the emptyDir volume with the sizeLimit property.
The ephemeral volume used by Kafka brokers for log directories is mounted at /var/lib/kafka/data/kafka-log<pod_id>.
Ephemeral storage is not suitable for Kafka topics with a replication factor of 1.
For more information on ephemeral storage configuration options, see the EphemeralStorage schema reference.
10.4.2.2. Configuring persistent storage
To use persistent storage, specify one of the following as the storage type:
-
persistent-claimfor a single persistent volume -
jbodfor multiple persistent volumes in a Kafka cluster (Recommended for Kafka in a production environment)
Example configuration for persistent storage
Streams for Apache Kafka uses Persistent Volume Claims (PVCs) to request storage on persistent volumes (PVs). The PVC binds to a PV that meets the requested storage criteria, without needing to know the underlying storage infrastructure.
PVCs created for Kafka pods follow the naming convention data-<kafka_cluster_name>-<pool_name>-<pod_id>, and the persistent volumes for Kafka logs are mounted at /var/lib/kafka/data/kafka-log<pod_id>.
You can also specify custom storage classes (StorageClass) and volume selectors in the storage configuration.
Example class and selector configuration
Storage classes define storage profiles and dynamically provision persistent volumes (PVs) based on those profiles. This is useful, for example, when storage classes are restricted to different availability zones or data centers. If a storage class is not specified, the default storage class in the OpenShift cluster is used. Selectors specify persistent volumes that offer specific features, such as solid-state drive (SSD) volumes.
For more information on persistent storage configuration options, see the PersistentClaimStorage schema reference.
10.4.2.3. Resizing persistent volumes
Persistent volumes can be resized by changing the size storage property without any risk of data loss, as long as the storage infrastructure supports it. Following a configuration update to change the size of the storage, Streams for Apache Kafka instructs the storage infrastructure to make the change.
Storage expansion is supported in Streams for Apache Kafka clusters that use persistent-claim volumes. Decreasing the size of persistent volumes is not supported in OpenShift. For more information about resizing persistent volumes in OpenShift, see Resizing Persistent Volumes using Kubernetes.
After increasing the value of the size property, OpenShift increases the capacity of the selected persistent volumes in response to a request from the Cluster Operator. When the resizing is complete, the Cluster Operator restarts all pods that use the resized persistent volumes. This happens automatically.
In this example, the volumes are increased to 2000Gi.
Kafka configuration to increase volume size to 2000Gi
Returning information on the PVs verifies the changes:
oc get pv
oc get pvStorage capacity of PVs
NAME CAPACITY CLAIM pvc-0ca459ce-... 2000Gi my-project/data-my-cluster-my-node-pool-2 pvc-6e1810be-... 2000Gi my-project/data-my-cluster-my-node-pool-0 pvc-82dc78c9-... 2000Gi my-project/data-my-cluster-my-node-pool-1
NAME CAPACITY CLAIM
pvc-0ca459ce-... 2000Gi my-project/data-my-cluster-my-node-pool-2
pvc-6e1810be-... 2000Gi my-project/data-my-cluster-my-node-pool-0
pvc-82dc78c9-... 2000Gi my-project/data-my-cluster-my-node-pool-1The output shows the names of each PVC associated with a broker pod.
Storage reduction is only possible when using multiple disks per broker. You can remove a disk after moving all partitions on the disk to other volumes within the same broker (intra-broker) or to other brokers within the same cluster (intra-cluster).
10.4.2.4. Configuring JBOD storage
To use JBOD storage, specify jbod as the storage type and add configuration for the JBOD volumes. JBOD volumes can be persistent or ephemeral, with the configuration options and constraints applicable to each type.
Example configuration for JBOD storage
PVCs are created for the JBOD volumes using the naming convention data-<volume_id>-<kafka_cluster_name>-<pool_name>-<pod_id>, and the JBOD volumes used for log directories are mounted at /var/lib/kafka/data-<volume_id>/kafka-log<pod_id>.
10.4.2.5. Adding or removing volumes from JBOD storage
Volume IDs cannot be changed once JBOD volumes are created, though you can add or remove volumes. When adding a new volume to the to the volumes array under an id which was already used in the past and removed, make sure that the previously used PersistentVolumeClaims have been deleted.
Use Cruise Control to reassign partitions when adding or removing volumes. For information on intra-broker disk balancing, see Section 21.1.3, “Tuning options for rebalances”.
10.4.3. Configuring KRaft metadata log storage
In KRaft mode, each node (including brokers and controllers) stores a copy of the Kafka cluster’s metadata log on one of its data volumes. By default, the log is stored on the volume with the lowest ID, but you can specify a different volume using the kraftMetadata property.
For controller-only nodes, storage is exclusively for the metadata log. Since the log is always stored on a single volume, using JBOD storage with multiple volumes does not improve performance or increase available disk space.
In contrast, broker nodes or nodes that combine broker and controller roles can share the same volume for both the metadata log and partition replica data, optimizing disk utilization. They can also use JBOD storage, where one volume is shared for the metadata log and partition replica data, while additional volumes are used solely for partition replica data.
Changing the volume that stores the metadata log triggers a rolling update of the cluster nodes, involving the deletion of the old log and the creation of a new one in the specified location. If kraftMetadata isn’t specified, adding a new volume with a lower ID also prompts an update and relocation of the metadata log.
Example JBOD storage configuration using volume with ID 1 to store the KRaft metadata
10.4.4. Managing storage using node pools
Storage management in Streams for Apache Kafka is usually straightforward, and requires little change when set up, but there might be situations where you need to modify your storage configurations. Node pools simplify this process, because you can set up separate node pools that specify your new storage requirements.
In this procedure we create and manage storage for a node pool called pool-a containing three nodes. The steps require a scaling operation to add a new node pool. Currently, scaling is only possible for broker-only node pools containing nodes that run as dedicated brokers.
We show how to change the storage class (volumes.class) that defines the type of persistent storage it uses. You can use the same steps to change the storage size (volumes.size). This approach is particularly useful if you want to reduce disk sizes. When increasing disk sizes, you have the option to dynamically resize persistent volumes.
We strongly recommend using block storage. Streams for Apache Kafka is only tested for use with block storage.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
- For storage that uses persistent volume claims for dynamic volume allocation, storage classes are defined and available in the OpenShift cluster that correspond to the storage solutions you need.
Procedure
Create the node pool with its own storage settings.
For example, node pool
pool-auses JBOD storage with persistent volumes:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Nodes in
pool-aare configured to use Amazon EBS (Elastic Block Store) GP2 volumes.-
Apply the node pool configuration for
pool-a. Check the status of the deployment and wait for the pods in
pool-ato be created and ready (1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows three Kafka nodes in the node pool
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0
NAME READY STATUS RESTARTS my-cluster-pool-a-0 1/1 Running 0 my-cluster-pool-a-1 1/1 Running 0 my-cluster-pool-a-2 1/1 Running 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow To migrate to a new storage class, create a new node pool with the required storage configuration:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Nodes in
pool-bare configured to use Amazon EBS (Elastic Block Store) GP3 volumes.-
Apply the node pool configuration for
pool-b. -
Check the status of the deployment and wait for the pods in
pool-bto be created and ready. Reassign the partitions from
pool-atopool-b.When migrating to a new storage configuration, use the Cruise Control
remove-brokersmode to move partition replicas off the brokers that are going to be removed.Using Cruise Control to reassign partition replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow We are reassigning partitions from
pool-a. The reassignment can take some time depending on the number of topics and partitions in the cluster.After the reassignment process is complete, delete the old node pool:
oc delete kafkanodepool pool-a
oc delete kafkanodepool pool-aCopy to Clipboard Copied! Toggle word wrap Toggle overflow
10.4.5. Managing storage affinity using node pools
In situations where storage resources, such as local persistent volumes, are constrained to specific worker nodes, or availability zones, configuring storage affinity helps to schedule pods to use the right nodes.
Node pools allow you to configure affinity independently. In this procedure, we create and manage storage affinity for two availability zones: zone-1 and zone-2.
You can configure node pools for separate availability zones, but use the same storage class. We define an all-zones persistent storage class representing the storage resources available in each zone.
We also use the .spec.template.pod properties to configure the node affinity and schedule Kafka pods on zone-1 and zone-2 worker nodes.
The storage class and affinity is specified in node pools representing the nodes in each availability zone:
-
pool-zone-1 -
pool-zone-2.
Prerequisites
- The Cluster Operator must be deployed.
- If you are not familiar with the concepts of affinity, see the Kubernetes node and pod affinity documentation.
Procedure
Define the storage class for use with each availability zone:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create node pools representing the two availability zones, specifying the
all-zonesstorage class and the affinity for each zone:Node pool configuration for zone-1
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Node pool configuration for zone-2
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Apply the node pool configuration.
Check the status of the deployment and wait for the pods in the node pools to be created and ready (
1/1).oc get pods -n <my_cluster_operator_namespace>
oc get pods -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows 3 Kafka nodes in
pool-zone-1and 4 Kafka nodes inpool-zone-2Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.4.6. Tiered storage
Tiered storage introduces a flexible approach to managing Kafka data whereby log segments are moved to a separate storage system. For example, you can combine the use of block storage on brokers for frequently accessed data and offload older or less frequently accessed data from the block storage to more cost-effective, scalable remote storage solutions, such as Amazon S3, without compromising data accessibility and durability.
Tiered storage is a production-ready feature in Kafka since version 3.9.0, and it is also supported in Streams for Apache Kafka. Before introducing tiered storage to your environment, review the known limitations of this feature.
Tiered storage requires an implementation of Kafka’s RemoteStorageManager interface to handle communication between Kafka and the remote storage system, which is enabled through configuration of the Kafka resource. Streams for Apache Kafka uses Kafka’s TopicBasedRemoteLogMetadataManager for Remote Log Metadata Management (RLMM) when custom tiered storage is enabled. The RLMM manages the metadata related to remote storage.
To use custom tiered storage, do the following:
- Include a tiered storage plugin for Kafka in the Streams for Apache Kafka image by building a custom container image. The plugin must provide the necessary functionality for a Kafka cluster managed by Streams for Apache Kafka to interact with the tiered storage solution.
-
Configure Kafka for tiered storage using
tieredStorageproperties in theKafkaresource. Specify the class name and path for the customRemoteStorageManagerimplementation, as well as any additional configuration. - If required, specify RLMM-specific tiered storage configuration.
Example custom tiered storage configuration for Kafka
- 1
- The
typemust be set tocustom. - 2
- The configuration for the custom
RemoteStorageManagerimplementation, including class name and path. - 3
- Configuration to pass to the custom
RemoteStorageManagerimplementation, which Streams for Apache Kafka automatically prefixes withrsm.config.. - 4
- Tiered storage configuration to pass to the RLMM, which requires an
rlmm.config.prefix. For more information on tiered storage configuration, see the Apache Kafka documentation.
10.4.6.1. Tiered storage plugin libraries
An open-source tiered-storage-for-apache-kafka project from Aiven demonstrates how Apache Kafka can use remote object storage for tiered storage. The project provides an implementation of the RemoteStorageManager interface for Kafka’s tiered storage feature.
Streams for Apache Kafka includes plugin libraries from this project for AWS S3 and file system storage by default in Kafka v4.0 and later.
For more information, see the tiered-storage-for-apache-kafka project on GitHub.
10.4.6.2. AWS S3 remote storage configuration
Streams for Apache Kafka includes plugin libraries from the open-source tiered-storage-for-apache-kafka project to enable tiered storage using AWS S3.
To configure AWS S3 as the remote storage, specify the following RemoteStorageManager properties in the Kafka resource:
-
className: io.aiven.kafka.tieredstorage.RemoteStorageManager -
classPath: /opt/kafka/libs/tiered-storage/* -
storage.backend.class: io.aiven.kafka.tieredstorage.storage.s3.S3Storage - Any additional S3-related configuration properties
Example Kafka configuration for Amazon S3 tiered storage
- 1
- The name of the AWS S3 bucket used for storing and retrieving data.
- 2
- The AWS region of the S3 bucket (for example,
us-east-1). - 3
- The AWS access key ID used to access the S3 bucket.
- 4
- The AWS secret access key used to access the S3 bucket.
- 5
- The size (in bytes) of chunks into which segment files are split. It’s recommended to start with "4194304" (4MiB).
10.4.6.3. File system remote storage configuration
Streams for Apache Kafka includes plugin libraries from the open-source tiered-storage-for-apache-kafka project to enable tiered storage using file system paths. The file system storage plugin supports any mountable file system that can be accessed using a local path (for example, /mnt/tiered-storage/).
To configure a file system as remote storage (this example uses NFS), follow these steps:
-
Mount the NFS volume in all
KafkaNodePoolresources performing a broker role using the additional volumes feature.KafkaNodePoolresources with only the controller role do not require this configuration, as they don’t use tiered storage. Specify the following
RemoteStorageManagerproperties in theKafkaresource:-
className: io.aiven.kafka.tieredstorage.RemoteStorageManager -
classPath: /opt/kafka/libs/tiered-storage/* -
storage.backend.class: io.aiven.kafka.tieredstorage.storage.filesystem.FileSystemStorage - Any additional file system–related configuration properties
-
Example node pool configuration for file system (NFS) tiered storage
Example Kafka configuration for file system (NFS) tiered storage
10.4.6.4. Common tiered storage tuning properties
In addition to the required configuration properties, Streams for Apache Kafka supports a set of optional tuning properties for the tiered storage plugin. These properties can help optimize storage performance, throughput, and resource usage depending on your environment.
Common tuning properties include the following:
storage.chunk.cache.prefetch.max.size-
Controls the maximum amount of data (in bytes) to prefetch and cache. This setting can improve performance when reading chunked segments.
Default: 0 storage.upload.rate.limit.bytes.per.second-
Sets an upper bound on the upload rate from disk (in bytes per second). The value must be at least 1 MiB/s.
Default: No limit storage.compression.enabled-
Enables additional compression of segment data to reduce storage usage.
Default: false storage.s3.multipart.upload.part.size-
Sets the part size (in bytes) used when performing multipart uploads to S3. Tuning this value can improve upload performance and compatibility with S3 limits.
Default: 26214400 (25 MiB)
For a full list of supported configuration options, see the tiered-storage-for-apache-kafka project documentation.
10.5. Configuring the Entity Operator
Use the entityOperator property in Kafka.spec to configure the Entity Operator. The Entity Operator is responsible for managing Kafka-related entities in a running Kafka cluster. It comprises the following operators:
- Topic Operator to manage Kafka topics
- User Operator to manage Kafka users
By configuring the Kafka resource, the Cluster Operator can deploy the Entity Operator, including one or both operators. Once deployed, the operators are automatically configured to handle the topics and users of the Kafka cluster.
Each operator can only monitor a single namespace. For more information, see Section 1.2.1, “Operator-watched Kafka resources”.
The entityOperator property supports several sub-properties:
-
topicOperator -
userOperator -
template
The template property contains the configuration of the Entity Operator pod, such as labels, annotations, affinity, and tolerations. For more information on configuring templates, see Section 10.12.14, “Customizing OpenShift resources”.
The topicOperator property contains the configuration of the Topic Operator. When this option is missing, the Entity Operator is deployed without the Topic Operator.
The userOperator property contains the configuration of the User Operator. When this option is missing, the Entity Operator is deployed without the User Operator.
For more information on the properties used to configure the Entity Operator, see the EntityOperatorSpec schema reference.
Example of basic configuration enabling both operators
If an empty object ({}) is used for the topicOperator and userOperator, all properties use their default values.
When both topicOperator and userOperator properties are missing, the Entity Operator is not deployed.
10.5.1. Configuring the Topic Operator
Use topicOperator properties in Kafka.spec.entityOperator to configure the Topic Operator.
The following properties are supported:
watchedNamespace-
The OpenShift namespace in which the Topic Operator watches for
KafkaTopicresources. Default is the namespace where the Kafka cluster is deployed. reconciliationIntervalMs-
The interval between periodic reconciliations in milliseconds. Default
120000. image-
The
imageproperty can be used to configure the container image which is used. To learn more, refer to the information provided on configuring theimageproperty`. resources-
The
resourcesproperty configures the amount of resources allocated to the Topic Operator. You can specify requests and limits formemoryandcpuresources. The requests should be enough to ensure a stable performance of the operator. logging-
The
loggingproperty configures the logging of the Topic Operator. To learn more, refer to the information provided on Topic Operator logging.
Example Topic Operator configuration
10.5.2. Configuring the User Operator
Use userOperator properties in Kafka.spec.entityOperator to configure the User Operator. The following properties are supported:
watchedNamespace-
The OpenShift namespace in which the User Operator watches for
KafkaUserresources. Default is the namespace where the Kafka cluster is deployed. reconciliationIntervalMs-
The interval between periodic reconciliations in milliseconds. Default
120000. image-
The
imageproperty can be used to configure the container image which will be used. To learn more, refer to the information provided on configuring theimageproperty`. resources-
The
resourcesproperty configures the amount of resources allocated to the User Operator. You can specify requests and limits formemoryandcpuresources. The requests should be enough to ensure a stable performance of the operator. logging-
The
loggingproperty configures the logging of the User Operator. To learn more, refer to the information provided on User Operator logging. secretPrefix-
The
secretPrefixproperty adds a prefix to the name of all Secrets created from the KafkaUser resource. For example,secretPrefix: kafka-would prefix all Secret names withkafka-. So a KafkaUser namedmy-userwould create a Secret namedkafka-my-user.
Example User Operator configuration
10.6. Configuring the Cluster Operator
Use environment variables to configure the Cluster Operator. Specify the environment variables for the container image of the Cluster Operator in its Deployment configuration file. You can use the following environment variables to configure the Cluster Operator. If you are running Cluster Operator replicas in standby mode, there are additional environment variables for enabling leader election.
Kafka, Kafka Connect, and Kafka MirrorMaker support multiple versions. Use their STRIMZI_<COMPONENT_NAME>_IMAGES environment variables to configure the default container images used for each version. The configuration provides a mapping between a version and an image. The required syntax is whitespace or comma-separated <version> = <image> pairs, which determine the image to use for a given version. For example, 4.1.0=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. Theses default images are overridden if image property values are specified in the configuration of a component. For more information on image configuration of components, see the Streams for Apache Kafka Custom Resource API Reference.
The Deployment configuration file provided with the Streams for Apache Kafka release artifacts is install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml.
STRIMZI_NAMESPACEA comma-separated list of namespaces that the operator operates in. When not set, set to empty string, or set to
*, the Cluster Operator operates in all namespaces.The Cluster Operator deployment might use the downward API to set this automatically to the namespace the Cluster Operator is deployed in.
Example configuration for Cluster Operator namespaces
env: - name: STRIMZI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceenv: - name: STRIMZI_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_FULL_RECONCILIATION_INTERVAL_MS- Optional, default is 120000 ms. The interval between periodic reconciliations, in milliseconds.
STRIMZI_OPERATION_TIMEOUT_MS- Optional, default 300000 ms. The timeout for internal operations, in milliseconds. Increase this value when using Streams for Apache Kafka on clusters where regular OpenShift operations take longer than usual (due to factors such as prolonged download times for container images, for example).
STRIMZI_OPERATIONS_THREAD_POOL_SIZE- Optional, default 10. The worker thread pool size, which is used for various asynchronous and blocking operations that are run by the Cluster Operator.
STRIMZI_OPERATOR_NAME- Optional, defaults to the pod’s hostname. The operator name identifies the Streams for Apache Kafka instance when emitting OpenShift events.
STRIMZI_OPERATOR_NAMESPACEThe name of the namespace where the Cluster Operator is running. Do not configure this variable manually. Use the downward API.
env: - name: STRIMZI_OPERATOR_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceenv: - name: STRIMZI_OPERATOR_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_OPERATOR_NAMESPACE_LABELSOptional. The labels of the namespace where the Streams for Apache Kafka Cluster Operator is running. Use namespace labels to configure the namespace selector in network policies. Network policies allow the Streams for Apache Kafka Cluster Operator access only to the operands from the namespace with these labels. When not set, the namespace selector in network policies is configured to allow access to the Cluster Operator from any namespace in the OpenShift cluster.
env: - name: STRIMZI_OPERATOR_NAMESPACE_LABELS value: label1=value1,label2=value2env: - name: STRIMZI_OPERATOR_NAMESPACE_LABELS value: label1=value1,label2=value2Copy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_POD_DISRUPTION_BUDGET_GENERATIONOptional. Default is
true. Controls automatic creation ofPodDisruptionBudgetresources for Kafka, Kafka Connect, MirrorMaker2, and Kafka Bridge. Each budget applies across all pods deployed for the associated component. For Kafka clusters, this includes all node pool pods.A pod disruption budget with the
maxUnavailablevalue set to zero prevents OpenShift from evicting pods automatically.Set this environment variable to
falseto disable pod disruption budget generation. You might do this, for example, if you want to manage the pod disruption budgets yourself, or if you have a development environment where availability is not important.STRIMZI_LABELS_EXCLUSION_PATTERNOptional, default regex pattern is
(^app.kubernetes.io/(?!part-of).*|^kustomize.toolkit.fluxcd.io.*). The regex exclusion pattern used to filter labels propagation from the main custom resource to its subresources. The labels exclusion filter is not applied to labels in template sections such asspec.kafka.template.pod.metadata.labels.env: - name: STRIMZI_LABELS_EXCLUSION_PATTERN value: "^key1.*"env: - name: STRIMZI_LABELS_EXCLUSION_PATTERN value: "^key1.*"Copy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_CUSTOM_<COMPONENT_NAME>_LABELSOptional. One or more custom labels to apply to all the pods created by the custom resource of the component. The Cluster Operator labels the pods when the custom resource is created or is next reconciled.
Labels can be applied to the following components:
-
KAFKA -
KAFKA_CONNECT -
KAFKA_CONNECT_BUILD -
ENTITY_OPERATOR -
KAFKA_MIRROR_MAKER2 -
KAFKA_MIRROR_MAKER -
CRUISE_CONTROL -
KAFKA_BRIDGE -
KAFKA_EXPORTER
-
STRIMZI_CUSTOM_RESOURCE_SELECTOROptional. The label selector to filter the custom resources handled by the Cluster Operator. The operator will operate only on those custom resources that have the specified labels set. Resources without these labels will not be seen by the operator. The label selector applies to
Kafka,KafkaConnect,KafkaBridge, andKafkaMirrorMaker2resources.KafkaRebalanceandKafkaConnectorresources are operated only when their corresponding Kafka and Kafka Connect clusters have the matching labels.env: - name: STRIMZI_CUSTOM_RESOURCE_SELECTOR value: label1=value1,label2=value2env: - name: STRIMZI_CUSTOM_RESOURCE_SELECTOR value: label1=value1,label2=value2Copy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_KAFKA_IMAGES-
Required. The mapping from the Kafka version to the corresponding image containing a Kafka broker for that version. For example
4.0.0=registry.redhat.io/amq-streams/kafka-40-rhel9:3.1.0, 4.1.0=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. STRIMZI_KAFKA_CONNECT_IMAGES-
Required. The mapping from the Kafka version to the corresponding image of Kafka Connect for that version. For example
4.0.0=registry.redhat.io/amq-streams/kafka-40-rhel9:3.1.0, 4.1.0=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. STRIMZI_KAFKA_MIRROR_MAKER2_IMAGES-
Required. The mapping from the Kafka version to the corresponding image of MirrorMaker 2 for that version. For example
4.0.0=registry.redhat.io/amq-streams/kafka-40-rhel9:3.1.0, 4.1.0=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/strimzi-rhel9-operator:3.1.0. The image name to use as the default when deploying the Topic Operator if no image is specified as theKafka.spec.entityOperator.topicOperator.imagein theKafkaresource. STRIMZI_DEFAULT_USER_OPERATOR_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/strimzi-rhel9-operator:3.1.0. The image name to use as the default when deploying the User Operator if no image is specified as theKafka.spec.entityOperator.userOperator.imagein theKafkaresource. STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. The image name to use as the default when deploying the Kafka Exporter if no image is specified as theKafka.spec.kafkaExporter.imagein theKafkaresource. STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. The image name to use as the default when deploying Cruise Control if no image is specified as theKafka.spec.cruiseControl.imagein theKafkaresource. STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/bridge-rhel9:3.1.0. The image name to use as the default when deploying the Kafka Bridge if no image is specified as theKafka.spec.kafkaBridge.imagein theKafkaresource. STRIMZI_DEFAULT_KAFKA_INIT_IMAGE-
Optional. The default is
registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0. The image name to use as the default for the Kafka initializer container if no image is specified in thebrokerRackInitImageof theKafkaresource or theclientRackInitImageof the Kafka Connect resource. The init container is started before the Kafka cluster for initial configuration work, such as rack support. STRIMZI_IMAGE_PULL_POLICY-
Optional. The
ImagePullPolicythat is applied to containers in all pods managed by the Cluster Operator. The valid values areAlways,IfNotPresent, andNever. If not specified, the OpenShift defaults are used. Changing the policy will result in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters. STRIMZI_IMAGE_PULL_SECRETS-
Optional. A comma-separated list of
Secretnames. The secrets referenced here contain the credentials to the container registries where the container images are pulled from. The secrets are specified in theimagePullSecretsproperty for all pods created by the Cluster Operator. Changing this list results in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters. STRIMZI_KUBERNETES_VERSIONOptional. Overrides the OpenShift version information detected from the API server.
Example configuration for OpenShift version override
Copy to Clipboard Copied! Toggle word wrap Toggle overflow KUBERNETES_SERVICE_DNS_DOMAINOptional. Overrides the default OpenShift DNS domain name suffix.
By default, services assigned in the OpenShift cluster have a DNS domain name that uses the default suffix
cluster.local.For example, for broker kafka-0:
<cluster-name>-kafka-0.<cluster-name>-kafka-brokers.<namespace>.svc.cluster.local
<cluster-name>-kafka-0.<cluster-name>-kafka-brokers.<namespace>.svc.cluster.localCopy to Clipboard Copied! Toggle word wrap Toggle overflow The DNS domain name is added to the Kafka broker certificates used for hostname verification.
If you are using a different DNS domain name suffix in your cluster, change the
KUBERNETES_SERVICE_DNS_DOMAINenvironment variable from the default to the one you are using in order to establish a connection with the Kafka brokers.STRIMZI_CONNECT_BUILD_TIMEOUT_MS- Optional, default 300000 ms. The timeout for building new Kafka Connect images with additional connectors, in milliseconds. Consider increasing this value when using Streams for Apache Kafka to build container images containing many connectors or using a slow container registry.
STRIMZI_NETWORK_POLICY_GENERATIONOptional, default
true. Network policy for resources. Network policies allow connections between Kafka components.Set this environment variable to
falseto disable network policy generation. You might do this, for example, if you want to use custom network policies. Custom network policies allow more control over maintaining the connections between components.STRIMZI_DNS_CACHE_TTL-
Optional, default
30. Number of seconds to cache successful name lookups in local DNS resolver. Any negative value means cache forever. Zero means do not cache, which can be useful for avoiding connection errors due to long caching policies being applied. STRIMZI_POD_SET_RECONCILIATION_ONLY-
Optional, default
false. When set totrue, the Cluster Operator reconciles only theStrimziPodSetresources and any changes to the other custom resources (Kafka,KafkaConnect, and so on) are ignored. This mode is useful for ensuring that your pods are recreated if needed, but no other changes happen to the clusters. STRIMZI_FEATURE_GATES- Optional. Enables or disables the features and functionality controlled by feature gates.
STRIMZI_POD_SECURITY_PROVIDER_CLASS-
Optional. Configuration for the pluggable
PodSecurityProviderclass, which can be used to provide the security context configuration for Pods and containers.
Use the STRIMZI_OPERATOR_NAMESPACE_LABELS environment variable to establish network policy for the Cluster Operator using namespace labels.
The Cluster Operator can run in the same namespace as the resources it manages, or in a separate namespace. By default, the STRIMZI_OPERATOR_NAMESPACE environment variable is configured to use the downward API to find the namespace the Cluster Operator is running in. If the Cluster Operator is running in the same namespace as the resources, only local access is required and allowed by Streams for Apache Kafka.
If the Cluster Operator is running in a separate namespace to the resources it manages, any namespace in the OpenShift cluster is allowed access to the Cluster Operator unless network policy is configured. By adding namespace labels, access to the Cluster Operator is restricted to the namespaces specified.
Network policy configured for the Cluster Operator deployment
Use the STRIMZI_FULL_RECONCILIATION_INTERVAL_MS variable to set the time interval for periodic reconciliations by the Cluster Operator. Replace its value with the required interval in milliseconds.
Reconciliation period configured for the Cluster Operator deployment
The Cluster Operator reacts to all notifications about applicable cluster resources received from the OpenShift cluster. If the operator is not running, or if a notification is not received for any reason, resources will get out of sync with the state of the running OpenShift cluster. In order to handle failovers properly, a periodic reconciliation process is executed by the Cluster Operator so that it can compare the state of the resources with the current cluster deployments in order to have a consistent state across all of them.
Sometimes it is useful to pause the reconciliation of custom resources managed by Streams for Apache Kafka operators, so that you can perform fixes or make updates. If reconciliations are paused, any changes made to custom resources are ignored by the operators until the pause ends.
If you want to pause reconciliation of a custom resource, set the strimzi.io/pause-reconciliation annotation to true in its configuration. This instructs the appropriate operator to pause reconciliation of the custom resource. For example, you can apply the annotation to the KafkaConnect resource so that reconciliation by the Cluster Operator is paused.
You can also create a custom resource with the pause annotation enabled. The custom resource is created, but it is ignored.
Prerequisites
- The Streams for Apache Kafka Operator that manages the custom resource is running.
Procedure
Annotate the custom resource in OpenShift, setting
pause-reconciliationtotrue:oc annotate <kind_of_custom_resource> <name_of_custom_resource> strimzi.io/pause-reconciliation="true"
oc annotate <kind_of_custom_resource> <name_of_custom_resource> strimzi.io/pause-reconciliation="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example, for the
KafkaConnectcustom resource:oc annotate KafkaConnect my-connect strimzi.io/pause-reconciliation="true"
oc annotate KafkaConnect my-connect strimzi.io/pause-reconciliation="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the status conditions of the custom resource show a change to
ReconciliationPaused:oc describe <kind_of_custom_resource> <name_of_custom_resource>
oc describe <kind_of_custom_resource> <name_of_custom_resource>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
typecondition changes toReconciliationPausedat thelastTransitionTime.Example custom resource with a paused reconciliation condition type
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Resuming from pause
-
To resume reconciliation, you can set the annotation to
false, or remove the annotation.
The default Cluster Operator configuration enables leader election to run multiple parallel replicas of the Cluster Operator. One replica is elected as the active leader and operates the deployed resources. The other replicas run in standby mode. When the leader stops or fails, one of the standby replicas is elected as the new leader and starts operating the deployed resources.
By default, Streams for Apache Kafka runs with a single Cluster Operator replica that is always the leader replica. When a single Cluster Operator replica stops or fails, OpenShift starts a new replica.
Running the Cluster Operator with multiple replicas is not essential. But it’s useful to have replicas on standby in case of large-scale disruptions caused by major failure. For example, suppose multiple worker nodes or an entire availability zone fails. This failure might cause the Cluster Operator pod and many Kafka pods to go down at the same time. If subsequent pod scheduling causes congestion through lack of resources, this can delay operations when running a single Cluster Operator.
Configure leader election environment variables when running additional Cluster Operator replicas. The following environment variables are supported:
STRIMZI_LEADER_ELECTION_ENABLED-
Optional, disabled (
false) by default. Enables or disables leader election, which allows additional Cluster Operator replicas to run on standby.
Leader election is disabled by default. It is only enabled when applying this environment variable on installation.
STRIMZI_LEADER_ELECTION_LEASE_NAME-
Required when leader election is enabled. The name of the OpenShift
Leaseresource that is used for the leader election. STRIMZI_LEADER_ELECTION_LEASE_NAMESPACERequired when leader election is enabled. The namespace where the OpenShift
Leaseresource used for leader election is created. You can use the downward API to configure it to the namespace where the Cluster Operator is deployed.env: - name: STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceenv: - name: STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespaceCopy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_LEADER_ELECTION_IDENTITYRequired when leader election is enabled. Configures the identity of a given Cluster Operator instance used during the leader election. The identity must be unique for each operator instance. You can use the downward API to configure it to the name of the pod where the Cluster Operator is deployed.
env: - name: STRIMZI_LEADER_ELECTION_IDENTITY valueFrom: fieldRef: fieldPath: metadata.nameenv: - name: STRIMZI_LEADER_ELECTION_IDENTITY valueFrom: fieldRef: fieldPath: metadata.nameCopy to Clipboard Copied! Toggle word wrap Toggle overflow STRIMZI_LEADER_ELECTION_LEASE_DURATION_MS- Optional, default 15000 ms. Specifies the duration the acquired lease is valid.
STRIMZI_LEADER_ELECTION_RENEW_DEADLINE_MS- Optional, default 10000 ms. Specifies the period the leader should try to maintain leadership.
STRIMZI_LEADER_ELECTION_RETRY_PERIOD_MS- Optional, default 2000 ms. Specifies the frequency of updates to the lease lock by the leader.
10.6.4.2. Configuring Cluster Operator replicas
To run additional Cluster Operator replicas in standby mode, you will need to increase the number of replicas and enable leader election. To configure leader election, use the leader election environment variables.
To make the required changes, configure the following Cluster Operator installation files located in install/cluster-operator/:
- 060-Deployment-strimzi-cluster-operator.yaml
- 022-ClusterRole-strimzi-cluster-operator-role.yaml
- 022-RoleBinding-strimzi-cluster-operator.yaml
Leader election has its own ClusterRole and RoleBinding RBAC resources that target the namespace where the Cluster Operator is running, rather than the namespace it is watching.
The default deployment configuration creates a Lease resource called strimzi-cluster-operator in the same namespace as the Cluster Operator. The Cluster Operator uses leases to manage leader election. The RBAC resources provide the permissions to use the Lease resource. If you use a different Lease name or namespace, update the ClusterRole and RoleBinding files accordingly.
Prerequisites
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Procedure
Edit the Deployment resource that is used to deploy the Cluster Operator, which is defined in the 060-Deployment-strimzi-cluster-operator.yaml file.
Change the
replicasproperty from the default (1) to a value that matches the required number of replicas.Increasing the number of Cluster Operator replicas
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the leader election
envproperties are set.If they are not set, configure them.
To enable leader election,
STRIMZI_LEADER_ELECTION_ENABLEDmust be set totrue(default).In this example, the name of the lease is changed to
my-strimzi-cluster-operator.Configuring leader election environment variables for the Cluster Operator
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For a description of the available environment variables, see Section 10.6.4.1, “Enabling leader election for Cluster Operator replicas”.
If you specified a different name or namespace for the
Leaseresource used in leader election, update the RBAC resources.(optional) Edit the
ClusterRoleresource in the022-ClusterRole-strimzi-cluster-operator-role.yamlfile.Update
resourceNameswith the name of theLeaseresource.Updating the ClusterRole references to the lease
Copy to Clipboard Copied! Toggle word wrap Toggle overflow (optional) Edit the
RoleBindingresource in the022-RoleBinding-strimzi-cluster-operator.yamlfile.Update
subjects.nameandsubjects.namespacewith the name of theLeaseresource and the namespace where it was created.Updating the RoleBinding references to the lease
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Cluster Operator:
oc create -f install/cluster-operator -n myproject
oc create -f install/cluster-operator -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments -n myproject
oc get deployments -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 3/3 3 3
NAME READY UP-TO-DATE AVAILABLE strimzi-cluster-operator 3/3 3 3Copy to Clipboard Copied! Toggle word wrap Toggle overflow READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows the correct number of replicas.
10.6.5. Configuring Cluster Operator HTTP proxy settings
If you are running a Kafka cluster behind a HTTP proxy, you can still pass data in and out of the cluster. For example, you can run Kafka Connect with connectors that push and pull data from outside the proxy. Or you can use a proxy to connect with an authorization server.
Configure the Cluster Operator deployment to specify the proxy environment variables. The Cluster Operator accepts standard proxy configuration (HTTP_PROXY, HTTPS_PROXY and NO_PROXY) as environment variables. The proxy settings are applied to all Streams for Apache Kafka containers.
The format for a proxy address is http://<ip_address>:<port_number>. To set up a proxy with a name and password, the format is http://<username>:<password>@<ip-address>:<port_number>.
Prerequisites
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Procedure
To add proxy environment variables to the Cluster Operator, update its
Deploymentconfiguration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml).Example proxy configuration for the Cluster Operator
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Alternatively, edit the
Deploymentdirectly:oc edit deployment strimzi-cluster-operator
oc edit deployment strimzi-cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you updated the YAML file instead of editing the
Deploymentdirectly, apply the changes:oc create -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml
oc create -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Streams for Apache Kafka automatically switches to FIPS mode when running on a FIPS-enabled OpenShift cluster. Disable FIPS mode by setting the FIPS_MODE environment variable to disabled in the deployment configuration for the Cluster Operator. With FIPS mode disabled, Streams for Apache Kafka automatically disables FIPS in the OpenJDK for all components. With FIPS mode disabled, Streams for Apache Kafka is not FIPS compliant. The Streams for Apache Kafka operators, as well as all operands, run in the same way as if they were running on an OpenShift cluster without FIPS enabled.
Procedure
To disable the FIPS mode in the Cluster Operator, update its
Deploymentconfiguration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml) and add theFIPS_MODEenvironment variable.Example FIPS configuration for the Cluster Operator
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Disables the FIPS mode.
Alternatively, edit the
Deploymentdirectly:oc edit deployment strimzi-cluster-operator
oc edit deployment strimzi-cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you updated the YAML file instead of editing the
Deploymentdirectly, apply the changes:oc apply -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml
oc apply -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow
10.7. Configuring Kafka Connect
Update the spec properties of the KafkaConnect custom resource to configure your Kafka Connect deployment.
Use Kafka Connect to set up external data connections to your Kafka cluster. Use the properties of the KafkaConnect resource to configure your Kafka Connect deployment.
You can also use the KafkaConnect resource to specify the following:
- Connector plugin configuration to build a container image that includes the plugins to make connections
- Configuration for the Kafka Connect worker pods that run connectors
-
An annotation to enable use of the
KafkaConnectorresource to manage connectors
The Cluster Operator manages Kafka Connect clusters deployed using the KafkaConnect resource and connectors created using the KafkaConnector resource.
For a deeper understanding of the Kafka Connect cluster configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.
Handling high volumes of messages
You can tune the configuration to handle high volumes of messages. For more information, see Handling high volumes of messages^.
Example KafkaConnect custom resource configuration
- 1
- Use
KafkaConnect. - 2
- Enables the use of
KafkaConnectorresources to start, stop, and manage connector instances. - 3
- The number of replica nodes for the workers that run tasks.
- 4
- Bootstrap address for connection to the Kafka cluster. The address takes the format
<cluster_name>-kafka-bootstrap:<port_number>. The Kafka cluster doesn’t need to be managed by Streams for Apache Kafka or deployed to an OpenShift cluster. - 5
- Kafka Connect configuration of workers (not connectors) that run connectors and their tasks. Standard Apache Kafka configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka. In this example, JSON convertors are specified. A replication factor of 3 is set for the internal topics used by Kafka Connect (minimum requirement for production environment). Changing the replication factor after the topics have been created has no effect.
- 6
- Requests for reservation of supported resources, currently
cpuandmemory, and limits to specify the maximum resources that can be consumed. - 7
- Authentication for the Kafka Connect cluster, specified as
tls,scram-sha-256,scram-sha-512,plain, oroauth. By default, Kafka Connect connects to Kafka brokers using a plaintext connection. For details on configuring authentication, see theKafkaConnectSpecschema properties. - 8
- TLS configuration for encrypted connections to the Kafka cluster, with trusted certificates stored in X.509 format within the specified secrets.
- 9
- Build configuration properties for building a container image with connector plugins automatically.
- 10
- (Required) Configuration of the container registry where new images are pushed.
- 11
- (Required) List of connector plugins and their artifacts to add to the new container image. Each plugin must be configured with at least one
artifact. - 12
- Kafka Connect loggers and log levels added directly (
inline) or indirectly (external) through aConfigMap. Custom Log4j configuration must be placed under thelog4j2.propertieskey in theConfigMap. You can set log levels toINFO,ERROR,WARN,TRACE,DEBUG,FATALorOFF. - 13
- Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).
- 14
- Prometheus metrics, which are enabled by referencing a ConfigMap containing configuration for the Prometheus JMX exporter in this example. You can enable metrics without further configuration using a reference to a ConfigMap containing an empty file under
metricsConfig.valueFrom.configMapKeyRef.key. - 15
- JVM configuration options to optimize performance for the Virtual Machine (VM) running Kafka Connect.
- 16
- ADVANCED OPTION: Container image configuration, which is recommended only in special situations.
- 17
- SPECIALIZED OPTION: Rack awareness configuration for the deployment. This is a specialized option intended for a deployment within the same location, not across regions. Use this option if you want connectors to consume from the closest replica rather than the leader replica. In certain cases, consuming from the closest replica can improve network utilization or reduce costs . The
topologyKeymust match a node label containing the rack ID. The example used in this configuration specifies a zone using the standardtopology.kubernetes.io/zonelabel. To consume from the closest replica, enable theRackAwareReplicaSelectorin the Kafka broker configuration. - 18
- Template customization. Here a pod is scheduled with anti-affinity, so the pod is not scheduled on nodes with the same hostname.
- 19
- Environment variables are set for distributed tracing and to pass credentials to connectors.
- 20
- Distributed tracing is enabled by using OpenTelemetry.
10.7.1. Configuring Kafka Connect for multiple instances
By default, Streams for Apache Kafka configures the group ID and names of the internal topics used by Kafka Connect. When running multiple instances of Kafka Connect, you must change these default settings using the following config properties:
Values for the three topics must be the same for all instances with the same group.id.
Unless you modify these default settings, each instance connecting to the same Kafka cluster is deployed with the same values. In practice, this means all instances form a cluster and use the same internal topics.
Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values of these properties for each instance.
10.7.2. Configuring Kafka Connect user authorization
When using authorization in Kafka, a Kafka Connect user requires read/write access to the cluster group and internal topics of Kafka Connect. This procedure outlines how access is granted using simple authorization and ACLs.
Properties for the Kafka Connect cluster group ID and internal topics are configured by Streams for Apache Kafka by default. Alternatively, you can define them explicitly in the spec of the KafkaConnect resource. This is useful when configuring Kafka Connect for multiple instances, as the values for the group ID and topics must differ when running multiple Kafka Connect instances.
Simple authorization uses ACL rules managed by the Kafka StandardAuthorizer plugin to ensure appropriate access levels. For more information on configuring a KafkaUser resource to use simple authorization, see the AclRule schema reference.
Prerequisites
- An OpenShift cluster
- A running Cluster Operator
Procedure
Edit the
authorizationproperty in theKafkaUserresource to provide access rights to the user.Access rights are configured for the Kafka Connect topics and cluster group using
literalname values. The following table shows the default names configured for the topics and cluster group ID.Expand Table 10.1. Names for the access rights configuration Property Name offset.storage.topicconnect-cluster-offsetsstatus.storage.topicconnect-cluster-statusconfig.storage.topicconnect-cluster-configsgroupconnect-clusterIn this example configuration, the default names are used to specify access rights. If you are using different names for a Kafka Connect instance, use those names in the ACLs configuration.
Example configuration for simple authorization
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource.
oc apply -f KAFKA-USER-CONFIG-FILE
oc apply -f KAFKA-USER-CONFIG-FILECopy to Clipboard Copied! Toggle word wrap Toggle overflow
10.8. Configuring Kafka Connect connectors
The KafkaConnector resource provides an OpenShift-native approach to management of connectors by the Cluster Operator. To create, delete, or reconfigure connectors with KafkaConnector resources, you must set the use-connector-resources annotation to true in your KafkaConnect custom resource.
Annotation to enable KafkaConnectors
When the use-connector-resources annotation is enabled in your KafkaConnect configuration, you must define and manage connectors using KafkaConnector resources.
Alternatively, you can manage connectors using the Kafka Connect REST API instead of KafkaConnector resources. To use the API, you must remove the strimzi.io/use-connector-resources annotation to use KafkaConnector resources in the KafkaConnect the resource.
KafkaConnector resources provide the configuration needed to create connectors within a Kafka Connect cluster, which interacts with a Kafka cluster as specified in the KafkaConnect configuration. The Kafka cluster does not need to be managed by Streams for Apache Kafka or deployed to an OpenShift cluster.
Kafka components contained in the same OpenShift cluster
The configuration also specifies how the connector instances interact with external data systems, including any required authentication methods. Additionally, you must define the data to watch. For example, in a source connector that reads data from a database, the configuration might include the database name. You can also define where this data should be placed in Kafka by specifying the target topic name.
Use the tasksMax property to specify the maximum number of tasks. For instance, a source connector with tasksMax: 2 might split the import of source data into two tasks.
Example source connector configuration
- 1
- Name of the
KafkaConnectorresource, which is used as the name of the connector. Use any name that is valid for an OpenShift resource. - 2
- Name of the Kafka Connect cluster to create the connector instance in. Connectors must be deployed to the same namespace as the Kafka Connect cluster they link to.
- 3
- Full name of the connector class. This should be present in the image being used by the Kafka Connect cluster.
- 4
- Maximum number of Kafka Connect tasks that the connector can create.
- 5
- Enables automatic restarts of failed connectors and tasks. By default, the number of restarts is indefinite, but you can set a maximum on the number of automatic restarts using the
maxRestartsproperty. - 6
- Connector configuration as key-value pairs.
- 7
- Location of the external data file. In this example, we’re configuring the
FileStreamSourceConnectorto read from the/opt/kafka/LICENSEfile. - 8
- Kafka topic to publish the source data to.
To include external connector configurations, such as user access credentials stored in a secret, use the template property of the KafkaConnect resource. You can also load values using configuration providers.
If you are using KafkaConnector resources to configure connectors, use the state configuration to either stop or pause a connector. In contrast to the paused state, where the connector and tasks remain instantiated, stopping a connector retains only the configuration, with no active processes. Stopping a connector from running may be more suitable for longer durations than just pausing. While a paused connector is quicker to resume, a stopped connector has the advantages of freeing up memory and resources.
The state configuration replaces the (deprecated) pause configuration in the KafkaConnectorSpec schema, which allows pauses on connectors. If you were previously using the pause configuration to pause connectors, we encourage you to transition to using the state configuration only to avoid conflicts.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaConnectorcustom resource that controls the connector you want to pause or stop:oc get KafkaConnector
oc get KafkaConnectorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
KafkaConnectorresource to stop or pause the connector.Example configuration for stopping a Kafka Connect connector
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change the
stateconfiguration tostoppedorpaused. The default state for the connector when this property is not set isrunning.Apply the changes to the
KafkaConnectorconfiguration.You can resume the connector by changing
statetorunningor removing the configuration.
Alternatively, you can expose the Kafka Connect API and use the stop and pause endpoints to stop a connector from running. For example, PUT /connectors/<connector_name>/stop. You can then use the resume endpoint to restart it.
10.8.2. Manually restarting Kafka Connect connectors
If you are using KafkaConnector resources to manage connectors, use the strimzi.io/restart annotation to manually trigger a restart of a connector.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaConnectorcustom resource that controls the Kafka connector you want to restart:oc get KafkaConnector
oc get KafkaConnectorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Restart the connector by annotating the
KafkaConnectorresource in OpenShift:oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart="true"
oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
restartannotation is set totrue.You can also refine the restart behavior with the
includeTasksandonlyFailedparameters, which both default tofalse.-
includeTasksrestarts both the connector instance and its task instances. -
onlyFailedrestarts only instances with aFAILEDstatus when set totrue.
For example:
strimzi.io/restart="includeTasks,onlyFailed"
strimzi.io/restart="includeTasks,onlyFailed"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Wait for the next reconciliation to occur (every two minutes by default).
The Kafka connector is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the
KafkaConnectorcustom resource.
If you are using KafkaConnector resources to manage connectors, use the strimzi.io/restart-task annotation to manually trigger a restart of a connector task.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaConnectorcustom resource that controls the Kafka connector task you want to restart:oc get KafkaConnector
oc get KafkaConnectorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Find the ID of the task to be restarted from the
KafkaConnectorcustom resource:oc describe KafkaConnector <kafka_connector_name>
oc describe KafkaConnector <kafka_connector_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Task IDs are non-negative integers, starting from 0.
Use the ID to restart the connector task by annotating the
KafkaConnectorresource in OpenShift:oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart-task="0"
oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart-task="0"Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, task
0is restarted.Wait for the next reconciliation to occur (every two minutes by default).
The Kafka connector task is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the
KafkaConnectorcustom resource.
10.8.4. Listing connector offsets
To track connector offsets using KafkaConnector resources, add the listOffsets configuration. The offsets, which keep track of the flow of data, are written to a config map specified in the configuration. If the config map does not exist, Streams for Apache Kafka creates it.
After the configuration is in place, annotate the KafkaConnector resource to write the list to the config map.
Sink connectors use Kafka’s standard consumer offset mechanism, while source connectors store offsets in a custom format within a Kafka topic.
- For sink connectors, the list shows Kafka topic partitions and the last committed offset for each partition.
- For source connectors, the list shows the source system’s partition and the last offset processed.
Prerequisites
- The Cluster Operator is running.
Procedure
Edit the
KafkaConnectorresource for the connector to include thelistOffsetsconfiguration.Example configuration to list offsets
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the command to write the list to the config map by annotating the
KafkaConnectorresource:oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=list -n <namespace>
oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=list -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The annotation remains until either the list operation succeeds or it is manually removed from the resource.
After the
KafkaConnectorresource is updated, use the following command to check if the config map with the offsets was created:oc get configmap my-connector-offsets -n <namespace>
oc get configmap my-connector-offsets -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Inspect the contents of the config map to verify the offsets are being listed:
oc describe configmap my-connector-offsets -n <namespace>
oc describe configmap my-connector-offsets -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Streams for Apache Kafka puts the offset information into the
offsets.jsonproperty. This does not overwrite any other properties when updating an existing config map.Example source connector offset list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The owner reference pointing to the
KafkaConnectorresource for the source connector. To provide a custom owner reference, create the config map in advance and set the owner reference. - 2
- The source partition, represented by the filename
/data/myfile.txtin this example for a file-based connector. - 3
- The last processed offset position in the source partition.
Example sink connector offset list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.8.5. Altering connector offsets
To alter connector offsets using KafkaConnector resources, configure the resource to stop the connector and add alterOffsets configuration to specify the offset changes in a config map. You can reuse the same config map used to list offsets.
After the connector is stopped and the configuration is in place, annotate the KafkaConnector resource to apply the offset alteration, then restart the connector.
Altering connector offsets can be useful, for example, to skip a poison record or replay a record.
In this procedure, we alter the offset position for a source connector named my-source-connector.
Prerequisites
- The Cluster Operator is running.
Procedure
Edit the
KafkaConnectorresource to stop the connector and include thealterOffsetsconfiguration.Example configuration to stop a connector and alter offsets
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the config map to make the alteration.
In this example, we’re resetting the offset position for a source connector to 15000.
Example source connector offset list configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the command to update the offset position by annotating the
KafkaConnectorresource:oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=alter -n <namespace>
oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=alter -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The annotation remains until either the update operation succeeds or it is manually removed from the resource.
- Check the changes by using the procedure to list connector offsets.
Restart the connector by changing the state to
running.Example configuration to start a connector
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.8.6. Resetting connector offsets
To reset connector offsets using KafkaConnector resources, configure the resource to stop the connector.
After the connector is stopped, annotate the KafkaConnector resource to clear the offsets, then restart the connector.
In this procedure, we reset the offset position for a source connector named my-source-connector.
Prerequisites
- The Cluster Operator is running.
Procedure
Edit the
KafkaConnectorresource to stop the connector.Example configuration to stop a connector
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Changes the state of the connector to
stopped. The default state for the connector when this property is not set isrunning.
Run the command to reset the offset position by annotating the
KafkaConnectorresource:oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=reset -n <namespace>
oc annotate kafkaconnector my-source-connector strimzi.io/connector-offsets=reset -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The annotation remains until either the reset operation succeeds or it is manually removed from the resource.
Check the changes by using the procedure to list connector offsets.
After resetting, the
offsets.jsonproperty is empty.Example source connector offset list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Restart the connector by changing the state to
running.Example configuration to start a connector
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.9. Configuring MirrorMaker 2
Configure your MirrorMaker 2 deployment by updating the spec properties of the KafkaMirrorMaker2 custom resource.
Start with a minimal configuration that defines the core requirements. Extend it with optional settings to support the following:
- Securing connections (TLS and authentication)
- Managing of replicated topic names
- Synchronizing consumer group offsets
- Synchronizing ACL rules
- Tuning Kafka Connect worker settings
-
common configuration
Common configuration includes setting resource limits and requests (recommended), JVM tuning, and metrics.
Certain settings, though optional, are recommended for production deployments, such as security and resource allocation.
For details of all configuration options, see the Streams for Apache Kafka Custom Resource API Reference.
10.9.1. Minimal configuration for MirrorMaker 2
A minimal KafkaMirrorMaker2 resource requires the following in its spec:
-
clusters: The connection details for source and target Kafka clusters. Each requires a unique alias. -
connectCluster: The alias of the target cluster. MirrorMaker 2 runs on the Kafka Connect framework, and its configuration and status are stored on this cluster. -
replicas: The number of Kafka Connect worker pods to deploy. -
mirrors: The replication flow from a source cluster to a target cluster.
By default, MirrorMaker 2 replicates all topics and consumer groups. To reduce load on the cluster and avoid replicating unnecessary data, specify which topics and groups to include using filters:
-
topicsPattern: A regex to select topics for replication. -
groupsPattern: A regex to select consumer groups for offset synchronization.
Minimal configuration for MirrorMaker 2
- 1
- Number of Kafka Connect worker replicas to run.
- 2
- Alias of the Kafka cluster used by Kafka Connect to store internal topics. This must be the alias of the target cluster.
- 3
- Each Kafka cluster (source and target) must define a unique alias and a bootstrap address. The bootstrap address takes the format
<cluster_name>-kafka-bootstrap:<port>. - 4
- Defines replication between the source and target clusters.
- 5
- Configuration for the source connector that replicates topics. At a minimum, you can add an empty
sourceConnectorproperty to use the default configuration. - 6
- Replicates only topics starting with
topic1ortopic2. - 7
- Replicates consumer groups starting with
group1orgroup2followed by two digits.
The Kafka clusters do not need to be managed by Streams for Apache Kafka or run on OpenShift.
10.9.2. Securing MirrorMaker 2 connections
This procedure describes in outline the configuration required to secure MirrorMaker 2 connections.
You configure these settings independently for the source Kafka cluster and the target Kafka cluster. You also need separate user configuration to provide the credentials required for MirrorMaker to connect to the source and target Kafka clusters.
For the Kafka clusters, you specify internal listeners for secure connections within an OpenShift cluster and external listeners for connections outside the OpenShift cluster.
You can configure authentication and authorization mechanisms. The security options implemented for the source and target Kafka clusters must be compatible with the security options implemented for MirrorMaker 2.
After you have created the cluster and user authentication credentials, you specify them in your MirrorMaker configuration for secure connections.
In this procedure, the certificates generated by the Cluster Operator are used, but you can replace them by installing your own certificates. You can also configure your listener to use a Kafka listener certificate managed by an external CA (certificate authority).
Before you start
Before starting this procedure, take a look at the example configuration files provided by Streams for Apache Kafka. They include examples for securing a deployment of MirrorMaker 2 using mTLS or SCRAM-SHA-512 authentication. The examples specify internal listeners for connecting within an OpenShift cluster.
The examples also provide the configuration for full authorization, including the ACLs that allow user operations on the source and target Kafka clusters.
When configuring user access to source and target Kafka clusters, ACLs must grant access rights to internal MirrorMaker 2 connectors and read/write access to the cluster group and internal topics used by the underlying Kafka Connect framework in the target cluster. If you’ve renamed the cluster group or internal topics, such as when configuring MirrorMaker 2 for multiple instances, use those names in the ACLs configuration.
Simple authorization uses ACL rules managed by the Kafka StandardAuthorizer plugin to ensure appropriate access levels. For more information on configuring a KafkaUser resource to use simple authorization, see the AclRule schema reference.
Prerequisites
- Streams for Apache Kafka is running
- Separate namespaces for source and target clusters
The procedure assumes that the source and target Kafka clusters are installed to separate namespaces. If you want to use the Topic Operator, you’ll need to do this. The Topic Operator only watches a single cluster in a specified namespace.
By separating the clusters into namespaces, you will need to copy the cluster secrets so they can be accessed outside the namespace. You need to reference the secrets in the MirrorMaker configuration.
Procedure
Configure two
Kafkaresources, one to secure the source Kafka cluster and one to secure the target Kafka cluster.You can add listener configuration for authentication and enable authorization.
In this example, an internal listener is configured for a Kafka cluster with TLS encryption and mTLS authentication. Kafka
simpleauthorization is enabled.Example source Kafka cluster configuration with TLS encryption and mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example target Kafka cluster configuration with TLS encryption and mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the
Kafkaresources in separate namespaces.oc apply -f <kafka_configuration_file> -n <namespace>
oc apply -f <kafka_configuration_file> -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The Cluster Operator creates the listeners and sets up the cluster and client certificate authority (CA) certificates to enable authentication within the Kafka cluster.
The certificates are created in the secret
<cluster_name>-cluster-ca-cert.Configure two
KafkaUserresources, one for a user of the source Kafka cluster and one for a user of the target Kafka cluster.-
Configure the same authentication and authorization types as the corresponding source and target Kafka cluster. For example, if you used
tlsauthentication and thesimpleauthorization type in theKafkaconfiguration for the source Kafka cluster, use the same in theKafkaUserconfiguration. - Configure the ACLs needed by MirrorMaker 2 to allow operations on the source and target Kafka clusters.
Example source user configuration for mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example target user configuration for mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteYou can use a certificate issued outside the User Operator by setting
typetotls-external. For more information, see theKafkaUserSpecschema reference.-
Configure the same authentication and authorization types as the corresponding source and target Kafka cluster. For example, if you used
Create or update a
KafkaUserresource in each of the namespaces you created for the source and target Kafka clusters.oc apply -f <kafka_user_configuration_file> -n <namespace>
oc apply -f <kafka_user_configuration_file> -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The User Operator creates the users representing the client (MirrorMaker), and the security credentials used for client authentication, based on the chosen authentication type.
The User Operator creates a new secret with the same name as the
KafkaUserresource. The secret contains a private and public key for mTLS authentication. The public key is contained in a user certificate, which is signed by the clients CA.Configure a
KafkaMirrorMaker2resource with the authentication details to connect to the source and target Kafka clusters.Example MirrorMaker 2 configuration with TLS encryption and mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The TLS certificates for the source Kafka cluster. If they are in a separate namespace, copy the cluster secrets from the namespace of the Kafka cluster.
- 2
- The user authentication for accessing the source Kafka cluster using the TLS mechanism. Supported authentication methods include
tls,scram-sha-256,scram-sha-512,plain, andoauth. - 3
- The TLS certificates for the target Kafka cluster.
- 4
- The user authentication for accessing the target Kafka cluster.
-
Apply the changes to the
KafkaMirrorMaker2resource to the same namespace as the target Kafka cluster.
10.9.3. Configuring replicated topic naming
By default, MirrorMaker 2 renames replicated topics by prepending the source cluster’s alias. For example, a topic named topic1 from a cluster called my-cluster-source is replicated as my-cluster-source.topic1. This allows MirrorMaker 2 to detect mirroring cycles and is especially useful when deploying complex topologies or performing bidirectional replication.
You can change this behavior using the replication.policy.class property in the connector configuration. There are two built-in policies available:
-
org.apache.kafka.connect.mirror.IdentityReplicationPolicykeeps original topic names.
This approach is suitable for unidirectional replication, migration, or failover scenarios. -
org.apache.kafka.connect.mirror.DefaultReplicationPolicy(default) prefixes topic names.
This is recommended for bidirectional replication. Usereplication.policy.separatorto specify the character that separates the cluster name from the topic name in the replicated topic.
Example configuration to keep topic names (IdentityReplicationPolicy)
- 1
- Replicates topics without prefixing the source cluster name.
Values for replication.policy.class and replication.policy.separator must be the same across all MirrorMaker 2 connectors (sourceConnector, heartbeatConnector, and checkpointConnector).
10.9.4. Synchronizing consumer group offsets
Configure MirrorMaker 2 to synchronize consumer group offsets from the source cluster to the target cluster.
The MirrorSourceConnector and MirrorCheckpointConnector work together using internal topics to coordinate offset tracking between clusters.
offset-syncstopic-
Format:
mm2-offset-syncs<separator><cluster_alias><separator>internal
Populated by theMirrorSourceConnector, this topic stores offset mappings between source and target clusters. By default, it’s created in the source cluster. checkpointstopic-
Format:
<cluster_alias><separator>checkpoints<separator>internal
Populated by theMirrorCheckpointConnectorin the target cluster, this topic captures the last committed offsets for each consumer group.
Configuring the MirrorCheckpointConnector to emit periodic offset checkpoints enables:
- Active/passive offset synchronization
- Failover recovery as consumers switch to the target cluster at the correct position
Offset synchronization occurs at regular intervals. The MirrorCheckpointConnector emits checkpoints for all consumer groups. However, applying those checkpoints in the target cluster requires the target group to be inactive. If consumers switch to the target cluster between checkpoints, they might reprocess some messages because the last synchronized offset lags behind the source. This duplication is expected.
The connector continues to emit checkpoints to the checkpoints topic even if the target consumer group is active. However, the offsets cannot be committed while the group has active members. In this case, the connector logs a warning and the offsets are not synchronized.
Example configuration to enable offset synchronization
- 1
- Replication factor for the
offset-syncsinternal topic that maps the offsets of the source and target clusters. A value of -1 uses the broker’s default replication factor, which is typically set to provide resilience (for example, 3). - 2
- Optional setting to change the frequency of checks for new topics.
- 3
- Enables consumer group offset synchronization.
- 4
- The frequency of the synchronization.
- 5
- Adjusts the frequency of checks for offset tracking. If you change the frequency of offset synchronization, you might also need to adjust the frequency of these checks.
- 6
- The frequency of checks for new consumer groups.
- 7
- Replication factor for the internal checkpoints topic that stores the last committed offsets for each consumer group. A value of -1 uses the broker’s default replication factor, which is typically set to provide resilience (for example, 3).
If you have an application written in Java, you can use the RemoteClusterUtils.java utility to fetch offsets through the application. The utility fetches remote offsets for a consumer group from the checkpoints topic.
The location of the offset-syncs topic is the source cluster by default. You can use the offset-syncs.topic.location connector configuration to change this to the target cluster. You need read/write access to the cluster that contains the topic. Using the target cluster as the location of the offset-syncs topic allows you to use MirrorMaker 2 even if you have only read access to the source cluster.
10.9.4.2. Listing the offsets of connectors
To list the offset positions of the internal MirrorMaker 2 connectors, use the same configuration that’s used to manage Kafka Connect connectors. For more information on setting up the configuration and listing offsets, see Section 10.8.4, “Listing connector offsets”.
In this example, the sourceConnector configuration is updated to return the connector offset position. The offset information is written to a specified ConfigMap.
Example configuration for MirrorMaker 2 connector
You must apply the following annotations to the KafkaMirrorMaker2 resource be able to manage connector offsets:
-
strimzi.io/connector-offsets -
strimzi.io/mirrormaker-connector
The strimzi.io/mirrormaker-connector annotation must be set to the name of the connector. These annotations remain until the operation succeeds or they are manually removed from the resource.
MirrorMaker 2 connectors are named using the aliases of the source and target clusters, followed by the connector type: <source_alias>-><target_alias>.<connector_type>.
In the following example, the annotations are applied for a connector named my-cluster-source->my-cluster-target.MirrorSourceConnector.
Example application of annotations for connector
oc annotate kafkamirrormaker2 my-mirror-maker-2 strimzi.io/connector-offsets=list strimzi.io/mirrormaker-connector="my-cluster-source->my-cluster-target.MirrorSourceConnector" -n kafka
oc annotate kafkamirrormaker2 my-mirror-maker-2 strimzi.io/connector-offsets=list strimzi.io/mirrormaker-connector="my-cluster-source->my-cluster-target.MirrorSourceConnector" -n kafka
The offsets are listed in the specified ConfigMap. Streams for Apache Kafka puts the offset information into a .json property named after the connector. This does not overwrite any other properties when updating an existing ConfigMap.
Example source connector offset list
- 1
- The owner reference pointing to the
KafkaMirrorMaker2resource. To provide a custom owner reference, create theConfigMapin advance and set the owner reference. - 2
- The
.jsonproperty uses the connector name. Since->characters are not allowed inConfigMapkeys,->is changed to--in the connector name.
10.9.5. Synchronizing ACL rules
MirrorMaker 2 can synchronize topic ACL rules from source to target clusters. Enable this feature by configuring the MirrorSourceConnector.
With simple authorization:
- ACL rules apply to both source and target topics
- Users with source topic access automatically get equivalent target topic access
-
This feature is not compatible with the User Operator, so disable by setting
sync.topic.acls.enabledtofalse. - When using OAuth 2.0 authorization, the setting has no effect.
Example configuration to enable offset synchronization
- 1
- Replicates all source topic ACLs to the target cluster.
10.9.6. Tuning Kafka Connect worker settings
Adjust Kafka Connect worker behavior for MirrorMaker 2 by adding configuration to your target cluster’s config section.
Example Kafka Connect configuration
By default, Streams for Apache Kafka configures the group ID and names of the internal topics used by the Kafka Connect framework that MirrorMaker 2 runs on. When running multiple instances of MirrorMaker 2, and they share the same connectCluster value, you must change these default settings using the following config properties:
Values for the three topics must be the same for all instances with the same group.id.
The connectCluster setting specifies the alias of the target Kafka cluster used by Kafka Connect for its internal topics. As a result, modifications to the connectCluster, group ID, and internal topic naming configuration are specific to the target Kafka cluster. You don’t need to make changes if two MirrorMaker 2 instances are using the same source Kafka cluster or in an active-active mode where each MirrorMaker 2 instance has a different connectCluster setting and target cluster.
However, if multiple MirrorMaker 2 instances share the same connectCluster, each instance connecting to the same target Kafka cluster is deployed with the same values. In practice, this means all instances form a cluster and use the same internal topics.
Multiple instances attempting to use the same internal topics will cause unexpected errors, so you must change the values of these properties for each instance.
MirrorMaker 2 can be configured for active/passive disaster recovery. To support this, the Kafka cluster should also be monitored for health and performance to detect issues that require failover promptly.
If failover occurs, which can be automated, operations switch from the active cluster to the passive cluster when the active cluster becomes unavailable. The original active cluster is typically considered permanently lost. The passive cluster is promoted to active status, taking over as the source for all application traffic. In this state, MirrorMaker 2 no longer replicates data from the original active cluster while it remains unavailable.
Failback, or restoring operations to the original active cluster, requires careful planning.
It is technically possible to reverse roles in MirrorMaker 2 by swapping the source and target clusters and deploying this configuration as a new instance. However, this approach risks data duplication, as records mirrored to the passive cluster may be mirrored back to the original active cluster. Avoiding duplicates requires resetting consumer offsets, which adds complexity. For a simpler and more reliable failback process, rebuild the original active cluster in a clean state and mirror data from the disaster recovery cluster.
Follow these best practices for disaster recovery in the event of failure of the active cluster in an active/passive configuration:
-
Promote the passive recovery cluster to an active role.
Designate the passive cluster as the active cluster for all client connections. This minimizes downtime and ensures operations can continue. -
Redirect applications to the new active recovery cluster.
MirrorMaker 2 synchronizes committed offsets to passive clusters, allowing consumer applications to resume from the last transferred offset when switching to the recovery cluster. However, because of the time lag in offset synchronization, switching consumers may result in some message duplication. To minimize duplication, switch all members of a consumer group together as soon as possible. Keeping the group intact minimizes the chance of a consumer processing duplicate messages. -
Remove the MirrorMaker 2 configuration for replication from the original active cluster to the passive cluster.
After failover, the original configuration is no longer needed and should be removed to avoid conflicts. - Re-create the failed cluster in a clean state, adhering to the original configuration.
-
Deploy a new MirrorMaker 2 instance to replicate data from the active recovery cluster to the rebuilt cluster.
Treat the rebuilt cluster as the passive cluster during this replication process. To prevent automatic renaming of topics, configure MirrorMaker 2 to use theIdentityReplicationPolicyby setting thereplication.policy.classproperty in the MirrorMaker 2 configuration. With this configuration applied, topics retain their original names in the target cluster. - Ensure the rebuilt cluster mirrors all data from the now-active recovery cluster.
- (Optional) Promote the rebuilt cluster back to active status by redirecting applications to the rebuilt cluster, after ensuring it is fully synchronized with the active cluster.
Before implementing any failover or failback processes, test your recovery approach in a controlled environment to minimize downtime and maintain data integrity.
10.10. Configuring MirrorMaker 2 connectors
MirrorMaker 2 uses internal connectors to synchronize data between Kafka clusters. Configure these connectors in your KafkaMirrorMaker2 custom resource.
MirrorMaker 2 consists of the following connectors:
MirrorSourceConnector- (Required) Replicates topics and ACLs from source to target cluster.
MirrorCheckpointConnector- (Optional) Synchronizes consumer group offsets for failover support. Only needed if offset tracking is required.
MirrorHeartbeatConnector- (Optional) Monitors connectivity between clusters. Useful for alerts but not required for replication.
The following table describes connector properties and the connectors you configure to use them.
| Property | sourceConnector | checkpointConnector | heartbeatConnector |
|---|---|---|---|
| ✓ | ✓ | ✓ |
| ✓ | ✓ | ✓ |
| ✓ | ✓ | ✓ |
| ✓ | ✓ | |
| ✓ | ✓ | |
| ✓ | ✓ | |
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ | ||
| ✓ |
The following properties must be identically configured across all three connectors (source, checkpoint, and heartbeat):
-
replication.policy.class -
replication.policy.separator -
offset-syncs.topic.location -
topic.filter.class
Mismatches cause replication failures or offset sync issues.
You can enable the optional MirrorHeartbeatConnector to monitor the replication stream between clusters. When enabled, this connector periodically creates heartbeat messages and sends them to a dedicated heartbeats topic in the source cluster.
These messages are then replicated by the MirrorSourceConnector to the target cluster. By consuming the heartbeats topic on the target cluster, you can confirm that data is flowing correctly from source to target. If you stop seeing new heartbeats on the target, it indicates a potential issue with the replication process.
Because the MirrorHeartbeatConnector writes heartbeat messages to the source cluster, it usually requires a different Connect cluster from the one that runs the MirrorSourceConnector and MirrorCheckpointConnector.
Before enabling the heartbeat connector in the KafkaMirrorMaker2 resource, consider the advantages and disadvantages of running and maintaining an additional Kafka Connect cluster. If you already operate a Kafka Connect cluster that stores state in the source Kafka cluster, you can deploy the MirrorHeartbeatConnector there instead of using the KafkaMirrorMaker2 resource.
Example configuration for the heartbeat connector defined in a KafkaMirrorMaker2 resource
- 1
- Alias of the Kafka Connect cluster where the connectors run. Must match the alias given in
targetCluster. - 2
- Bootstrap address of the source cluster.
- 3
- Bootstrap address of the target cluster. Use the same alias as the target cluster, but specify the bootstrap servers of the source cluster, because the heartbeat topic is created in the source.
- 4
- Alias of the source cluster. Must match the source alias defined for the
MirrorSourceConnector. - 5
- Alias of the target cluster. Must match the target alias defined for the
MirrorSourceConnector. - 6
- Configuration for the
MirrorHeartbeatConnectorthat performs connectivity checks. Overrides the default configuration options. - 7
- Replication factor for the heartbeat topic.
- 8
- Interval, in seconds, at which heartbeat messages are sent.
The source and target cluster aliases in the MirrorHeartbeatConnector configuration must match the aliases defined for the MirrorSourceConnector in the KafkaMirrorMaker2 resource.
Use tasksMax to control how many tasks are assigned to connectors. Increasing the number of tasks helps improve performance when replicating many partitions or synchronizing the offsets of a large number of consumer groups.
MirrorMaker 2 connectors create distributed tasks that move data between clusters. These tasks follow a set of operational behaviors and allocation rules.
Task behavior:
- Each task runs on a single worker pod.
- Tasks run in parallel.
- A worker pod can run multiple tasks, but each task runs in isolation.
- You don’t need more pods than tasks. If there are fewer pods, tasks are distributed across available workers.
Task allocation rules:
-
By default, connectors run with 1 task, unless
tasksMaxis set. -
The
MirrorHeartbeatConnectoralways uses 1 task. -
For the
MirrorSourceConnector, the maximum possible tasks = number of partitions. -
For the
MirrorCheckpointConnector, the maximum possible tasks = number of consumer groups. -
Actual tasks started is the lower value of
tasksMaxand the maximum possible tasks.
If the infrastructure can support it, increasing the number of tasks improves throughput and reduces latency during high-volume replication.
Increasing the number of tasks for the source connector
Enable automatic restarts of failed connectors and tasks using autoRestart. By default, the number of restarts is indefinite, but you can set a maximum on the number of automatic restarts using the maxRestarts property.
Increasing the number of tasks for the checkpoint connector is useful when you have a large number of consumer groups.
Increasing the number of tasks for the checkpoint connector
By default, MirrorMaker 2 checks for new consumer groups every 10 minutes. You can adjust the refresh.groups.interval.seconds configuration to change the frequency. Take care when adjusting lower. More frequent checks can have a negative impact on performance.
10.10.2.1. Checking connector task operations
If you are using Prometheus and Grafana to monitor your deployment, you can check MirrorMaker 2 performance. The example MirrorMaker 2 Grafana dashboard provided with Streams for Apache Kafka shows the following metrics related to tasks and latency.
- The number of tasks
- Replication latency
- Offset synchronization latency
If you are using KafkaMirrorMaker2 resources to configure internal MirrorMaker connectors, use the state configuration to either stop or pause a connector. In contrast to the paused state, where the connector and tasks remain instantiated, stopping a connector retains only the configuration, with no active processes. Stopping a connector from running may be more suitable for longer durations than just pausing. While a paused connector is quicker to resume, a stopped connector has the advantages of freeing up memory and resources.
The state configuration replaces the (deprecated) pause configuration in the KafkaMirrorMaker2ConnectorSpec schema, which allows pauses on connectors. If you were previously using the pause configuration to pause connectors, we encourage you to transition to using the state configuration only to avoid conflicts.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaMirrorMaker2custom resource that controls the MirrorMaker 2 connector you want to pause or stop:oc get KafkaMirrorMaker2
oc get KafkaMirrorMaker2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
KafkaMirrorMaker2resource to stop or pause the connector.Example configuration for stopping a MirrorMaker 2 connector
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change the
stateconfiguration tostoppedorpaused. The default state for the connector when this property is not set isrunning.Apply the changes to the
KafkaMirrorMaker2configuration.You can resume the connector by changing
statetorunningor removing the configuration.
Alternatively, you can expose the Kafka Connect API and use the stop and pause endpoints to stop a connector from running. For example, PUT /connectors/<connector_name>/stop. You can then use the resume endpoint to restart it.
10.10.4. Manually restarting MirrorMaker 2 connectors
Use the strimzi.io/restart-connector annotation to manually trigger a restart of a MirrorMaker 2 connector.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaMirrorMaker2custom resource that controls the Kafka MirrorMaker 2 connector you want to restart:oc get KafkaMirrorMaker2
oc get KafkaMirrorMaker2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Find the name of the Kafka MirrorMaker 2 connector to be restarted from the
KafkaMirrorMaker2custom resource:oc describe KafkaMirrorMaker2 <mirrormaker_cluster_name>
oc describe KafkaMirrorMaker2 <mirrormaker_cluster_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Use the name of the connector to restart the connector by annotating the
KafkaMirrorMaker2resource in OpenShift:oc annotate KafkaMirrorMaker2 <mirrormaker_cluster_name> "strimzi.io/restart-connector=<mirrormaker_connector_name>"
oc annotate KafkaMirrorMaker2 <mirrormaker_cluster_name> "strimzi.io/restart-connector=<mirrormaker_connector_name>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, connector
source-cluster→target-cluster.MirrorCheckpointConnectorin themy-mirror-maker-2cluster is restarted:oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector=source-cluster->target-cluster.MirrorCheckpointConnector"
oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector=source-cluster->target-cluster.MirrorCheckpointConnector"Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can also refine the restart behavior with the
includeTasksandonlyFailedparameters, which both default tofalse.-
includeTasksrestarts both the connector instance and its task instances. -
onlyFailedrestarts only instances with aFAILEDstatus when set totrue.
Separate the connector name and the parameters with a colon (
:). For example:oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector=source-cluster->target-cluster.MirrorCheckpointConnector:includeTasks,onlyFailed"
oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector=source-cluster->target-cluster.MirrorCheckpointConnector:includeTasks,onlyFailed"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Wait for the next reconciliation to occur (every two minutes by default).
The MirrorMaker 2 connector is restarted, as long as the annotation was detected by the reconciliation process. When MirrorMaker 2 accepts the request, the annotation is removed from the
KafkaMirrorMaker2custom resource.
Use the strimzi.io/restart-connector-task annotation to manually trigger a restart of a MirrorMaker 2 connector.
Prerequisites
- The Cluster Operator is running.
Procedure
Find the name of the
KafkaMirrorMaker2custom resource that controls the MirrorMaker 2 connector task you want to restart:oc get KafkaMirrorMaker2
oc get KafkaMirrorMaker2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Find the name of the connector and the ID of the task to be restarted from the
KafkaMirrorMaker2custom resource:oc describe KafkaMirrorMaker2 <mirrormaker_cluster_name>
oc describe KafkaMirrorMaker2 <mirrormaker_cluster_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Task IDs are non-negative integers, starting from 0.
Use the name and ID to restart the connector task by annotating the
KafkaMirrorMaker2resource in OpenShift:oc annotate KafkaMirrorMaker2 <mirrormaker_cluster_name> "strimzi.io/restart-connector-task=<mirrormaker_connector_name>:<task_id>"
oc annotate KafkaMirrorMaker2 <mirrormaker_cluster_name> "strimzi.io/restart-connector-task=<mirrormaker_connector_name>:<task_id>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, task
0for connectorsource-cluster→target-cluster.MirrorSourceConnectorin themy-mirror-maker-2cluster is restarted:oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector-task=source-cluster->target-cluster.MirrorSourceConnector:2"
oc annotate KafkaMirrorMaker2 my-mirror-maker-2 "strimzi.io/restart-connector-task=source-cluster->target-cluster.MirrorSourceConnector:2"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the next reconciliation to occur (every two minutes by default).
The MirrorMaker 2 connector task is restarted, as long as the annotation was detected by the reconciliation process. When MirrorMaker 2 accepts the request, the annotation is removed from the
KafkaMirrorMaker2custom resource.
MirrorMaker 2 connectors use internal producers and consumers. If needed, you can configure these producers and consumers to override the default settings.
For example, you can increase the batch.size for the source producer that sends topics to the target Kafka cluster to better accommodate large volumes of messages.
Producer and consumer configuration options depend on the MirrorMaker 2 implementation, and may be subject to change.
The following tables describe the producers and consumers for each of the connectors and where you can add configuration.
| Type | Description | Configuration |
|---|---|---|
| Producer | Sends topic messages to the target Kafka cluster. Consider tuning the configuration of this producer when it is handling large volumes of data. |
|
| Producer |
Writes to the |
|
| Consumer | Retrieves topic messages from the source Kafka cluster. |
|
| Type | Description | Configuration |
|---|---|---|
| Producer | Emits consumer offset checkpoints. |
|
| Consumer |
Loads the |
|
You can set offset-syncs.topic.location to target to use the target Kafka cluster as the location of the offset-syncs topic.
| Type | Description | Configuration |
|---|---|---|
| Producer | Emits heartbeats. |
|
The following example shows how you configure the producers and consumers.
Example configuration for connector producers and consumers
10.11. Configuring the Kafka Bridge
Update the spec properties of the KafkaBridge custom resource to configure your Kafka Bridge deployment.
In order to prevent issues arising when client consumer requests are processed by different Kafka Bridge instances, address-based routing must be employed to ensure that requests are routed to the right Kafka Bridge instance. Additionally, each independent Kafka Bridge instance must have a replica. A Kafka Bridge instance has its own state which is not shared with another instances.
For a deeper understanding of the Kafka Bridge and its cluster configuration options, refer to the Using the Kafka Bridge guide and the Streams for Apache Kafka Custom Resource API Reference.
Example KafkaBridge custom resource configuration
- 1
- The number of replica nodes.
- 2
- Bootstrap address for connection to the target Kafka cluster. The address takes the format
<cluster_name>-kafka-bootstrap:<port_number>. The Kafka cluster doesn’t need to be managed by Streams for Apache Kafka or deployed to an OpenShift cluster. - 3
- HTTP access to Kafka brokers.
- 4
- CORS access specifying selected resources and access methods. Additional HTTP headers in requests describe the origins that are permitted access to the Kafka cluster.
- 5
- Requests for reservation of supported resources, currently
cpuandmemory, and limits to specify the maximum resources that can be consumed. - 6
- TLS configuration for encrypted connections to the Kafka cluster, with trusted certificates stored in X.509 format within the specified secrets.
- 7
- Authentication for the Kafka Bridge cluster, specified as
tls,scram-sha-256,scram-sha-512,plain, oroauth. By default, the Kafka Bridge connects to Kafka brokers without authentication. For details on configuring authentication, see theKafkaBridgeSpecschema properties - 8
- Consumer configuration options.
- 9
- Producer configuration options.
- 10
- Kafka Bridge loggers and log levels added directly (
inline) or indirectly (external) through aConfigMap. Custom Log4j configuration must be placed under thelog4j2.propertieskey in theConfigMap. You can set log levels toINFO,ERROR,WARN,TRACE,DEBUG,FATALorOFF. - 11
- JVM configuration options to optimize performance for the Virtual Machine (VM) running the Kafka Bridge.
- 12
- Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).
- 13
- Optional: Container image configuration, which is recommended only in special situations.
- 14
- Template customization. Here a pod is scheduled with anti-affinity, so the pod is not scheduled on nodes with the same hostname.
- 15
- Environment variables are set for distributed tracing.
- 16
- Distributed tracing is enabled by using OpenTelemetry.
- 17
- Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter.
10.12. Applying optional common configuration
You can further configure Streams for Apache Kafka components by applying any of the following optional common configuration settings. Common configuration is configured independently for each component, such as the following:
- Resource limits and requests (Recommended)
- Metrics configuration
- Liveness and readiness probes
- JVM options for maximum and minimum memory allocation
- Adding additional volumes and volume mounts
- Template configuration for pods and containers
- Logging frequency
Advanced or specialized options include:
- Custom container images
- Rack awareness
- Distributed tracing
Configure common options for Streams for Apache Kafka custom resources in the .spec section of the custom resource. For more information on these configuration options, refer to Common configuration properties.
10.12.1. Resource limits and requests (recommended)
To ensure stability and optimal performance for your Kafka clusters, we recommend defining CPU and memory resource limits and requests for all Streams for Apache Kafka containers. By default, the Streams for Apache Kafka Cluster Operator does not set these values, but fine-tuning them based on your workload requirements helps performance and improves reliability.
Example resource configuration
10.12.2. Metrics configuration
Enable metrics collection for monitoring.
Example metrics configuration
Configuration varies depending on the component and exporter used: Prometheus JMX Exporter or Streams for Apache Kafka Metrics Reporter. For more information, see Introducing metrics.
10.12.3. Liveness and readiness probes
Configure health checks for the container.
Example liveness and readiness probes
10.12.4. JVM options
Configure the Java Virtual Machine (JVM) for the component. To enable garbage collector (GC) logging, set gcLoggingEnabled to true.
Example JVM options
10.12.5. Additional volumes and mounts
Add extra volumes to the container and mount them in specific locations.
Example additional volumes
You can use template configuration to add other customizations to pods and containers, such as affinity and security context. For more information, see Configuring pod scheduling and Applying security context to Streams for Apache Kafka pods and containers.
10.12.6. Custom container image
Override the default container image. Use only in special situations.
Example custom image
# ... spec: image: my-org/custom-kafka-image:latest # ...
# ...
spec:
image: my-org/custom-kafka-image:latest
# ...10.12.7. Rack awareness
Enable rack-aware broker assignment to improve fault tolerance. This is a specialized option intended for a deployment within the same location, not across regions.
Example rack awareness configuration
# ...
spec:
rack:
topologyKey: topology.kubernetes.io/zone
# ...
# ...
spec:
rack:
topologyKey: topology.kubernetes.io/zone
# ...10.12.8. Distributed tracing configuration
Enable distributed tracing using OpenTelemetry to monitor Kafka component operations.
Example tracing configuration
# ...
spec:
tracing:
type: opentelemetry
# ...
# ...
spec:
tracing:
type: opentelemetry
# ...For more information see Introducing distributed tracing.
10.12.9. Configuring logging levels
Streams for Apache Kafka operators and Kafka components use log4j2 for logging. However, Kafka 3.9 and earlier versions rely on log4j1. For log4j1-based configuration examples, refer to the Streams for Apache Kafka 2.9 documentation.
You can set log levels to INFO, ERROR, WARN, TRACE, DEBUG, FATAL or OFF. Configure the logging levels of Kafka components and Streams for Apache Kafka operators through their custom resources. You can use either of these options:
-
Specify logging levels directly in the
spec.loggingproperty of the custom resource. -
Define logging properties in a custom
ConfigMapand reference it using theconfigMapKeyRefproperty.
Advantages of using a ConfigMap:
- Centralized maintenance
- Reusable with multiple resources
- Flexibility to append logging specifications to add filters
Specify a logging type in your logging specification:
-
inlinewhen specifying logging levels directly -
externalwhen referencing aConfigMap
For inline configuration, use the loggers property to set the root logger level and levels for specific classes or loggers.
Example inline logging configuration
For external configuration, use a ConfigMap to define logging configurations using a full log4j2.properties file. Set the logging name and key properties to reference the ConfigMap. Both properties are mandatory.
Example external logging configuration
Default logging is used if logging is not specified in the resource using either method. Loggers that haven’t been explicitly configured inherit settings from their parent loggers.
When a resource managed by the Cluster Operator is created, a ConfigMap with the specified logging configuration is also created. For components managed by the Streams for Apache Kafka operators, changes to logging levels are applied dynamically.
Setting a log level to DEBUG or TRACE may result in a large amount of log output and may have performance implications.
10.12.9.1. Configurable loggers
The following Kafka components and operators have specific loggers available for configuration:
For information about log levels, see Apache logging services.
10.12.9.2. Creating a ConfigMap for logging
To use a ConfigMap to define logging properties, you create and then reference it as part of the logging definition in the spec of a resource. Place the configuration under log4j2.properties
In this procedure a ConfigMap defines a root logger for a Kafka resource.
Procedure
Create the
ConfigMapas a YAML file or from a properties file.Example ConfigMap for logging
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you are using a properties file, define the logging configuration and specify the file at the command line when creating the
ConfigMap.Properties file definition
# Define the logger rootLogger.level = "INFO" # ...
# Define the logger rootLogger.level = "INFO" # ...Copy to Clipboard Copied! Toggle word wrap Toggle overflow Specifying the properties file
oc create configmap logging-configmap --from-file=log4j2.properties
oc create configmap logging-configmap --from-file=log4j2.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Add
externallogging to thespecof theKafkaresource, specifying thenameandkeyof theConfigMap:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Kafkaconfiguration.
10.12.9.3. Configuring Cluster Operator logging
Cluster Operator logging is configured through a ConfigMap named strimzi-cluster-operator. This ConfigMap, created with default values during installation, is described in the file install/cluster-operator/050-ConfigMap-strimzi-cluster-operator.yaml. Configure Cluster Operator logging by changing data.log4j2.properties values in the file.
To update the logging configuration, edit the 050-ConfigMap-strimzi-cluster-operator.yaml file and then run the following command:
oc create -f install/cluster-operator/050-ConfigMap-strimzi-cluster-operator.yaml
oc create -f install/cluster-operator/050-ConfigMap-strimzi-cluster-operator.yaml
Alternatively, edit the ConfigMap directly:
oc edit configmap strimzi-cluster-operator
oc edit configmap strimzi-cluster-operator
With this ConfigMap, you can control the following aspects of logging:
- Root logger level
- Log output format
- Log levels for different components
-
Kafka
AdminClientlogging levels - Netty logging Levels
- How often logging configuration is loaded
Netty is a framework used in Streams for Apache Kafka for network communication. The monitorInterval setting determines how often in seconds the logging configuration is dynamically reloaded. The default is 30 seconds.
If the ConfigMap is missing when the Cluster Operator is deployed, the default logging values are used.
If the ConfigMap is accidentally deleted after the Cluster Operator is deployed, the most recently loaded logging configuration is used. Create a new ConfigMap to load a new logging configuration.
Do not remove the monitorInterval option from the ConfigMap.
Add logging filters to Streams for Apache Kafka operators by using a ConfigMap that contains a complete log4j2.properties file with your custom log4j2 configuration.
Filters are useful when too many logging messages are being produced. For instance, if rootLogger.level="DEBUG", filters reduce the number of logs to focus on a specific resource by logging only messages matching the filter.
Markers specify what to include in the log using kind, namespace, and resource name values. For example, to isolate the logs of a failing Kafka cluster, set the kind to Kafka and use the namespace and name of the cluster.
Basic logging filter configuration
For multiple filters, define each one separately:
Multiple logging filter configuration
Adding filters to the Cluster Operator
To add filters to the Cluster Operator, update the ConfigMap YAML file (install/cluster-operator/050-ConfigMap-strimzi-cluster-operator.yaml):
Procedure
Update the
050-ConfigMap-strimzi-cluster-operator.yamlfile to add the filter properties:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Alternatively, edit the
ConfigMapdirectly:oc edit configmap strimzi-cluster-operator
oc edit configmap strimzi-cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
If updating the YAML file, apply the changes to the
ConfigMapconfiguration.
Adding filters to the Topic or User Operator
To add filters to the Topic or User Operator, create or edit a logging ConfigMap. The same method applies for both operators.
Procedure
Create the
ConfigMapas a YAML file or from a properties file.Example filter properties for
my-topictopicCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you are using a properties file, define the logging configuration and specify the file at the command line when creating the
ConfigMap.Properties file definition
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Specifying the properties file
oc create configmap logging-configmap --from-file=log4j2.properties
oc create configmap logging-configmap --from-file=log4j2.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Define
externallogging in thetopicOperatororuserOperatorconfiguration of theKafkaresource, specifying thenameandkeyof the ConfigMap:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Kafkaconfiguration.
10.12.10. Configuring pod scheduling
To optimize the resilience and performance of your Kafka cluster, you can control how its pods are scheduled across OpenShift nodes. Pod scheduling strategies can help you to achieve the following:
- Increase fault tolerance by spreading Kafka pods across different nodes.
- Avoid resource contention by separating pods from critical workloads.
- Maintain resource availability by assigning Kafka pods to nodes with sufficient capacity.
10.12.10.1. Scheduling strategies
Kafka components can be scheduled onto OpenShift nodes using affinity rules, tolerations, and topology constraints. These strategies help isolate workloads, optimize resource usage, and improve overall cluster performance.
The following scheduling techniques support different deployment goals:
- Use pod anti-affinity to avoid critical applications sharing nodes
- Use pod anti-affinity to prevent critical applications from being scheduled on the same disk. In Kafka deployments, configure pod anti-affinity to ensure that Kafka brokers do not share nodes with other workloads, such as databases.
- Use node affinity to schedule workloads onto specific nodes
-
OpenShift clusters often include nodes optimized for different workloads, such as CPU, memory, storage, or network. Node affinity enables scheduling Kafka components onto nodes that match specific labels, such as
node.kubernetes.io/instance-typeor custom labels, to optimize performance and cost. - Use node affinity, taints, and tolerations for dedicated nodes
To reserve nodes for Kafka, you can taint them to exclude them from general workloads. Kafka pods can still be scheduled onto these nodes by configuring:
- Tolerations, which allow the pods to bypass the taint.
- Node affinity, for the pods to run on those specific nodes.
These settings direct Kafka pods to dedicated nodes while preventing other workloads from being scheduled there This approach isolates Kafka from other workloads, reducing resource contention and improving stability.
- Use topology spread constraints to balance pods across zones or nodes
- Topology spread constraints help distribute pods evenly across specified topology domains, such as zones, regions, or nodes. For Kafka, this strategy reduces the risk of scheduling too many brokers on the same zone or node, improving availability and resilience.
You can apply these scheduling strategies in the template.pod property within the spec of a Streams for Apache Kafka custom resource.
For Kafka brokers specifically, Streams for Apache Kafka provides a two-level configuration:
-
Kafka.spec.kafka.template.pod
Use this to set a cluster-wide default scheduling policy for all Kafka broker and controller pods. -
KafkaNodePool.spec.template.pod
For more granular control, if needed, you can override the default policy for specific node pools using theKafkaNodePoolresource.
If any property is defined under KafkaNodePool.spec.template, all Kafka.spec.kafka.template settings are ignored for pods in that node pool. Properties are not merged. In other words, node pools using KafkaNodePool.spec.template settings do not inherit any settings from the cluster-wide template. Every required setting must be specified in the pool’s template.
Other Streams for Apache Kafka components have their own template properties for scheduling:
-
Kafka.spec.kafka.template.pod -
KafkaNodePool.spec.template.pod -
Kafka.spec.entityOperator.template.pod -
Kafka.spec.cruiseControl.template.pod -
KafkaConnect.spec.template.pod -
KafkaBridge.spec.template.pod -
KafkaMirrorMaker2.spec.template.pod
The following procedures provide scheduling configuration examples for the Kafka and KafkaNodePool resources. The same configurations shown in these examples can be applied to the template.pod property of any other component.
Scheduling properties follow the OpenShift specification.
Additional resources
- Kubernetes node and pod affinity documentation
- Kubernetes taints and tolerations
- Controlling pod placement by using pod topology spread constraints (as described in the Openshift Nodes guide)
To improve fault tolerance, you can prevent multiple Kafka broker pods from running on the same worker node. Configure podAntiAffinity so that the pods are scheduled on different worker nodes.
This procedure provides configuration examples for the cluster-wide Kafka resource and the pool-specific KafkaNodePool resource.
If you configure KafkaNodePool.spec.template, its settings replace Kafka.spec.kafka.template for that node pool. Properties are not merged. For more information, see Scheduling strategies.
Prerequisites
Procedure
Configure
podAntiAffinityin either theKafkaorKafkaNodePoolresource.-
To set a cluster-wide rule, edit the
affinityproperty inspec.kafka.template.podof yourKafkaresource. Use thestrimzi.io/namelabel to select all broker pods. -
To set a pool-specific rule, edit the
affinityproperty inspec.template.podof yourKafkaNodePoolresource. Use thestrimzi.io/pool-namelabel to select only the pods in that pool.
In both cases, set the
topologyKeyto"kubernetes.io/hostname"to prevent pods from being placed on the same host.This example applies a rule to a
Kafkaresource namedmy-clusterthat prevents any of its broker pods from running on the same node.Example cluster-wide anti-affinity configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This example applies a rule to a
KafkaNodePoolresource namedbrokerthat prevents pods from that specific pool from running on the same node.Example node pool-specific anti-affinity configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
To set a cluster-wide rule, edit the
- Apply the changes to your custom resource configuration.
To improve stability and performance, you can prevent Kafka pods from running on the same worker nodes as other resource-intensive applications, such as databases. Configure podAntiAffinity so that these workloads are scheduled on separate nodes.
This procedure provides configuration examples for the cluster-wide Kafka resource and the pool-specific KafkaNodePool resource.
If you configure KafkaNodePool.spec.template, its settings replace Kafka.spec.kafka.template for that node pool. Properties are not merged. For more information, see Scheduling strategies.
Prerequisites
- The Cluster Operator must be deployed.
- The other workloads in your cluster use consistent labels.
Procedure
Configure
podAntiAffinityin either theKafkaorKafkaNodePoolresource.-
To set a cluster-wide rule, edit the
affinityproperty inspec.kafka.template.podof yourKafkaresource. -
To set a pool-specific rule, edit the
affinityproperty inspec.template.podof yourKafkaNodePoolresource.
In both cases, use a
labelSelectorto identify the application pods you want to keep separate from your Kafka pods and set thetopologyKeyto"kubernetes.io/hostname"to prevent pods from being placed on the same host.This example applies a rule to a
Kafkaresource namedmy-clusterthat prevents any of its broker pods from running on the same node as pods labeledpostgresqlandmongodb.Example cluster-wide anti-affinity configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This example applies a rule to a
KafkaNodePoolresource namedbrokerthat prevents pods from that pool from running on the same node as pods labeledpostgresqlandmongodb.Example node pool-specific anti-affinity configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
To set a cluster-wide rule, edit the
- Apply the changes to your custom resource configuration.
Your OpenShift cluster might have different types of worker nodes, some with specialized hardware (fast SSDs, powerful CPUs) or a particular location (a specific availability zone). To optimize performance and resource usage, configure nodeAffinity so that Kafka pods are scheduled only on the nodes that match your requirements.
This procedure provides configuration examples for the cluster-wide Kafka resource and the pool-specific KafkaNodePool resource.
If you configure KafkaNodePool.spec.template, its settings replace Kafka.spec.kafka.template for that node pool. Properties are not merged. For more information, see Scheduling strategies.
Prerequisites
- The Cluster Operator must be deployed.
-
Worker nodes are labeled to identify their specific characteristics (such as
disk:sdd).
Procedure
Label the worker nodes to identify them when scheduling the kafka pods.
oc label node <name_of_node> disk=sdd
oc label node <name_of_node> disk=sddCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure
nodeAffinityin either theKafkaorKafkaNodePoolresource to match the label.-
To set a cluster-wide rule, edit the
affinityproperty inspec.kafka.template.podof yourKafkaresource. -
To set a pool-specific rule, edit the
affinityproperty inspec.template.podof yourKafkaNodePoolresource.
In both cases, use
nodeSelectorTermswithmatchExpressionsto specify the key-value label of the nodes you want to schedule pods on.This example applies a rule to a
Kafkaresource that assigns all its broker pods to run only on nodes with the labeldisk: ssd.Example cluster-wide affinity configuration for specific nodes
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example availability zone scheduling for node pools
For node pools, a common production scenario is to configure node pools so that pods are scheduled only on nodes within a specified availability zone. For more information, see Managing storage affinity using node pools
-
To set a cluster-wide rule, edit the
- Apply the changes to your custom resource configuration.
You can dedicate a set of worker nodes exclusively to your Kafka brokers so that no other applications can compete with Kafka for resources on those nodes.
To configure dedicated worker nodes for Kafka pods in a specific pool, combine the following:
- Taints
- Apply taints to worker nodes to prevent other pods from being scheduled on them.
- Tolerations
- Apply tolerations to Kafka pods to allow them to be scheduled on tainted nodes.
- Affinity
- Apply affinity to Kafka pods to schedule them on specifically labeled nodes.
This procedure provides configuration examples for the cluster-wide Kafka resource and the pool-specific KafkaNodePool resource.
If you configure KafkaNodePool.spec.template, its settings replace Kafka.spec.kafka.template for that node pool. Properties are not merged. For more information, see Scheduling strategies.
Prerequisites
- The Cluster Operator must be deployed.
- Dedicated worker nodes without scheduled workloads
Procedure
Taint and label the dedicated worker nodes to prevent other pods from being scheduled on them and to identify them when scheduling the Kafka pods:
oc adm taint node <name_of_node> dedicated=kafka:NoSchedule oc label node <name_of_node> dedicated=kafka
oc adm taint node <name_of_node> dedicated=kafka:NoSchedule oc label node <name_of_node> dedicated=kafkaCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure
tolerationsandnodeAffinityin either yourKafkaorKafkaNodePoolcustom resource to match the taint and label.-
To set a cluster-wide rule, edit the
affinityproperty inspec.kafka.template.podof yourKafkaresource. -
To set a pool-specific rule, edit the
affinityproperty inspec.template.podof yourKafkaNodePoolresource.
In both cases, use
nodeSelectorTermswithmatchExpressionsto specify the key-value label of the nodes you want to schedule pods on.This example applies a rule to a
Kafkaresource that assigns all its broker pods to run only on nodes that have been tainted and labeled withdedicated=kafka.Example cluster-wide affinity configuration for dedicated nodes
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This example applies a rule to a
KafkaNodePoolresource namedbrokerthat assigns its pods to run on dedicated nodes marked withdedicated=broker-kafka.Example node pool-specific affinity configuration for dedicated nodes
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
To set a cluster-wide rule, edit the
- Apply the changes to your custom resource configuration.
10.12.11. Disabling PodDisruptionBudget generation
Streams for Apache Kafka automatically generates a PodDisruptionBudget resource for each of the following components:
- Kafka cluster
- Kafka Connect
- MirrorMaker 2
- Kafka Bridge
Each budget applies across all pods deployed for that component.
The Kafka cluster’s PodDisruptionBudget covers all associated node pool pods.
To disable automatic PodDisruptionBudget generation, set the STRIMZI_POD_DISRUPTION_BUDGET_GENERATION environment variable to false in the Cluster Operator configuration. You can then define custom PodDisruptionBudget resources if needed. For more information, see Section 10.6, “Configuring the Cluster Operator”.
10.12.12. Using ConfigMap resources to add configuration
Add specific configuration to your Streams for Apache Kafka deployment using ConfigMap resources. Config maps use key-value pairs to store non-confidential data. Configuration data added to config maps is maintained in one place and can be reused amongst components.
Config maps can only store the following types of configuration data:
- Logging configuration
- Metrics configuration
- External configuration for Kafka Connect connectors
You can’t use config maps for other areas of configuration.
When you configure a component, you can add a reference to a ConfigMap using the configMapKeyRef property.
For example, you can use configMapKeyRef to reference a ConfigMap that provides configuration for logging. You define logging levels using the log4j2.properties key in the ConfigMap and then reference it in the logging configuration of the resource.
Example reference to a ConfigMap
To use a ConfigMap for metrics configuration, you add a reference to the metricsConfig configuration of the component in the same way.
template properties allow data from a ConfigMap or Secret to be mounted in a pod as environment variables or volumes. You can use external configuration data for the connectors used by Kafka Connect. The data might be related to an external data source, providing the values needed for the connector to communicate with that data source.
For example, you can use the configMapKeyRef property to pass configuration data from a ConfigMap as an environment variable.
Example ConfigMap providing environment variable values
If you are using config maps that are managed externally, use configuration providers to load the data in the config maps.
10.12.12.1. Naming custom config maps
Streams for Apache Kafka creates its own config maps and other resources when it is deployed to OpenShift. The config maps contain data necessary for running components. The config maps created by Streams for Apache Kafka must not be edited.
Make sure that any custom config maps you create do not have the same name as these default config maps. If they have the same name, they are overwritten. For example, if the custom ConfigMap has the same name as the ConfigMap for the Kafka cluster, it is overwritten when there is an update to the Kafka cluster.
Use configuration providers to load configuration data from external sources. The providers operate independently of Streams for Apache Kafka. You can use them to load configuration data for all Kafka components, including producers and consumers. You reference the external source in the configuration of the component and provide access rights. The provider loads data without needing to restart the Kafka component or extracting files, even when referencing a new external source. For example, use providers to supply the credentials for the Kafka Connect connector configuration. The configuration must include any access rights to the external source.
10.12.13.1. Enabling configuration providers
You can enable one or more configuration providers using the config.providers properties in the spec configuration of a component.
Example configuration to enable a configuration provider
- KubernetesSecretConfigProvider
- Loads configuration data from OpenShift secrets. You specify the name of the secret and the key within the secret where the configuration data is stored. This provider is useful for storing sensitive configuration data like passwords or other user credentials.
- KubernetesConfigMapConfigProvider
- Loads configuration data from OpenShift config maps. You specify the name of the config map and the key within the config map where the configuration data is stored. This provider is useful for storing non-sensitive configuration data.
- EnvVarConfigProvider
- Loads configuration data from environment variables. You specify the name of the environment variable where the configuration data is stored. This provider is useful for configuring applications running in containers, for example, to load certificates or JAAS configuration from environment variables mapped from secrets.
- FileConfigProvider
- Loads configuration data from a file. You specify the path to the file where the configuration data is stored. This provider is useful for loading configuration data from files that are mounted into containers.
- DirectoryConfigProvider
- Loads configuration data from files within a directory. You specify the path to the directory where the configuration files are stored. This provider is useful for loading multiple configuration files and for organizing configuration data into separate files.
To use KubernetesSecretConfigProvider and KubernetesConfigMapConfigProvider, which are part of the OpenShift Configuration Provider plugin, you must set up access rights to the namespace that contains the configuration file.
You can use the other providers without setting up access rights. You can supply connector configuration for Kafka Connect or MirrorMaker 2 in this way by doing the following:
- Mount config maps or secrets into the Kafka Connect pod as environment variables or volumes
-
Enable
EnvVarConfigProvider,FileConfigProvider, orDirectoryConfigProviderin the Kafka Connect or MirrorMaker 2 configuration -
Pass connector configuration using the
templateproperty in thespecof theKafkaConnectorKafkaMirrorMaker2resource
Using providers help prevent the passing of restricted information through the Kafka Connect REST interface. You can use this approach in the following scenarios:
- Mounting environment variables with the values a connector uses to connect and communicate with a data source
- Mounting a properties file with values that are used to configure Kafka Connect connectors
- Mounting files in a directory that contains values for the TLS truststore and keystore used by a connector
A restart is required when using a new Secret or ConfigMap for a connector, which can disrupt other connectors.
Use the KubernetesSecretConfigProvider to provide configuration properties from a secret or the KubernetesConfigMapConfigProvider to provide configuration properties from a config map.
In this procedure, a config map provides configuration properties for a connector. The properties are specified as key values of the config map. The config map is mounted into the Kafka Connect pod as a volume.
Prerequisites
- A Kafka cluster is running.
- The Cluster Operator is running.
- You have a config map containing the connector configuration.
Example config map with connector properties
Procedure
Configure the
KafkaConnectresource.-
Enable the
KubernetesConfigMapConfigProvider
The specification shown here can support loading values from config maps and secrets.
Example Kafka Connect configuration to use config maps and secrets
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from
config.providers, taking the formconfig.providers.${alias}.class. - 2
KubernetesConfigMapConfigProviderprovides values from config maps.- 3
KubernetesSecretConfigProviderprovides values from secrets.
-
Enable the
Create or update the resource to enable the provider.
oc apply -f <kafka_connect_configuration_file>
oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a role that permits access to the values in the external config map.
Example role to access values from a config map
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The rule gives the role permission to access the
my-connector-configurationconfig map.Create a role binding to permit access to the namespace that contains the config map.
Example role binding to access the namespace that contains the config map
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The role binding gives the role permission to access the
my-projectnamespace.The service account must be the same one used by the Kafka Connect deployment. The service account name format is
<cluster_name>-connect, where<cluster_name>is the name of theKafkaConnectcustom resource.Reference the config map in the connector configuration.
Example connector configuration referencing the config map
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The placeholder structure is
configmaps:<path_and_file_name>:<property>.KubernetesConfigMapConfigProviderreads and extracts theoption1property value from the external config map.
Use the EnvVarConfigProvider to provide configuration properties as environment variables. Environment variables can contain values from config maps or secrets.
In this procedure, environment variables provide configuration properties for a connector to communicate with Amazon AWS. The connector must be able to read the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The values of the environment variables are derived from a secret mounted into the Kafka Connect pod.
The names of user-defined environment variables cannot start with KAFKA_ or STRIMZI_.
Prerequisites
- A Kafka cluster is running.
- The Cluster Operator is running.
- You have a secret containing the connector configuration.
Example secret with values for environment variables
Procedure
Configure the
KafkaConnectresource.-
Enable the
EnvVarConfigProvider -
Specify the environment variables using the
templateproperty.
Example Kafka Connect configuration to use external environment variables
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from
config.providers, taking the formconfig.providers.${alias}.class. - 2
EnvVarConfigProviderprovides values from environment variables.- 3
- The environment variable takes a value from the secret.
- 4
- The name of the secret containing the environment variable.
- 5
- The name of the key stored in the secret.
NoteThe
secretKeyRefproperty references keys in a secret. If you are using a config map instead of a secret, use theconfigMapKeyRefproperty.-
Enable the
Create or update the resource to enable the provider.
oc apply -f <kafka_connect_configuration_file>
oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reference the environment variable in the connector configuration.
Example connector configuration referencing the environment variable
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The placeholder structure is
env:<environment_variable_name>.EnvVarConfigProviderreads and extracts the environment variable values from the mounted secret.
Use the FileConfigProvider to provide configuration properties from a file within a directory. Files can be stored in config maps or secrets.
In this procedure, a file provides configuration properties for a connector. A database name and password are specified as properties of a secret. The secret is mounted to the Kafka Connect pod as a volume. Volumes are mounted on the path /mnt/<volume-name>.
Prerequisites
- A Kafka cluster is running.
- The Cluster Operator is running.
- You have a secret containing the connector configuration.
Example secret with database properties
Procedure
Configure the
KafkaConnectresource.-
Enable the
FileConfigProvider -
Specify the additional volume using the
templateproperty.
Example Kafka Connect configuration to use an external property file
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The alias for the configuration provider is used to define other configuration parameters.
- 2
FileConfigProviderprovides values from properties files. The parameter uses the alias fromconfig.providers, taking the formconfig.providers.${alias}.class.- 3
- The name of the volume containing the secret.
- 4
- The name of the secret.
- 5
- The name of the mounted volume, which must match the volume name in the
volumeslist. - 6
- The path where the secret is mounted, which must start with
/mnt/.
-
Enable the
Create or update the resource to enable the provider.
oc apply -f <kafka_connect_configuration_file>
oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reference the file properties in the connector configuration as placeholders.
Example connector configuration referencing the file
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The placeholder structure is
file:<path_and_file_name>:<property>.FileConfigProviderreads and extracts the database username and password property values from the mounted secret.
Use the DirectoryConfigProvider to provide configuration properties from multiple files within a directory. Files can be config maps or secrets.
In this procedure, a secret provides the TLS keystore and truststore user credentials for a connector. The credentials are in separate files. The secrets are mounted into the Kafka Connect pod as volumes. Volumes are mounted on the path /mnt/<volume-name>.
Prerequisites
- A Kafka cluster is running.
- The Cluster Operator is running.
- You have a secret containing the user credentials.
Example secret with user credentials
The my-user secret provides the keystore credentials (user.crt and user.key) for the connector.
The <cluster_name>-cluster-ca-cert secret generated when deploying the Kafka cluster provides the cluster CA certificate as truststore credentials (ca.crt).
Procedure
Configure the
KafkaConnectresource.-
Enable the
DirectoryConfigProvider -
Specify the additional volume using the
templateproperty.
Example Kafka Connect configuration to use external property files
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The alias for the configuration provider is used to define other configuration parameters.
- 2
DirectoryConfigProviderprovides values from files in a directory. The parameter uses the alias fromconfig.providers, taking the formconfig.providers.${alias}.class.- 3
- The name of the volume containing the secret.
- 4
- The name of the secret.
- 5
- The name of the mounted volume, which must match the volume name in the
volumeslist. - 6
- The path where the secret is mounted, which must start with
/mnt/.
-
Enable the
Create or update the resource to enable the provider.
oc apply -f <kafka_connect_configuration_file>
oc apply -f <kafka_connect_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reference the file properties in the connector configuration as placeholders.
Example connector configuration referencing the files
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The placeholder structure is
directory:<path>:<file_name>.DirectoryConfigProviderreads and extracts the credentials from the mounted secrets.
10.12.14. Customizing OpenShift resources
A Streams for Apache Kafka deployment creates OpenShift resources, such as Deployment, Pod, and Service resources. These resources are managed by Streams for Apache Kafka operators. Only the operator that is responsible for managing a particular OpenShift resource can change that resource. If you try to manually change an operator-managed OpenShift resource, the operator will revert your changes back.
Changing an operator-managed OpenShift resource can be useful if you want to perform certain tasks, such as the following:
-
Adding custom labels or annotations that control how
Podsare treated by Istio or other services -
Managing how
Loadbalancer-type Services are created by the cluster
To make the changes to an OpenShift resource, you can use the template property within the spec section of various Streams for Apache Kafka custom resources.
Here is a list of the custom resources where you can apply the changes:
-
Kafka.spec.kafka -
Kafka.spec.entityOperator -
Kafka.spec.kafkaExporter -
Kafka.spec.cruiseControl -
KafkaNodePool.spec -
KafkaConnect.spec -
KafkaMirrorMaker2.spec -
KafkaBridge.spec -
KafkaUser.spec
For more information about these properties, see the Streams for Apache Kafka Custom Resource API Reference.
The Streams for Apache Kafka Custom Resource API Reference provides more details about the customizable fields.
In the following example, the template property is used to modify the labels in a Kafka broker’s pod.
Example template customization
10.12.14.1. Customizing the image pull policy
Streams for Apache Kafka allows you to customize the image pull policy for containers in all pods deployed by the Cluster Operator. The image pull policy is configured using the environment variable STRIMZI_IMAGE_PULL_POLICY in the Cluster Operator deployment. The STRIMZI_IMAGE_PULL_POLICY environment variable can be set to three different values:
Always- Container images are pulled from the registry every time the pod is started or restarted.
IfNotPresent- Container images are pulled from the registry only when they were not pulled before.
Never- Container images are never pulled from the registry.
Currently, the image pull policy can only be customized for all Kafka, Kafka Connect, and Kafka MirrorMaker clusters at once. Changing the policy will result in a rolling update of all your Kafka, Kafka Connect, and Kafka MirrorMaker clusters.
10.12.14.2. Applying a termination grace period
Apply a termination grace period to give a Kafka cluster enough time to shut down cleanly.
Specify the time using the terminationGracePeriodSeconds property. Add the property to the template.pod configuration of the Kafka custom resource.
The time you add will depend on the size of your Kafka cluster. The OpenShift default for the termination grace period is 30 seconds. If you observe that your clusters are not shutting down cleanly, you can increase the termination grace period.
A termination grace period is applied every time a pod is restarted. The period begins when OpenShift sends a term (termination) signal to the processes running in the pod. The period should reflect the amount of time required to transfer the processes of the terminating pod to another pod before they are stopped. After the period ends, a kill signal stops any processes still running in the pod.
The following example adds a termination grace period of 120 seconds to the Kafka custom resource. You can also specify the configuration in the custom resources of other Kafka components.
Example termination grace period configuration
10.13. Restrictions on OpenShift labels
OpenShift labels make it easier to organize, manage, and discover OpenShift resources within your applications. The Cluster Operator is responsible for applying the following OpenShift labels to the operands it deploys. These labels cannot be overridden through template configuration of Streams for Apache Kafka resources:
-
app.kubernetes.io/name: Identifies the component type within Streams for Apache Kafka, such askafkaorcruise-control. -
app.kubernetes.io/instance: Represents the name of the custom resource to which the operand belongs to. For instance, if a Kafka custom resource is namedmy-cluster, this label will bear that name on the associated pods. -
app.kubernetes.io/part-of: Similar toapp.kubernetes.io/instance, but prefixed withstrimzi-. -
app.kubernetes.io/managed-by: Defines the application responsible for managing the operand, such asstrimzi-cluster-operatororstrimzi-user-operator.
Example OpenShift labels on a Kafka pod when deploying a Kafka custom resource named my-cluster
The KafkaTopic resource configures topics, including partition and replication factor settings. When you create, modify, or delete a topic using KafkaTopic, the Topic Operator ensures that these changes are reflected in the Kafka cluster.
For more information on the KafkaTopic resource, see the KafkaTopic schema reference.
11.1. Topic management
The KafkaTopic resource is responsible for managing a single topic within a Kafka cluster.
The Topic Operator operates as follows:
-
When a
KafkaTopicis created, deleted, or changed, the Topic Operator performs the corresponding operation on the Kafka topic.
If a topic is created, deleted, or modified directly within the Kafka cluster, without the presence of a corresponding KafkaTopic resource, the Topic Operator does not manage that topic. The Topic Operator will only manage Kafka topics associated with KafkaTopic resources and does not interfere with topics managed independently within the Kafka cluster. If a KafkaTopic does exist for a Kafka topic, any configuration changes made outside the resource are reverted.
The Topic Operator can detect cases where where multiple KafkaTopic resources are attempting to manage a Kafka topic using the same .spec.topicName. Only the oldest resource is reconciled, while the other resources fail with a resource conflict error.
11.2. Topic naming conventions
A KafkaTopic resource includes a name for the topic and a label that identifies the name of the Kafka cluster it belongs to.
Label identifying a Kafka cluster for topic handling
The label provides the cluster name of the Kafka resource. The Topic Operator uses the label as a mechanism for determining which KafkaTopic resources to manage. If the label does not match the Kafka cluster, the Topic Operator cannot see the KafkaTopic, and the topic is not created.
Kafka and OpenShift have their own naming validation rules, and a Kafka topic name might not be a valid resource name in OpenShift. If possible, try and stick to a naming convention that works for both.
Consider the following guidelines:
- Use topic names that reflect the nature of the topic
- Be concise and keep the name under 63 characters
- Use all lower case and hyphens
- Avoid special characters, spaces or symbols
The KafkaTopic resource allows you to specify the Kafka topic name using the metadata.name field. However, if the desired Kafka topic name is not a valid OpenShift resource name, you can use the spec.topicName property to specify the actual name. The spec.topicName field is optional, and when it’s absent, the Kafka topic name defaults to the metadata.name of the topic. When a topic is created, the topic name cannot be changed later.
Example of supplying a valid Kafka topic name
If more than one KafkaTopic resource refers to the same Kafka topic, the resource that was created first is considered to be the one managing the topic. The status of the newer resources is updated to indicate a conflict, and their Ready status is changed to False.
A Kafka client application, such as Kafka Streams, can automatically create topics with invalid OpenShift resource names. If you want to manage these topics, you must create KafkaTopic resources with a different .metadata.name, as shown in the previous example.
For more information on the requirements for identifiers and names in a cluster, refer to the OpenShift documentation Object Names and IDs.
11.3. Handling changes to topics
Configuration changes only go in one direction: from the KafkaTopic resource to the Kafka topic. Any changes to a Kafka topic managed outside the KafkaTopic resource are reverted.
If you are reverting back to a version of Streams for Apache Kafka earlier than 2.8, which uses internal topics for the storage of topic metadata, you still downgrade your Cluster Operator to the previous version, then downgrade Kafka brokers and client applications to the previous Kafka version as standard.
11.3.2. Automatic creation of topics
Applications can trigger the automatic creation of topics in the Kafka cluster. By default, the Kafka broker configuration auto.create.topics.enable is set to true, allowing the broker to create topics automatically when an application attempts to produce or consume from a non-existing topic. Applications might also use the Kafka AdminClient to automatically create topics. When an application is deployed along with its KafkaTopic resources, it is possible that automatic topic creation in the cluster happens before the Topic Operator can react to the KafkaTopic.
The topics created for an application deployment are initially created with default topic configuration. If the Topic Operator attempts to reconfigure the topics based on KafkaTopic resource specifications included with the application deployment, the operation might fail because the required change to the configuration is not allowed. For example, if the change means lowering the number of topic partitions. For this reason, it is recommended to disable auto.create.topics.enable in the Kafka cluster configuration.
11.4. Configuring Kafka topics
Use the properties of the KafkaTopic resource to configure Kafka topics. Changes made to topic configuration in the KafkaTopic are propagated to Kafka.
You can use oc apply to create or modify topics, and oc delete to delete existing topics.
For example:
-
oc apply -f <topic_config_file> -
oc delete KafkaTopic <topic_name>
To be able to delete topics, delete.topic.enable must be set to true (default) in the spec.kafka.config of the Kafka resource.
This procedure shows how to create a topic with 10 partitions and 2 replicas.
Before you begin
The KafkaTopic resource does not allow the following changes:
-
Renaming the topic defined in
spec.topicName. A mismatch betweenspec.topicNameandstatus.topicNamewill be detected. -
Decreasing the number of partitions using
spec.partitions(not supported by Kafka). -
Modifying the number of replicas specified in
spec.replicas.
Increasing spec.partitions for topics with keys will alter the partitioning of records, which can cause issues, especially when the topic uses semantic partitioning.
Prerequisites
- A running Kafka cluster configured with a Kafka broker listener using mTLS authentication and TLS encryption.
- A running Topic Operator (typically deployed with the Entity Operator).
-
For deleting a topic,
delete.topic.enable=true(default) in thespec.kafka.configof theKafkaresource.
Procedure
Configure the
KafkaTopicresource.Example Kafka topic configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow TipWhen modifying a topic, you can get the current version of the resource using
oc get kafkatopic my-topic-1 -o yaml.Create the
KafkaTopicresource in OpenShift.oc apply -f <topic_config_file>
oc apply -f <topic_config_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the ready status of the topic to change to
True:oc get kafkatopics -o wide -w -n <namespace>
oc get kafkatopics -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Kafka topic status
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-1 my-cluster 10 3 True my-topic-2 my-cluster 10 3 my-topic-3 my-cluster 10 3 True
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-1 my-cluster 10 3 True my-topic-2 my-cluster 10 3 my-topic-3 my-cluster 10 3 TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Topic creation is successful when the
READYoutput showsTrue.If the
READYcolumn stays blank, get more details on the status from the resource YAML or from the Topic Operator logs.Status messages provide details on the reason for the current status.
oc get kafkatopics my-topic-2 -o yaml
oc get kafkatopics my-topic-2 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Details on a topic with a
NotReadystatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the reason the topic is not ready is because the original number of partitions was reduced in the
KafkaTopicconfiguration. Kafka does not support this.After resetting the topic configuration, the status shows the topic is ready.
oc get kafkatopics my-topic-2 -o wide -w -n <namespace>
oc get kafkatopics my-topic-2 -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Status update of the topic
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-2 my-cluster 10 3 True
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-2 my-cluster 10 3 TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Fetching the details shows no messages
oc get kafkatopics my-topic-2 -o yaml
oc get kafkatopics my-topic-2 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Details on a topic with a
READYstatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
The recommended configuration for topics managed by the Topic Operator is a topic replication factor of 3, and a minimum of 2 in-sync replicas.
- 1
- The number of partitions for the topic.
- 2
- The number of replica topic partitions. Changing the number of replicas in the topic configuration requires a deployment of Cruise Control. For more information, see Chapter 22, Using Cruise Control to modify topic replication factor.
- 3
- The minimum number of replica partitions that a message must be successfully written to, or an exception is raised.
In-sync replicas are used in conjunction with the acks configuration for producer applications. The acks configuration determines the number of follower partitions a message must be replicated to before the message is acknowledged as successfully received. Replicas need to be reassigned when adding or removing brokers (see Chapter 20, Scaling clusters by adding or removing brokers).
This procedure describes how to convert Kafka topics that are currently managed through the KafkaTopic resource into unmanaged topics. This capability can be useful in various scenarios. For instance, you might want to update the metadata.name of a KafkaTopic resource. You can only do that by deleting the original KafkaTopic resource and recreating a new one.
By annotating a KafkaTopic resource with strimzi.io/managed=false, you indicate that the Topic Operator should no longer manage that particular topic. This allows you to retain the Kafka topic while making changes to the resource’s configuration or other administrative tasks.
Prerequisites
Procedure
Annotate the
KafkaTopicresource in OpenShift, settingstrimzi.io/managedtofalse:oc annotate kafkatopic my-topic-1 strimzi.io/managed="false" --overwrite
oc annotate kafkatopic my-topic-1 strimzi.io/managed="false" --overwriteCopy to Clipboard Copied! Toggle word wrap Toggle overflow Specify the
metadata.nameof the topic in yourKafkaTopicresource, which ismy-topic-1in this example.Check the status of the
KafkaTopicresource to make sure the request was successful:oc get kafkatopic my-topic-1 -o yaml
oc get kafkatopic my-topic-1 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example topic with an
UnmanagedstatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow You can now make changes to the
KafkaTopicresource without it affecting the Kafka topic it was managing.For example, to change the
metadata.name, do as follows:Delete the original
KafkaTopicresource:oc delete kafkatopic <kafka_topic_name>
oc delete kafkatopic <kafka_topic_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Recreate the
KafkaTopicresource with a differentmetadata.name, but usespec.topicNameto refer to the same topic that was managed by the original
-
If you haven’t deleted the original
KafkaTopicresource, and you wish to resume management of the Kafka topic again, set thestrimzi.io/managedannotation totrueor remove the annotation.
This procedure describes how to enable topic management for topics that are not currently managed through the KafkaTopic resource. You do this by creating a matching KafkaTopic resource.
Prerequisites
Procedure
Create a
KafkaTopicresource with ametadata.namethat is the same as the Kafka topic.Or use
spec.topicNameif the name of the topic in Kafka would not be a legal OpenShift resource name.Example Kafka topic configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the Kafka topic is named
my-topic-1.The Topic Operator checks whether the topic is managed by another
KafkaTopicresource. If it is, the older resource takes precedence and a resource conflict error is returned in the status of the new resource.Apply the
KafkaTopicresource:oc apply -f <topic_configuration_file>
oc apply -f <topic_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the operator to update the topic in Kafka.
The operator updates the Kafka topic with the
specof theKafkaTopicthat has the same name.Check the status of the
KafkaTopicresource to make sure the request was successful:oc get kafkatopics my-topic-1 -o yaml
oc get kafkatopics my-topic-1 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example topic with a
ReadystatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Successful reconciliation of the resource means the topic is now managed.
The value of
metadata.generation(the current version of the deployment) mustmatch status.observedGeneration(the latest reconciliation of the resource).
11.8. Deleting managed topics
The Topic Operator supports the deletion of topics managed through the KafkaTopic resource with or without OpenShift finalizers. This is determined by the STRIMZI_USE_FINALIZERS Topic Operator environment variable. By default, this is set to true, though it can be set to false in the Topic Operator env configuration if you do not want the Topic Operator to add finalizers.
Finalizers ensure orderly and controlled deletion of KafkaTopic resources. A finalizer for the Topic Operator is added to the metadata of the KafkaTopic resource:
Finalizer to control topic deletion
In this example, the finalizer is added for topic my-topic-1. The finalizer prevents the topic from being fully deleted until the finalization process is complete. If you then delete the topic using oc delete kafkatopic my-topic-1, a timestamp is added to the metadata:
Finalizer timestamp on deletion
The resource is still present. If the deletion fails, it is shown in the status of the resource.
When the finalization tasks are successfully executed, the finalizer is removed from the metadata, and the resource is fully deleted.
Finalizers also serve to prevent related resources from being deleted. If the Topic Operator is not running, it won’t be able to remove its finalizer from the metadata.finalizers. And any attempt to directly delete the KafkaTopic resources or the namespace will fail or timeout, leaving the namespace in a stuck terminating state. If this happens, you can bypass the finalization process by removing the finalizers on topics.
11.9. Removing finalizers on topics
If the Topic Operator is not running, and you want to bypass the finalization process when deleting managed topics, you must remove the finalizers. You can do this manually by editing the resources directly or by using a command.
To remove finalizers on all topics, use the following command:
Removing finalizers on topics
oc get kt -o=json | jq '.items[].metadata.finalizers = null' | oc apply -f -
oc get kt -o=json | jq '.items[].metadata.finalizers = null' | oc apply -f -
The command uses the jq command line JSON parser tool to modify the KafkaTopic (kt) resources by setting the finalizers to null. You can also use the command for a specific topic:
Removing a finalizer on a specific topic
oc get kt <topic_name> -o=json | jq '.metadata.finalizers = null' | oc apply -f -
oc get kt <topic_name> -o=json | jq '.metadata.finalizers = null' | oc apply -f -After running the command, you can go ahead and delete the topics. Alternatively, if the topics were already being deleted but were blocked due to outstanding finalizers then their deletion should complete.
Be careful when removing finalizers, as any cleanup operations associated with the finalization process are not performed if the Topic Operator is not running. For example, if you remove the finalizer from a KafkaTopic resource and subsequently delete the resource, the related Kafka topic won’t be deleted.
11.10. Considerations when disabling topic deletion
When the delete.topic.enable configuration in Kafka is set to false, topics cannot be deleted. This might be required in certain scenarios, but it introduces a consideration when using the Topic Operator.
As topics cannot be deleted, finalizers added to the metadata of a KafkaTopic resource to control topic deletion are never removed by the Topic Operator (though they can be removed manually). Similarly, any Custom Resource Definitions (CRDs) or namespaces associated with topics cannot be deleted.
Before configuring delete.topic.enable=false, assess these implications to ensure it aligns with your specific requirements.
To avoid using finalizers, you can set the STRIMZI_USE_FINALIZERS Topic Operator environment variable to false.
11.11. Tuning request batches for topic operations
The Topic Operator uses the request batching capabilities of the Kafka Admin API for operations on topic resources. You can fine-tune the batching mechanism using the following operator configuration properties:
-
STRIMZI_MAX_QUEUE_SIZEto set the maximum size of the topic event queue. The default value is 1024. -
STRIMZI_MAX_BATCH_SIZEto set the maximum number of topic events allowed in a single batch. The default value is 100. -
MAX_BATCH_LINGER_MSto specify the maximum time to wait for a batch to accumulate items before processing. The default is 100 milliseconds.
If the maximum size of the request batching queue is exceeded, the Topic Operator shuts down and is restarted. To prevent frequent restarts, consider adjusting the STRIMZI_MAX_QUEUE_SIZE property to accommodate the typical load.
When you create, modify or delete a user using the KafkaUser resource, the User Operator ensures that these changes are reflected in the Kafka cluster.
For more information on the KafkaUser resource, see the KafkaUser schema reference.
12.1. Configuring Kafka users
Use the properties of the KafkaUser resource to configure Kafka users.
You can use oc apply to create or modify users, and oc delete to delete existing users.
For example:
-
oc apply -f <user_config_file> -
oc delete KafkaUser <user_name>
Users represent Kafka clients. When you configure Kafka users, you enable the user authentication and authorization mechanisms required by clients to access Kafka. The mechanism used must match the equivalent Kafka configuration. For more information on using Kafka and KafkaUser resources to secure access to Kafka brokers, see https://docs.redhat.com/en/documentation/red_hat_streams_for_apache_kafka/latest/html-single/deploying_and_managing_streams_for_apache_kafka_on_openshift/index#assembly-securing-access-str.
Prerequisites
- A running Kafka cluster configured with a Kafka broker listener using mTLS authentication and TLS encryption.
- A running User Operator (typically deployed with the Entity Operator).
Procedure
Configure the
KafkaUserresource.This example specifies mTLS authentication and simple authorization using ACLs.
Example Kafka user configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
KafkaUserresource in OpenShift.oc apply -f <user_config_file>
oc apply -f <user_config_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the ready status of the user to change to
True:oc get kafkausers -o wide -w -n <namespace>
oc get kafkausers -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Kafka user status
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-1 my-cluster tls simple True my-user-2 my-cluster tls simple my-user-3 my-cluster tls simple True
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-1 my-cluster tls simple True my-user-2 my-cluster tls simple my-user-3 my-cluster tls simple TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow User creation is successful when the
READYoutput showsTrue.If the
READYcolumn stays blank, get more details on the status from the resource YAML or User Operator logs.Messages provide details on the reason for the current status.
oc get kafkausers my-user-2 -o yaml
oc get kafkausers my-user-2 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Details on a user with a
NotReadystatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the reason the user is not ready is because simple authorization is not enabled in the
Kafkaconfiguration.Kafka configuration for simple authorization
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After updating the Kafka configuration, the status shows the user is ready.
oc get kafkausers my-user-2 -o wide -w -n <namespace>
oc get kafkausers my-user-2 -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Status update of the user
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-2 my-cluster tls simple True
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-2 my-cluster tls simple TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Fetching the details shows no messages.
oc get kafkausers my-user-2 -o yaml
oc get kafkausers my-user-2 -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Details on a user with a
READYstatusCopy to Clipboard Copied! Toggle word wrap Toggle overflow
12.2. Ignoring specific users
In some scenarios, you might want to exclude certain users from being managed by the User Operator. This is useful for the following types of users:
-
Special built-in users (for example,
ANONYMOUSor*) - Internal users managed by other mechanisms
When you exclude users from being managed, the User Operator ignores their:
- ACLs
- Quotas
- SCRAM-SHA credentials
To define which users should be ignored, set the STRIMZI_IGNORED_USERS_PATTERN environment variable with a regular expression. For example, to ignore ANONYMOUS or * users, set it to ^\*|ANONYMOUS$. You can set the environment variable in your Kafka custom resource (.spec.teplate.entityOperator.userOperatorContainer.env) or in the Deployment resource of your standalone User Operator.
Example Kafka configuration excluding users
Example standalone Deployment configuration excluding users
You can use the Red Hat build of Apicurio Registry with Streams for Apache Kafka.
Apicurio Registry is a datastore for sharing standard event schemas and API designs across API and event-driven architectures. You can use Apicurio Registry to decouple the structure of your data from your client applications, and to share and manage your data types and API descriptions at runtime using a REST interface.
Apicurio Registry stores schemas used to serialize and deserialize messages, which can then be referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas. Apicurio Registry provides Kafka client serializers/deserializers for Kafka producer and consumer applications. Kafka producer applications use serializers to encode messages that conform to specific event schemas. Kafka consumer applications use deserializers, which validate that the messages have been serialized using the correct schema, based on a specific schema ID.
You can enable your applications to use a schema from the registry. This ensures consistent schema usage and helps to prevent data errors at runtime.
The Red Hat build of Debezium is a distributed change data capture platform. It captures row-level changes in databases, creates change event records, and streams the records to Kafka topics. Debezium is built on Apache Kafka. You can deploy and integrate the Red Hat build of Debezium with Streams for Apache Kafka. Following a deployment of Streams for Apache Kafka, you deploy Debezium as a connector configuration through Kafka Connect. Debezium passes change event records to Streams for Apache Kafka on OpenShift. Applications can read these change event streams and access the change events in the order in which they occurred.
Debezium has multiple uses, including:
- Data replication
- Updating caches and search indexes
- Simplifying monolithic applications
- Data integration
- Enabling streaming queries
To capture database changes, deploy Kafka Connect with a Debezium database connector. You configure a KafkaConnector resource to define the connector instance.
For more information on deploying the Red Hat build of Debezium with Streams for Apache Kafka, refer to the product documentation. The documentation includes a Getting Started with Debezium guide that guides you through the process of setting up the services and connector required to view change event records for database updates.
Chapter 15. Setting up client access to a Kafka cluster
After you have deployed Streams for Apache Kafka, you can set up client access to your Kafka cluster. To verify the deployment, you can deploy example producer and consumer clients. Otherwise, create listeners that provide client access within or outside the OpenShift cluster.
15.1. Deploying example clients
Send and receive messages from a Kafka cluster installed on OpenShift.
This procedure describes how to deploy Kafka clients to the OpenShift cluster, then produce and consume messages to test your installation. The clients are deployed using the Kafka container image.
Prerequisites
- The Kafka cluster is available for the clients.
Procedure
Deploy a Kafka producer.
This example deploys a Kafka producer that connects to the Kafka cluster
my-cluster.A topic named
my-topicis created.Deploying a Kafka producer to OpenShift
oc run kafka-producer -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
oc run kafka-producer -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topicCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Type a message into the console where the producer is running.
- Press Enter to send the message.
Deploy a Kafka consumer.
The consumer should consume messages produced to
my-topicin the Kafka clustermy-cluster.Deploying a Kafka consumer to OpenShift
oc run kafka-consumer -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
oc run kafka-consumer -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginningCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Confirm that you see the incoming messages in the consumer console.
15.2. Configuring listeners to connect to Kafka
Use listeners to enable client connections to Kafka. Streams for Apache Kafka provides a generic GenericKafkaListener schema with properties to configure listeners through the Kafka resource.
When configuring a Kafka cluster, you specify a listener type based on your requirements, environment, and infrastructure. Services, routes, load balancers, and ingresses for clients to connect to a cluster are created according to the listener type.
Internal and external listener types are supported.
- Internal listeners
Use internal listener types to connect clients within a kubernetes cluster.
-
internalto connect within the same OpenShift cluster cluster-ipto expose Kafka using per-brokerClusterIPservicesInternal listeners use a headless service and the DNS names assigned to the broker pods. By default, they do not use the OpenShift service DNS domain (typically
.cluster.local). However, you can customize this configuration using theuseServiceDnsDomainproperty. Consider using acluster-iptype listener if routing through the headless service isn’t feasible or if you require a custom access mechanism, such as when integrating with specific Ingress controllers or the OpenShift Gateway API.
-
- External listeners
Use external listener types to connect clients outside an OpenShift cluster.
-
nodeportto use ports on OpenShift nodes -
loadbalancerto use loadbalancer services -
ingressto use KubernetesIngressand the Ingress NGINX Controller for Kubernetes (Kubernetes only) routeto use OpenShiftRouteand the default HAProxy router (OpenShift only)External listeners handle access to a Kafka cluster from networks that require different authentication mechanisms. For example, loadbalancers might not be suitable for certain infrastructure, such as bare metal, where node ports provide a better option.
-
Do not use the built-in ingress controller on OpenShift, use the route type instead. The Ingress NGINX Controller is only intended for use on Kubernetes. The route type is only supported on OpenShift.
Each listener is defined as an array in the Kafka resource.
Example listener configuration
You can configure as many listeners as required, as long as their names and ports are unique. You can also configure listeners for secure connection using authentication.
If you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.
15.3. Listener naming conventions
From the listener configuration, the resulting listener bootstrap and per-broker service names are structured according to the following naming conventions:
| Listener type | Bootstrap service name | Per-Broker service name |
|---|---|---|
|
| <cluster_name>-kafka-bootstrap | Not applicable |
|
| <cluster_name>-kafka-<listener-name>-bootstrap | <cluster_name>-kafka-<listener-name>-<idx> |
For example, my-cluster-kafka-bootstrap, my-cluster-kafka-external1-bootstrap, and my-cluster-kafka-external1-0. The names are assigned to the services, routes, load balancers, and ingresses created through the listener configuration.
You can use certain backwards compatible names and port numbers to transition listeners initially configured under the retired KafkaListeners schema. The resulting external listener naming convention varies slightly. The specific combinations of listener name and port configuration values in the following table are backwards compatible.
| Listener name | Port | Bootstrap service name | Per-Broker service name |
|---|---|---|---|
|
|
| <cluster_name>-kafka-bootstrap | Not applicable |
|
|
| <cluster-name>-kafka-bootstrap | Not applicable |
|
|
| <cluster_name>-kafka-bootstrap | <cluster_name>-kafka-bootstrap-<idx> |
15.4. Accessing Kafka using node ports
Use node ports to access a Kafka cluster from an external client outside the OpenShift cluster.
To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.
The procedure shows basic nodeport listener configuration. You can use listener properties to enable TLS encryption (tls) and specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the following configuration properties with nodeport listeners:
preferredNodePortAddressType- Specifies the first address type that’s checked as the node address.
externalTrafficPolicy- Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
nodePort- Overrides the assigned node port numbers for the bootstrap and broker services.
For more information on listener configuration, see the GenericKafkaListener schema reference.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external4.
Procedure
Configure a
Kafkaresource with an external listener set to thenodeporttype.For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource.
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow A cluster CA certificate to verify the identity of the kafka brokers is created in the secret
my-cluster-cluster-ca-cert.NodePorttype services are created for each Kafka broker, as well as an external bootstrap service.Node port services created for the bootstrap and brokers
NAME TYPE CLUSTER-IP PORT(S) my-cluster-kafka-external4-0 NodePort 172.30.55.13 9094:31789/TCP my-cluster-kafka-external4-1 NodePort 172.30.250.248 9094:30028/TCP my-cluster-kafka-external4-2 NodePort 172.30.115.81 9094:32650/TCP my-cluster-kafka-external4-bootstrap NodePort 172.30.30.23 9094:32650/TCP
NAME TYPE CLUSTER-IP PORT(S) my-cluster-kafka-external4-0 NodePort 172.30.55.13 9094:31789/TCP my-cluster-kafka-external4-1 NodePort 172.30.250.248 9094:30028/TCP my-cluster-kafka-external4-2 NodePort 172.30.115.81 9094:32650/TCP my-cluster-kafka-external4-bootstrap NodePort 172.30.30.23 9094:32650/TCPCopy to Clipboard Copied! Toggle word wrap Toggle overflow The bootstrap address used for client connection is propagated to the
statusof theKafkaresource.Example status for the bootstrap address
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafkaresource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external4")].bootstrapServers}{"\n"}' ip-10-0-224-199.us-west-2.compute.internal:32650oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external4")].bootstrapServers}{"\n"}' ip-10-0-224-199.us-west-2.compute.internal:32650Copy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtoc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure your client to connect to the brokers.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
ip-10-0-224-199.us-west-2.compute.internal:32650. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
15.5. Accessing Kafka using loadbalancers
Use loadbalancers to access a Kafka cluster from an external client outside the OpenShift cluster.
To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.
The procedure shows basic loadbalancer listener configuration. You can use listener properties to enable TLS encryption (tls) and specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the following configuration properties with loadbalancer listeners:
loadBalancerSourceRanges- Restricts traffic to a specified list of CIDR (Classless Inter-Domain Routing) ranges.
externalTrafficPolicy- Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
loadBalancerIP- Requests a specific IP address when creating a loadbalancer.
For more information on listener configuration, see the GenericKafkaListener schema reference.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external3.
Procedure
Configure a
Kafkaresource with an external listener set to theloadbalancertype.For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource.
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow A cluster CA certificate to verify the identity of the kafka brokers is also created in the secret
my-cluster-cluster-ca-cert.loadbalancertype services and loadbalancers are created for each Kafka broker, as well as an external bootstrap service.Loadbalancer services and loadbalancers created for the bootstraps and brokers
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The bootstrap address used for client connection is propagated to the
statusof theKafkaresource.Example status for the bootstrap address
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The DNS addresses used for client connection are propagated to the
statusof each loadbalancer service.Example status for the bootstrap loadbalancer
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafkaresource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external3")].bootstrapServers}{"\n"}' a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external3")].bootstrapServers}{"\n"}' a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094Copy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtoc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure your client to connect to the brokers.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
15.6. Accessing Kafka using OpenShift routes
Use OpenShift routes to access a Kafka cluster from clients outside the OpenShift cluster.
To be able to use routes, add configuration for a route type listener in the Kafka custom resource. When applied, the configuration creates a dedicated route and service for an external bootstrap and each broker in the cluster. Clients connect to the bootstrap route, which routes them through the bootstrap service to connect to a broker. Per-broker connections are then established using DNS names, which route traffic from the client to the broker through the broker-specific routes and services.
To connect to a broker, you specify a hostname for the route bootstrap address, as well as the certificate used for TLS encryption. For access using routes, the port is always 443.
An OpenShift route address comprises the Kafka cluster name, the listener name, the project name, and the domain of the router. For example, my-cluster-kafka-external1-bootstrap-my-project.domain.com (<cluster_name>-kafka-<listener_name>-bootstrap-<namespace>.<domain>). Each DNS label (between periods “.”) must not exceed 63 characters, and the total length of the address must not exceed 255 characters.
The procedure shows basic listener configuration. TLS encryption (tls) must be enabled. You can also specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the host configuration property with route listeners to specify the hostnames used by the bootstrap and per-broker services.
For more information on listener configuration, see the GenericKafkaListener schema reference.
TLS passthrough
TLS passthrough is enabled for routes created by Streams for Apache Kafka. Kafka uses a binary protocol over TCP, but routes are designed to work with a HTTP protocol. To be able to route TCP traffic through routes, Streams for Apache Kafka uses TLS passthrough with Server Name Indication (SNI).
SNI helps with identifying and passing connection to Kafka brokers. In passthrough mode, TLS encryption is always used. Because the connection passes to the brokers, the listeners use TLS certificates signed by the internal cluster CA and not the ingress certificates. To configure listeners to use your own listener certificates, use the brokerCertChainAndKey property.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external1.
Procedure
Configure a
Kafkaresource with an external listener set to theroutetype.For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- For
routetype listeners, TLS encryption must be enabled (true).
Create or update the resource.
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow A cluster CA certificate to verify the identity of the kafka brokers is created in the secret
my-cluster-cluster-ca-cert.ClusterIPtype services are created for each Kafka broker, as well as an external bootstrap service.A
routeis also created for each service, with a DNS address (host/port) to expose them using the default OpenShift HAProxy router.The routes are preconfigured with TLS passthrough.
Routes created for the bootstraps and brokers
NAME HOST/PORT SERVICES PORT TERMINATION my-cluster-kafka-external1-0 my-cluster-kafka-external1-0-my-project.router.com my-cluster-kafka-external1-0 9094 passthrough my-cluster-kafka-external1-1 my-cluster-kafka-external1-1-my-project.router.com my-cluster-kafka-external1-1 9094 passthrough my-cluster-kafka-external1-2 my-cluster-kafka-external1-2-my-project.router.com my-cluster-kafka-external1-2 9094 passthrough my-cluster-kafka-external1-bootstrap my-cluster-kafka-external1-bootstrap-my-project.router.com my-cluster-kafka-external1-bootstrap 9094 passthrough
NAME HOST/PORT SERVICES PORT TERMINATION my-cluster-kafka-external1-0 my-cluster-kafka-external1-0-my-project.router.com my-cluster-kafka-external1-0 9094 passthrough my-cluster-kafka-external1-1 my-cluster-kafka-external1-1-my-project.router.com my-cluster-kafka-external1-1 9094 passthrough my-cluster-kafka-external1-2 my-cluster-kafka-external1-2-my-project.router.com my-cluster-kafka-external1-2 9094 passthrough my-cluster-kafka-external1-bootstrap my-cluster-kafka-external1-bootstrap-my-project.router.com my-cluster-kafka-external1-bootstrap 9094 passthroughCopy to Clipboard Copied! Toggle word wrap Toggle overflow The DNS addresses used for client connection are propagated to the
statusof each route.Example status for the bootstrap route
status: ingress: - host: >- my-cluster-kafka-external1-bootstrap-my-project.router.com # ...status: ingress: - host: >- my-cluster-kafka-external1-bootstrap-my-project.router.com # ...Copy to Clipboard Copied! Toggle word wrap Toggle overflow Use a target broker to check the client-server TLS connection on port 443 using the OpenSSL
s_client.openssl s_client -connect my-cluster-kafka-external1-0-my-project.router.com:443 -servername my-cluster-kafka-external1-0-my-project.router.com -showcerts
openssl s_client -connect my-cluster-kafka-external1-0-my-project.router.com:443 -servername my-cluster-kafka-external1-0-my-project.router.com -showcertsCopy to Clipboard Copied! Toggle word wrap Toggle overflow The server name is the Server Name Indication (SNI) for passing the connection to the broker.
If the connection is successful, the certificates for the broker are returned.
Certificates for the broker
Certificate chain 0 s:O = io.strimzi, CN = my-cluster-kafka i:O = io.strimzi, CN = cluster-ca v0
Certificate chain 0 s:O = io.strimzi, CN = my-cluster-kafka i:O = io.strimzi, CN = cluster-ca v0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the address of the bootstrap service from the status of the
Kafkaresource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external1")].bootstrapServers}{"\n"}' my-cluster-kafka-external1-bootstrap-my-project.router.com:443oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external1")].bootstrapServers}{"\n"}' my-cluster-kafka-external1-bootstrap-my-project.router.com:443Copy to Clipboard Copied! Toggle word wrap Toggle overflow The address comprises the Kafka cluster name, the listener name, the project name and the domain of the router (
router.comin this example).Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtoc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure your client to connect to the brokers.
- Specify the address for the bootstrap service and port 443 in your Kafka client as the bootstrap address to connect to the Kafka cluster.
Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
15.7. Discovering connection details for clients
Service discovery makes it easier for client applications running in the same OpenShift cluster as Streams for Apache Kafka to interact with a Kafka cluster.
A service discovery label and annotation are created for the following services:
- Internal Kafka bootstrap service
Kafka Bridge service
- Service discovery label
-
The service discovery label,
strimzi.io/discovery, is set totrueforServiceresources to make them discoverable for client connections. - Service discovery annotation
- The service discovery annotation provides connection details in JSON format for each service for client applications to use to establish connections.
Example internal Kafka bootstrap service
Example Kafka Bridge service
Find services by specifying the discovery label when fetching services from the command line or a corresponding API call.
Returning services using the discovery label
oc get service -l strimzi.io/discovery=true
oc get service -l strimzi.io/discovery=trueConnection details are returned when retrieving the service discovery label.
Chapter 16. Securing access to a Kafka cluster
Secure connections by configuring Kafka and Kafka users. Through configuration, you can implement encryption, authentication, and authorization mechanisms.
Kafka configuration
To establish secure access to Kafka, configure the Kafka resource to set up the following configurations based on your specific requirements:
Listeners with specified authentication types to define how clients authenticate
- TLS encryption for communication between Kafka and clients
- Supported TLS versions and cipher suites for additional security
- Authorization for the entire Kafka cluster
- Network policies for restricting access
- Super users for unconstrained access to brokers
Authentication is configured independently for each listener, while authorization is set up for the whole Kafka cluster.
For more information on access configuration for Kafka, see the Kafka schema reference and GenericKafkaListener schema reference.
User (client-side) configuration
To enable secure client access to Kafka, configure KafkaUser resources. These resources represent clients and determine how they authenticate and authorize with the Kafka cluster.
Configure the KafkaUser resource to set up the following configurations based on your specific requirements:
Authentication that must match the enabled listener authentication
- Supported TLS versions and cipher suites that must match the Kafka configuration
Simple authorization to apply Access Control List (ACL) rules
- ACLs for fine-grained control over user access to topics and actions
- Quotas to limit client access based on byte rates or CPU utilization
The User Operator creates the user representing the client and the security credentials used for client authentication, based on the chosen authentication type.
For more information on access configuration for users, see the KafkaUser schema reference.
16.1. Configuring client authentication on listeners
Configure client authentication for Kafka brokers when creating listeners. Specify the listener authentication type using the Kafka.spec.kafka.listeners.authentication property in the Kafka resource.
For clients inside the OpenShift cluster, you can create plain (without encryption) or tls internal listeners. The internal listener type use a headless service and the DNS names given to the broker pods. As an alternative to the headless service, you can also create a cluster-ip type of internal listener to expose Kafka using per-broker ClusterIP services. For clients outside the OpenShift cluster, you create external listeners and specify a connection mechanism, which can be nodeport, loadbalancer, ingress (Kubernetes only), or route (OpenShift only).
For more information on the configuration options for connecting an external client, see Chapter 15, Setting up client access to a Kafka cluster.
Supported authentication options:
- mTLS authentication (only on the listeners with TLS enabled encryption)
- SCRAM-SHA-512 authentication
- OAuth 2.0 token-based authentication
- Custom authentication
- TLS versions and cipher suites
If you’re using OAuth 2.0 for client access management, user authentication and authorization credentials are handled through the authorization server.
The authentication option you choose depends on how you wish to authenticate client access to Kafka brokers.
Try exploring the standard authentication options before using custom authentication. Custom authentication allows for any type of Kafka-supported authentication. It can provide more flexibility, but also adds complexity.
Figure 16.1. Kafka listener authentication options
The listener authentication property is used to specify an authentication mechanism specific to that listener.
If no authentication property is specified then the listener does not authenticate clients which connect through that listener. The listener will accept all connections without authentication.
Authentication must be configured when using the User Operator to manage KafkaUsers.
The following example shows:
-
A
plainlistener configured for SCRAM-SHA-512 authentication -
A
tlslistener with mTLS authentication -
An
externallistener with mTLS authentication
Each listener is configured with a unique name and port within a Kafka cluster.
When configuring listeners for client access to brokers, you can use port 9092 or higher (9093, 9094, and so on), but with a few exceptions. The listeners cannot be configured to use the ports reserved for interbroker communication (9090 and 9091), Prometheus metrics (9404), and JMX (Java Management Extensions) monitoring (9999).
Example listener authentication configuration
16.1.1. mTLS authentication
mTLS authentication is always used for the communication between Kafka nodes.
Streams for Apache Kafka can configure Kafka to use TLS (Transport Layer Security) to provide encrypted communication between Kafka brokers and clients either with or without mutual authentication. For mutual, or two-way, authentication, both the server and the client present certificates. When you configure mTLS authentication, the broker authenticates the client (client authentication) and the client authenticates the broker (server authentication).
mTLS listener configuration in the Kafka resource requires the following:
-
tls: trueto specify TLS encryption and server authentication -
authentication.type: tlsto specify the client authentication
When a Kafka cluster is created by the Cluster Operator, it creates a new secret with the name <cluster_name>-cluster-ca-cert. The secret contains a CA certificate. The CA certificate is in PEM and PKCS #12 format. To verify a Kafka cluster, add the CA certificate to the truststore in your client configuration. To verify a client, add a user certificate and key to the keystore in your client configuration. For more information on configuring a client for mTLS, see Section 16.3.2, “Configuring user authentication”.
TLS authentication is more commonly one-way, with one party authenticating the identity of another. For example, when HTTPS is used between a web browser and a web server, the browser obtains proof of the identity of the web server.
16.1.2. SCRAM-SHA-512 authentication
SCRAM (Salted Challenge Response Authentication Mechanism) is an authentication protocol that can establish mutual authentication using passwords. Streams for Apache Kafka can configure Kafka to use SASL (Simple Authentication and Security Layer) SCRAM-SHA-512 to provide authentication on both unencrypted and encrypted client connections.
When SCRAM-SHA-512 authentication is used with a TLS connection, the TLS protocol provides the encryption, but is not used for authentication.
The following properties of SCRAM make it safe to use SCRAM-SHA-512 even on unencrypted connections:
- The passwords are not sent in the clear over the communication channel. Instead the client and the server are each challenged by the other to offer proof that they know the password of the authenticating user.
- The server and client each generate a new challenge for each authentication exchange. This means that the exchange is resilient against replay attacks.
When KafkaUser.spec.authentication.type is configured with scram-sha-512 the User Operator will generate a random 32-character password consisting of upper and lowercase ASCII letters and numbers.
Control listener access by configuring the networkPolicyPeers property in the Kafka resource.
By default, Streams for Apache Kafka automatically creates a NetworkPolicy resource for every enabled Kafka listener, allowing connections from all namespaces.
To restrict listener access to specific applications or namespaces at the network level, configure the networkPolicyPeers property. Each listener can have its own networkPolicyPeers configuration. For more information on network policy peers, refer to the NetworkPolicyPeer API reference.
If you want to use custom network policies, you can set the STRIMZI_NETWORK_POLICY_GENERATION environment variable to false in the Cluster Operator configuration. For more information, see Section 10.6, “Configuring the Cluster Operator”.
Your configuration of OpenShift must support ingress NetworkPolicies in order to use network policies.
Prerequisites
- An OpenShift cluster with support for Ingress NetworkPolicies.
- The Cluster Operator is running.
Procedure
Configure the
networkPolicyPeersproperty to define the application pods or namespaces allowed to access the Kafka cluster.This example shows configuration for a
tlslistener to allow connections only from application pods with the labelappset tokafka-client:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Kafkaresource configuration.
This procedure shows how to configure custom server certificates for TLS listeners or external listeners which have TLS encryption enabled.
By default, Kafka listeners use certificates signed by Streams for Apache Kafka’s internal CA (certificate authority). The Cluster Operator automatically generates a CA certificate when creating a Kafka cluster. To configure a client for TLS, the CA certificate is included in its truststore configuration to authenticate the Kafka cluster. Alternatively, you have the option to install and use your own CA certificates.
However, if you prefer more granular control by using your own custom certificates at the listener-level, you can configure listeners using brokerCertChainAndKey properties. You create a secret with your own private key and server certificate, then specify them in the brokerCertChainAndKey configuration.
User-provided certificates allow you to leverage existing security infrastructure. You can use a certificate signed by a public (external) CA or a private CA. Kafka clients need to trust the CA which was used to sign the listener certificate. If signed by a public CA, you usually won’t need to add it to a client’s truststore configuration.
Custom certificates are not managed by Streams for Apache Kafka, so you need to renew them manually.
Listener certificates are used for TLS encryption and server authentication only. They are not used for TLS client authentication. If you want to use your own certificate for TLS client authentication as well, you must install and use your own clients CA.
Prerequisites
- The Cluster Operator is running.
Each listener requires the following:
A compatible server certificate signed by an external CA. (Provide an X.509 certificate in PEM format.)
You can use one listener certificate for multiple listeners.
- Subject Alternative Names (SANs) are specified in the certificate for each listener. For more information, see Section 16.1.5, “Specifying SANs for custom listener certificates”.
If you are not using a self-signed certificate, you can provide a certificate that includes the whole CA chain in the certificate.
You can only use the brokerCertChainAndKey properties if TLS encryption (tls: true) is configured for the listener.
Streams for Apache Kafka does not support the use of encrypted private keys for TLS. The private key stored in the secret must be unencrypted for this to work.
Procedure
Create a
Secretcontaining your private key and server certificate:oc create secret generic <my_secret> --from-file=<my_listener_key.key> --from-file=<my_listener_certificate.crt>
oc create secret generic <my_secret> --from-file=<my_listener_key.key> --from-file=<my_listener_certificate.crt>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
Kafkaresource for your cluster.Configure the listener to use your
Secret, certificate file, and private key file in theconfiguration.brokerCertChainAndKeyproperty.Example configuration for a
loadbalancerexternal listener with TLS encryption enabledCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example configuration for a TLS listener
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the changes to the
Kafkaresource configuration.The Cluster Operator starts a rolling update of the Kafka cluster, which updates the configuration of the listeners.
NoteA rolling update is also started if you update a Kafka listener certificate in a
Secretthat is already used by a listener.
16.1.5. Specifying SANs for custom listener certificates
In order to use TLS hostname verification with custom Kafka listener certificates, you must specify the correct Subject Alternative Names (SANs) for each listener.
The certificate SANs must specify hostnames for the following:
- All of the Kafka brokers in your cluster
- The Kafka cluster bootstrap service
You can use wildcard certificates if they are supported by your CA.
16.1.5.1. Examples of SANs for internal listeners
Use the following examples to help you specify hostnames of the SANs in your certificates for your internal listeners.
Replace <cluster-name> with the name of the Kafka cluster and <namespace> with the OpenShift namespace where the cluster is running.
Wildcards example for a type: internal listener
Non-wildcards example for a type: internal listener
Non-wildcards example for a type: cluster-ip listener
16.1.5.2. Examples of SANs for external listeners
For external listeners which have TLS encryption enabled, the hostnames you need to specify in certificates depends on the external listener type.
| External listener type | In the SANs, specify… |
|---|---|
|
|
Addresses of all Kafka broker You can use a matching wildcard name. |
|
|
Addresses of all Kafka broker You can use a matching wildcard name. |
|
|
Addresses of all Kafka broker You can use a matching wildcard name. |
|
| Addresses of all OpenShift worker nodes that the Kafka broker pods might be scheduled to. You can use a matching wildcard name. |
16.2. Configuring authorized access to Kafka
Configure authorized access to a Kafka cluster using the Kafka.spec.kafka.authorization property in the Kafka resource. If the authorization property is missing, no authorization is enabled and clients have no restrictions. When enabled, authorization is applied to all enabled listeners. The authorization method is defined in the type field.
Supported authorization options:
- Simple authorization
- OAuth 2.0 authorization (if you are using OAuth 2.0 token based authentication)
- Open Policy Agent (OPA) authorization
- Custom authorization
Figure 16.2. Kafka cluster authorization options
16.2.1. Designating super users
Super users can access all resources in your Kafka cluster regardless of any access restrictions, and are supported by all authorization mechanisms.
To designate super users for a Kafka cluster, add a list of user principals to the superUsers property of the Kafka resource. If a user uses mTLS authentication, the username is the common name from the TLS certificate subject prefixed with CN=. If you are not using the User Operator and using your own certificates for mTLS, the username is the full certificate subject.
A full certificate subject can include the following fields:
-
CN=<common_name> -
OU=<organizational_unit> -
O=<organization> -
L=<locality> -
ST=<state> -
C=<country_code>
Omit any fields that are not applicable.
An example configuration with super users
16.3. Configuring user (client-side) security mechanisms
When configuring security mechanisms in clients, the clients are represented as users. Use the KafkaUser resource to configure the authentication, authorization, and access rights for Kafka clients.
Authentication permits user access, and authorization constrains user access to permissible actions. You can also create super users that have unconstrained access to Kafka brokers.
The authentication and authorization mechanisms must match the specification for the listener used to access the Kafka brokers.
For more information on configuring a KafkaUser resource to access Kafka brokers securely, see Section 16.4, “Example: Setting up secure client access”.
16.3.1. Associating users with Kafka clusters
A KafkaUser resource includes a label that defines the appropriate name of the Kafka cluster (derived from the name of the Kafka resource) to which it belongs.
The label enables the User Operator to identify the KafkaUser resource and create and manager the user.
If the label does not match the Kafka cluster, the User Operator cannot identify the KafkaUser, and the user is not created.
If the status of the KafkaUser resource remains empty, check your label configuration.
16.3.2. Configuring user authentication
Use the KafkaUser custom resource to configure authentication credentials for users (clients) that require access to a Kafka cluster. Configure the credentials using the authentication property in KafkaUser.spec. By specifying a type, you control what credentials are generated.
Supported authentication types:
-
tlsfor mTLS authentication -
tls-externalfor mTLS authentication using external certificates -
scram-sha-512for SCRAM-SHA-512 authentication
If tls or scram-sha-512 is specified, the User Operator creates authentication credentials when it creates the user. If tls-external is specified, the user still uses mTLS, but no authentication credentials are created. Use this option when you’re providing your own certificates. When no authentication type is specified, the User Operator does not create the user or its credentials.
You can use tls-external to authenticate with mTLS using a certificate issued outside the User Operator. The User Operator does not generate a TLS certificate or a secret. You can still manage ACL rules and quotas through the User Operator in the same way as when you’re using the tls mechanism. This means that you use the CN=USER-NAME format when specifying ACL rules and quotas. USER-NAME is the common name given in a TLS certificate.
16.3.2.1. mTLS authentication
To use mTLS authentication, you set the type field in the KafkaUser resource to tls.
Example user with mTLS authentication enabled
The authentication type must match the equivalent configuration for the Kafka listener used to access the Kafka cluster.
When the user is created by the User Operator, it creates a new secret with the same name as the KafkaUser resource. The secret contains a private and public key for mTLS. The public key is contained in a user certificate, which is signed by a clients CA (certificate authority) when it is created. All keys are in X.509 format.
If you are using the clients CA generated by the Cluster Operator, the user certificates generated by the User Operator are also renewed when the clients CA is renewed by the Cluster Operator.
The user secret provides keys and certificates in PEM and PKCS #12 formats.
Example secret with user credentials
When you configure a client, you specify the following:
- Truststore properties for the public cluster CA certificate to verify the identity of the Kafka cluster
- Keystore properties for the user authentication credentials to verify the client
The configuration depends on the file format (PEM or PKCS #12). This example uses PKCS #12 stores, and the passwords required to access the credentials in the stores.
Example client configuration using mTLS in PKCS #12 format
- 1
- The bootstrap server address to connect to the Kafka cluster.
- 2
- The security protocol option when using TLS for encryption.
- 3
- The truststore location contains the public key certificate (
ca.p12) for the Kafka cluster. A cluster CA certificate and password is generated by the Cluster Operator in the<cluster_name>-cluster-ca-certsecret when the Kafka cluster is created. - 4
- The password (
ca.password) for accessing the truststore. - 5
- The keystore location contains the public key certificate (
user.p12) for the Kafka user. - 6
- The password (
user.password) for accessing the keystore.
To use mTLS authentication using a certificate issued outside the User Operator, you set the type field in the KafkaUser resource to tls-external. A secret and credentials are not created for the user.
Example user with mTLS authentication that uses a certificate issued outside the User Operator
16.3.2.3. SCRAM-SHA-512 authentication
To use the SCRAM-SHA-512 authentication mechanism, you set the type field in the KafkaUser resource to scram-sha-512.
Example user with SCRAM-SHA-512 authentication enabled
When the user is created by the User Operator, it creates a new secret with the same name as the KafkaUser resource. The secret contains the generated password in the password key, which is encoded with base64. In order to use the password, it must be decoded.
Example secret with user credentials
Decoding the generated password:
echo "Z2VuZXJhdGVkcGFzc3dvcmQ=" | base64 --decode
echo "Z2VuZXJhdGVkcGFzc3dvcmQ=" | base64 --decode16.3.2.3.1. Custom password configuration
When a user is created, Streams for Apache Kafka generates a random password. You can use your own password instead of the one generated by Streams for Apache Kafka. To do so, create a secret with the password and reference it in the KafkaUser resource.
Example user with a password set for SCRAM-SHA-512 authentication
16.3.3. Configuring user authorization
Use the KafkaUser custom resource to configure authorization rules for users (clients) that require access to a Kafka cluster. Configure the rules using the authorization property in KafkaUser.spec. By specifying a type, you control what rules are used.
To use simple authorization, you set the type property to simple in KafkaUser.spec.authorization. The simple authorization uses the Kafka Admin API to manage the ACL rules inside your Kafka cluster. Whether ACL management in the User Operator is enabled or not depends on your authorization configuration in the Kafka cluster.
- For simple authorization, ACL management is always enabled.
- For OPA authorization (deprecated), ACL management is always disabled.
- For Red Hat build of Keycloak authorization, you can manage the ACL rules directly in Red Hat build of Keycloak. You can also delegate authorization to the simple authorizer as a fallback option in the configuration. When delegation to the simple authorizer is enabled, the User Operator will enable management of ACL rules as well.
-
For custom authorization using a custom authorization plugin, use the
supportsAdminApiproperty in the.spec.kafka.authorizationconfiguration of theKafkacustom resource to enable or disable the support.
Authorization is cluster-wide. The authorization type must match the equivalent configuration in the Kafka custom resource.
If ACL management is not enabled, Streams for Apache Kafka rejects a resource if it contains any ACL rules.
If you’re using a standalone deployment of the User Operator, ACL management is enabled by default. You can disable it using the STRIMZI_ACLS_ADMIN_API_SUPPORTED environment variable.
If no authorization is specified, the User Operator does not provision any access rights for the user. Whether such a KafkaUser can still access resources depends on the authorizer being used. For example, for simple authorization, this is determined by the allow.everyone.if.no.acl.found configuration in the Kafka cluster.
16.3.4. Defining ACL rules
The simple authorization mechanism uses ACL rules to manage access to Kafka brokers. Enabling simple authorization for a Kafka cluster means access is denied unless explicit ACL rules are configured in the KafkaUser resource. ACL rules define access rights for users, which you specify using the acls property.
Example user with ACL rules configuration
For more information about the AclRule object and the properties used to configure ACLs, see the AclRule schema reference.
16.3.5. Configuring user quotas
Configure the spec for the KafkaUser resource to enforce quotas so that a user does not overload Kafka brokers. Set size-based network usage and time-based CPU utilization thresholds.
Partition mutations occur in response to the following types of user requests:
- Creating partitions for a new topic
- Adding partitions to an existing topic
- Deleting partitions from a topic
You can also add a partition mutation quota to control the rate at which requests to change partitions are accepted.
Example KafkaUser with user quotas
- 1
- Byte-per-second quota on the amount of data the user can push to a Kafka broker.
- 2
- Byte-per-second quota on the amount of data the user can fetch from a Kafka broker.
- 3
- CPU utilization limit as a percentage of time for a client group.
- 4
- Number of concurrent partition creation and deletion operations (mutations) allowed per second.
Using quotas for Kafka clients might be useful in a number of situations. Consider a wrongly configured Kafka producer which is sending requests at too high a rate. Such misconfiguration can cause a denial of service to other clients, so the problematic client ought to be blocked. By using a network limiting quota, it is possible to prevent this situation from significantly impacting other clients.
Streams for Apache Kafka supports user-level quotas, but not client-level quotas.
16.4. Example: Setting up secure client access
This procedure shows how to configure client access to a Kafka cluster from outside OpenShift or from another OpenShift cluster. It’s split into two parts:
- Securing Kafka brokers
- Securing user access to Kafka
Resource configuration
Client access to the Kafka cluster is secured with the following configuration:
-
An external listener is configured with TLS encryption and mutual TLS (mTLS) authentication in the
Kafkaresource, as well assimpleauthorization. -
A
KafkaUseris created for the client, utilizing mTLS authentication, and Access Control Lists (ACLs) are defined forsimpleauthorization.
At least one listener supporting the desired authentication must be configured for the KafkaUser.
Listeners can be configured for mutual TLS, SCRAM-SHA-512, or OAuth authentication. While mTLS always uses encryption, it’s also recommended when using SCRAM-SHA-512 and OAuth 2.0 authentication.
Authorization options for Kafka include simple, OAuth, OPA, or custom. When enabled, authorization is applied to all enabled listeners.
To ensure compatibility between Kafka and clients, configuration of the following authentication and authorization mechanisms must align:
-
For
type: tlsandtype: scram-sha-512authentication types,Kafka.spec.kafka.listeners[*].authenticationmust matchKafkaUser.spec.authentication -
For
type: simpleauthorization,Kafka.spec.kafka.authorizationmust matchKafkaUser.spec.authorization
For example, mTLS authentication for a user is only possible if it’s also enabled in the Kafka configuration.
Automation and certificate management
Streams for Apache Kafka operators automate the configuration process and create the certificates required for authentication:
- The Cluster Operator creates the listeners and sets up the cluster and client certificate authority (CA) certificates to enable authentication within the Kafka cluster.
- The User Operator creates the user representing the client and the security credentials used for client authentication, based on the chosen authentication type.
You add the certificates to your client configuration.
In this procedure, the CA certificates generated by the Cluster Operator are used. Alternatively, you can replace them by installing your own custom CA certificates. You can also configure listeners to use Kafka listener certificates managed by an external CA.
Certificates are available in PEM (.crt) and PKCS #12 (.p12) formats. This procedure uses PEM certificates. Use PEM certificates with clients that support the X.509 certificate format.
For internal clients in the same OpenShift cluster and namespace, you can mount the cluster CA certificate in the pod specification. For more information, see Configuring internal clients to trust the cluster CA.
Prerequisites
- The Kafka cluster is available for connection by a client running outside the OpenShift cluster
- The Cluster Operator and User Operator are running in the cluster
16.4.1. Securing Kafka brokers
Configure the Kafka cluster with a Kafka listener.
- Define the authentication required to access the Kafka broker through the listener.
Enable authorization on the Kafka broker.
Example listener configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Configuration options for enabling external listeners are described in the Generic Kafka listener schema reference.
- 2
- Name to identify the listener. Must be unique within the Kafka cluster.
- 3
- Port number used by the listener inside Kafka. The port number has to be unique within a given Kafka cluster. Allowed port numbers are 9092 and higher with the exception of ports 9404 and 9999, which are already used for Prometheus and JMX. Depending on the listener type, the port number might not be the same as the port number that connects Kafka clients.
- 4
- External listener type specified as
route(OpenShift only),loadbalancer,nodeportoringress(Kubernetes only). An internal listener is specified asinternalorcluster-ip. - 5
- Required. TLS encryption on the listener. For
routeandingresstype listeners it must be set totrue. For mTLS authentication, also use theauthenticationproperty. - 6
- Client authentication mechanism on the listener. For server and client authentication using mTLS, you specify
tls: trueandauthentication.type: tls. - 7
- (Optional) Depending on the requirements of the listener type, you can specify additional listener configuration.
- 8
- Authorization specified as
simple, which uses theStandardAuthorizerKafka plugin. - 9
- (Optional) Super users can access all brokers regardless of any access restrictions defined in ACLs.
WarningAn OpenShift route address comprises the Kafka cluster name, the listener name, the project name, and the domain of the router. For example,
my-cluster-kafka-external1-bootstrap-my-project.domain.com(<cluster_name>-kafka-<listener_name>-bootstrap-<namespace>.<domain>). Each DNS label (between periods “.”) must not exceed 63 characters, and the total length of the address must not exceed 255 characters.
Apply the changes to the
Kafkaresource configuration.The Kafka cluster is configured with a Kafka broker listener using mTLS authentication.
A service is created for each Kafka broker pod.
A service is created to serve as the bootstrap address for connection to the Kafka cluster.
A service is also created as the external bootstrap address for external connection to the Kafka cluster using
nodeportlisteners.The cluster CA certificate to verify the identity of the kafka brokers is also created in the secret
<cluster_name>-cluster-ca-cert.NoteIf you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.
Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafkaresource.oc get kafka <kafka_cluster_name> -o=jsonpath='{.status.listeners[?(@.name=="<listener_name>")].bootstrapServers}{"\n"}'oc get kafka <kafka_cluster_name> -o=jsonpath='{.status.listeners[?(@.name=="<listener_name>")].bootstrapServers}{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example:
oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Use the bootstrap address in your Kafka client to connect to the Kafka cluster.
16.4.2. Securing user access to Kafka
Create or modify a user representing the client that requires access to the Kafka cluster.
-
Specify the same authentication type as the
Kafkalistener. Specify the authorization ACLs for
simpleauthorization.Example user configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Specify the same authentication type as the
Apply the changes to the
KafkaUserresource configuration.The user is created, as well as a secret with the same name as the
KafkaUserresource. The secret contains a public and private key for mTLS authentication.Example secret with user credentials
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the cluster CA certificate from the
<cluster_name>-cluster-ca-certsecret of the Kafka cluster.oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtoc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the user CA certificate from the
<user_name>secret.oc get secret <user_name> -o jsonpath='{.data.user\.crt}' | base64 -d > user.crtoc get secret <user_name> -o jsonpath='{.data.user\.crt}' | base64 -d > user.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the private key of the user from the
<user_name>secret.oc get secret <user_name> -o jsonpath='{.data.user\.key}' | base64 -d > user.keyoc get secret <user_name> -o jsonpath='{.data.user\.key}' | base64 -d > user.keyCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure your client with the bootstrap address hostname and port for connecting to the Kafka cluster:
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<hostname>:<port>");
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<hostname>:<port>");Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure your client with the truststore credentials to verify the identity of the Kafka cluster.
Specify the public cluster CA certificate.
Example truststore configuration
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG, "<ca.crt_file_content>");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG, "<ca.crt_file_content>");Copy to Clipboard Copied! Toggle word wrap Toggle overflow SSL is the specified security protocol for mTLS authentication. Specify
SASL_SSLfor SCRAM-SHA-512 authentication over TLS. PEM is the file format of the truststore.Configure your client with the keystore credentials to verify the user when connecting to the Kafka cluster.
Specify the public certificate and private key.
Example keystore configuration
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "<user.crt_file_content>"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "<user.key_file_content>");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "<user.crt_file_content>"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "<user.key_file_content>");Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the keystore certificate and the private key directly to the configuration. Add as a single-line format. Between the
BEGIN CERTIFICATEandEND CERTIFICATEdelimiters, start with a newline character (\n). End each line from the original certificate with\ntoo.Example keystore configuration
props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "-----BEGIN CERTIFICATE----- \n<user_certificate_content_line_1>\n<user_certificate_content_line_n>\n-----END CERTIFICATE---"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "----BEGIN PRIVATE KEY-----\n<user_key_content_line_1>\n<user_key_content_line_n>\n-----END PRIVATE KEY-----");
props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "-----BEGIN CERTIFICATE----- \n<user_certificate_content_line_1>\n<user_certificate_content_line_n>\n-----END CERTIFICATE---"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "----BEGIN PRIVATE KEY-----\n<user_key_content_line_1>\n<user_key_content_line_n>\n-----END PRIVATE KEY-----");Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Off-cluster access using node ports with TLS encryption enabled does not support TLS hostname verification. This is because Streams for Apache Kafka does not know the address of the node where the broker pod is scheduled and cannot include it in the broker certificate. Consequently, clients that perform hostname verification will fail to connect.
For example, a Java client will fail with the following exception:
Exception for TLS hostname verification
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 168.72.15.231 found ...
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 168.72.15.231 found
...
To connect, you must disable hostname verification. In the Java client, set the ssl.endpoint.identification.algorithm configuration option to an empty string.
When configuring the client using a properties file, you can do it this way:
ssl.endpoint.identification.algorithm=
ssl.endpoint.identification.algorithm=When configuring the client directly in Java, set the configuration option to an empty string:
props.put("ssl.endpoint.identification.algorithm", "");
props.put("ssl.endpoint.identification.algorithm", "");
Alternatively, if you know the addresses of the worker nodes where the brokers are scheduled, you can add them as additional SANs (Subject Alternative Names) to the broker certificates manually. For example, this might apply if your cluster is running on a bare metal deployment with a limited number of available worker nodes. Use the alternativeNames property to specify additional SANS.
Chapter 17. Enabling OAuth 2.0 token-based access
Streams for Apache Kafka supports OAuth 2.0 for securing Kafka clusters by integrating with an OAuth 2.0 authorization server. Kafka brokers and clients both need to be configured to use OAuth 2.0.
OAuth 2.0 enables standardized token-based authentication and authorization between applications, using a central authorization server to issue tokens that grant limited access to resources. You can define specific scopes for fine-grained access control. Scopes correspond to different levels of access to Kafka topics or operations within the cluster.
OAuth 2.0 also supports single sign-on and integration with identity providers.
17.1. Configuring an OAuth 2.0 authorization server
Before you can use OAuth 2.0 token-based access, you must configure an authorization server for integration with Streams for Apache Kafka. The steps are dependent on the chosen authorization server. Consult the product documentation for the authorization server for information on how to set up OAuth 2.0 access.
Prepare the authorization server to work with Streams for Apache Kafka by defining OAuth 2.0 clients for Kafka and each Kafka client component of your application. In relation to the authorization server, the Kafka cluster and Kafka clients are both regarded as OAuth 2.0 clients.
In general, configure OAuth 2.0 clients in the authorization server with the following client credentials enabled:
-
Client ID (for example,
kafkafor the Kafka cluster) - Client ID and secret as the authentication mechanism
You only need to use a client ID and secret when using a non-public introspection endpoint of the authorization server. The credentials are not typically required when using public authorization server endpoints, as with fast local JWT token validation.
17.2. Using OAuth 2.0 token-based authentication
Streams for Apache Kafka supports the use of OAuth 2.0 for token-based authentication. An OAuth 2.0 authorization server handles the granting of access and inquiries about access. Kafka clients authenticate to Kafka brokers. Brokers and clients communicate with the authorization server, as necessary, to obtain or validate access tokens.
For a deployment of Streams for Apache Kafka, OAuth 2.0 integration provides the following support:
- Server-side OAuth 2.0 authentication for Kafka brokers
- Client-side OAuth 2.0 authentication for Kafka MirrorMaker, Kafka Connect, and the Kafka Bridge
To secure Kafka brokers with OAuth 2.0 authentication, configure a listener in the Kafka resource to use OAuth 2.0 authentication and a client authentication mechanism, and add further configuration depending on the authentication mechanism and type of token validation used in the authentication.
Configuring listeners to use oauth authentication
Specify a listener in the Kafka resource with an oauth authentication type. You can configure internal and external listeners. We recommend using OAuth 2.0 authentication together with TLS encryption (tls: true). Without encryption, the connection is vulnerable to network eavesdropping and unauthorized access through token theft.
Example listener configuration with OAuth 2.0 authentication
Enabling SASL authentication mechanisms
Use one or both of the following SASL mechanisms for clients to exchange credentials and establish authenticated sessions with Kafka.
OAUTHBEARERUsing the
OAUTHBEARERauthentication mechanism, credentials exchange uses a bearer token provided by an OAuth callback handler. Token provision can be configured to use the following methods:- Client ID and secret (using the OAuth 2.0 client credentials mechanism)
- Client ID and client assertion
- Long-lived access token or Service account token
- Long-lived refresh token obtained manually
OAUTHBEARERis recommended as it provides a higher level of security thanPLAIN, though it can only be used by Kafka clients that support theOAUTHBEARERmechanism at the protocol level. Client credentials are never shared with Kafka.PLAINPLAINis a simple authentication mechanism used by all Kafka client tools. Consider usingPLAINonly with Kafka clients that do not supportOAUTHBEARER. Using thePLAINauthentication mechanism, credentials exchange can be configured to use the following methods:- Client ID and secret (using the OAuth 2.0 client credentials mechanism)
-
Long-lived access token
Regardless of the method used, the client must provideusernameandpasswordproperties to Kafka.
Credentials are handled centrally behind a compliant authorization server, similar to how
OAUTHBEARERauthentication is used. The username extraction process depends on the authorization server configuration.
OAUTHBEARER is automatically enabled in the oauth listener configuration for the Kafka broker. To use the PLAIN mechanism, you must set the enablePlain property to true.
In the following example, the PLAIN mechanism is enabled, and the OAUTHBEARER mechanism is disabled on a listener using the enableOauthBearer property.
Example listener configuration for the PLAIN mechanism
When you have defined the type of authentication as OAuth 2.0, you add configuration based on the type of validation, either as fast local JWT validation or token validation using an introspection endpoint.
Configuring fast local JWT token validation
Fast local JWT token validation involves checking a JWT token signature locally to ensure that the token meets the following criteria:
-
Contains a
typ(type) ortoken_typeheader claim value ofBearerto indicate it is an access token - Is currently valid and not expired
-
Has an issuer that matches a
validIssuerURI
You specify a validIssuerURI attribute when you configure the listener, so that any tokens not issued by the authorization server are rejected.
The authorization server does not need to be contacted during fast local JWT token validation. You activate fast local JWT token validation by specifying a jwksEndpointUri attribute, the endpoint exposed by the OAuth 2.0 authorization server. The endpoint contains the public keys used to validate signed JWT tokens, which are sent as credentials by Kafka clients.
All communication with the authorization server should be performed using TLS encryption. You can configure a certificate truststore as an OpenShift Secret in your Streams for Apache Kafka project namespace, and use the tlsTrustedCertificates property to point to the OpenShift secret containing the truststore file.
You might want to configure a userNameClaim to properly extract a username from the JWT token. If required, you can use a JsonPath expression like "['user.info'].['user.id']" to retrieve the username from nested JSON attributes within a token.
If you want to use Kafka ACL authorization, identify the user by their username during authentication. (The sub claim in JWT tokens is typically a unique ID, not a username.)
Example configuration for fast local JWT token validation
- 1
- Listener type set to
oauth. - 2
- URI of the token issuer used for authentication.
- 3
- URI of the JWKS certificate endpoint used for local JWT validation.
- 4
- The token claim (or key) that contains the actual username used to identify the user. Its value depends on the authorization server. If necessary, a JsonPath expression like
"['user.info'].['user.id']"can be used to retrieve the username from nested JSON attributes within a token. - 5
- (Optional) Activates the Kafka re-authentication mechanism that enforces session expiry to the same length of time as the access token. If the specified value is less than the time left for the access token to expire, then the client will have to re-authenticate before the actual token expiry. By default, the session does not expire when the access token expires, and the client does not attempt re-authentication.
- 6
- (Optional) Certificates stored in X.509 format within the specified secrets for TLS connection to the authorization server.
- 7
- (Optional) Disable TLS hostname verification. Default is
false. - 8
- The duration the JWKS certificates are considered valid before they expire. Default is
360seconds. If you specify a longer time, consider the risk of allowing access to revoked certificates. - 9
- The period between refreshes of JWKS certificates. The interval must be at least 60 seconds shorter than the expiry interval. Default is
300seconds. - 10
- The minimum pause in seconds between consecutive attempts to refresh JWKS public keys. When an unknown signing key is encountered, the JWKS keys refresh is scheduled outside the regular periodic schedule with at least the specified pause since the last refresh attempt. The refreshing of keys follows the rule of exponential backoff, retrying on unsuccessful refreshes with ever increasing pause, until it reaches
jwksRefreshSeconds. The default value is 1.
Configuring fast local JWT token validation with OpenShift service accounts
To configure the listener for OpenShift service accounts, the Kubernetes API server must be used as the authorization server.
Example configuration for fast local JWT token validation using Kubernetes API server as authorization server
- 1
- URI of the token issuer used for authentication. Must use FQDN, including the
.cluster.localextension, which may vary based on the OpenShift cluster configuration. - 2
- URI of the JWKS certificate endpoint used for local JWT validation. Must use FQDN, including the
.cluster.localextension, which may vary based on the OpenShift cluster configuration. - 3
- Location to the access token used by the Kafka broker to authenticate to the Kubernetes API server in order to access the
jwksEndpointUri. - 4
- Skip the access token type check, as the claim for this is not present in service account tokens.
- 5
- Skip sending
Acceptheader in HTTP requests to the JWKS endpoint, as the Kubernetes API server does not support it. - 6
- Trusted certificates to connect to authorization server. This should point to a manually created Secret that contains the Kubernetes API server public certificate, which is mounted to the running pods under
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt. You can use the following command to create the Secret:oc get cm kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" > /tmp/ca.crt oc create secret generic oauth-server-cert --from-file=ca.crt=/tmp/ca.crtoc get cm kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" > /tmp/ca.crt oc create secret generic oauth-server-cert --from-file=ca.crt=/tmp/ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow - 7
- (Optional) Additional constraints that JWT token has to fulfill in order to be accepted, expressed as JsonPath filter query. In this example the service account has to belong to
myprojectnamespace in order to be allowed to authenticate.
The above configuration uses the sub claim from the service account JWT token as the user ID. For example, the default service account for pods deployed in the myproject namespace has the username: system:serviceaccount:myproject:default.
When configuring ACLs the general form of how to refer to the ServiceAccount user should in that case be: User:system:serviceaccount:<Namespace>:<ServiceAccount-name>
Configuring token validation using an introspection endpoint
Token validation using an OAuth 2.0 introspection endpoint treats a received access token as opaque. The Kafka broker sends an access token to the introspection endpoint, which responds with the token information necessary for validation. Importantly, it returns up-to-date information if the specific access token is valid, and also information about when the token expires.
To configure OAuth 2.0 introspection-based validation, you specify an introspectionEndpointUri attribute rather than the jwksEndpointUri attribute specified for fast local JWT token validation. Depending on the authorization server, you typically have to specify a clientId and clientSecret, because the introspection endpoint is usually protected.
Example token validation configuration using an introspection endpoint
- 1
- URI of the token introspection endpoint.
- 2
- Client ID to identify the client.
- 3
- Client Secret and client ID is used for authentication.
- 4
- The token claim (or key) that contains the actual username used to identify the user. Its value depends on the authorization server. If necessary, a JsonPath expression like
"['user.info'].['user.id']"can be used to retrieve the username from nested JSON attributes within a token. - 5
- (Optional) Activates the Kafka re-authentication mechanism that enforces session expiry to the same length of time as the access token. If the specified value is less than the time left for the access token to expire, then the client will have to re-authenticate before the actual token expiry. By default, the session does not expire when the access token expires, and the client does not attempt re-authentication.
Authenticating brokers to the authorization server protected endpoints
Usually, the certificates endpoint of the authorization server (jwksEndpointUri) is publicly accessible, while the introspection endpoint (introspectionEndpointUri) is protected. However, this may vary depending on the authorization server configuration.
The Kafka broker can authenticate to the authorization server’s protected endpoints in one of two ways using HTTP authentication schemes:
- HTTP Basic authentication uses a client ID and secret.
- HTTP Bearer authentication uses a bearer token.
To configure HTTP Basic authentication, set the following properties:
-
clientId -
clientSecret
For HTTP Bearer authentication, set the following property:
-
serverBearerTokenLocationto specify the file path on disk containing the bearer token.
Including additional configuration options
Specify additional settings depending on the authentication requirements and the authorization server you are using. Some of these properties apply only to certain authentication mechanisms or when used in combination with other properties.
For example, when using OAuth over PLAIN, access tokens are passed as password property values with or without an $accessToken: prefix.
-
If a
tokenEndpointUriis configured in the listener, the$accessToken:prefix must be used. -
If no
tokenEndpointUriis configured, the prefix is not required, and thepasswordis interpreted directly as a raw access token.
When an access token is used as the password, the username must match the principal name extracted from that token. You can specify username extraction options in your listener using the userNameClaim, usernamePrefix, fallbackUserNameClaim, fallbackUsernamePrefix, and userInfoEndpointUri properties. The username extraction process also depends on your authorization server; in particular, how it maps client IDs to account names.
Example optional configuration settings
- 1
- If your authorization server does not provide an
issclaim, it is not possible to perform an issuer check. In this situation, setcheckIssuertofalseand do not specify avalidIssuerUri. Default istrue. - 2
- If your authorization server provides an
aud(audience) claim, and you want to enforce an audience check, setcheckAudiencetotrue. Audience checks identify the intended recipients of tokens. As a result, the Kafka broker will reject tokens that do not have itsclientIdin theiraudclaim. Default isfalse. - 3
- The prefix used when constructing the user ID. This only takes effect if
userNameClaimis configured. - 4
- An authorization server may not provide a single attribute to identify both regular users and clients. When a client authenticates in its own name, the server might provide a client ID. When a user authenticates using a username and password to obtain a refresh token or an access token, the server might provide a username attribute in addition to a client ID. Use this fallback option to specify the username claim (attribute) to use if a primary user ID attribute is not available. If necessary, a JsonPath expression like
"['client.info'].['client.id']"can be used to retrieve the fallback username to retrieve the username from nested JSON attributes within a token. - 5
- In situations where
fallbackUserNameClaimis applicable, it may also be necessary to prevent name collisions between the values of the username claim, and those of the fallback username claim. Consider a situation where a client calledproducerexists, but also a regular user calledproducerexists. In order to differentiate between the two, you can use this property to add a prefix to the user ID of the client. - 6
- The location of the access token used by the Kafka broker to authenticate to the Kubernetes API server for accessing protected endpoints. The authorization server must support
OAUTHBEARERauthentication. This is an alternative to specifyingclientIdandclientSecret, which usesPLAINauthentication. - 7
- (Only applicable when using
introspectionEndpointUri) Depending on the authorization server you are using, the introspection endpoint may or may not return the token type attribute, or it may contain different values. You can specify a valid token type value that the response from the introspection endpoint has to contain. - 8
- (Only applicable when using
introspectionEndpointUri) The authorization server may be configured or implemented in such a way to not provide any identifiable information in an introspection endpoint response. In order to obtain the user ID, you can configure the URI of theuserinfoendpoint as a fallback. TheuserNameClaim,fallbackUserNameClaim, andfallbackUserNamePrefixsettings are applied to the response ofuserinfoendpoint. - 9
- Set this to
falseto disable theOAUTHBEARERmechanism on the listener. At least one ofPLAINorOAUTHBEARERhas to be enabled. Default istrue. - 10
- Set to
trueto enablePLAINauthentication on the listener, which is supported for clients on all platforms. - 11
- Additional configuration for the
PLAINmechanism. If specified, clients can authenticate overPLAINby passing an access token as thepasswordusing an$accessToken:prefix. For production, always usehttps://urls. - 12
- Additional custom rules can be imposed on the JWT access token during validation by setting this to a JsonPath filter query. If the access token does not contain the necessary data, it is rejected. When using the
introspectionEndpointUri, the custom check is applied to the introspection endpoint response JSON. - 13
- Grant type used when requesting a token from the authorization server. Applicable only with OAuth over PLAIN when the
usernameandpasswordin the client configuration are passed as theclientIdandsecret. - 14
- An
audienceparameter passed to the token endpoint. An audience is used when obtaining an access token for inter-broker authentication. It is also used in the name of a client for OAuth overPLAINclient authentication using aclientIdandsecret. This only affects the ability to obtain the token, and the content of the token, depending on the authorization server. It does not affect token validation rules by the listener. - 15
- A
scopeparameter passed to the token endpoint. A scope is used when obtaining an access token for inter-broker authentication. It is also used in the name of a client for OAuth overPLAINclient authentication using aclientIdandsecret. This only affects the ability to obtain the token, and the content of the token, depending on the authorization server. It does not affect token validation rules by the listener. - 16
- The connect timeout in seconds when connecting to the authorization server. The default value is 60.
- 17
- The read timeout in seconds when connecting to the authorization server. The default value is 60.
- 18
- The maximum number of times to retry a failed HTTP request to the authorization server. The default value is
0, meaning that no retries are performed. To use this option effectively, consider reducing the timeout times for theconnectTimeoutSecondsandreadTimeoutSecondsoptions. However, note that retries may prevent the current worker thread from being available to other requests, and if too many requests stall, it could make the Kafka broker unresponsive. - 19
- The time to wait before attempting another retry of a failed HTTP request to the authorization server. By default, this time is set to zero, meaning that no pause is applied. This is because many issues that cause failed requests are per-request network glitches or proxy issues that can be resolved quickly. However, if your authorization server is under stress or experiencing high traffic, you may want to set this option to a value of 100 ms or more to reduce the load on the server and increase the likelihood of successful retries.
- 20
- A JsonPath query that is used to extract groups information from either the JWT token or the introspection endpoint response. This option is not set by default. By configuring this option, a custom authorizer can make authorization decisions based on user groups.
- 21
- A delimiter used to parse groups information when it is returned as a single delimited string. The default value is ',' (comma).
- 22
- Some authorization servers have issues with client sending
Accept: application/jsonheader. By settingincludeAcceptHeader: falsethe header will not be sent. Default istrue.
17.2.2. Configuring OAuth 2.0 on client applications
To configure OAuth 2.0 on client applications, you must specify the following:
- SASL (Simple Authentication and Security Layer) security protocols
- SASL mechanisms
- A JAAS (Java Authentication and Authorization Service) module
- Authentication properties to access the authorization server
Configuring SASL protocols
Specify SASL protocols in the client configuration:
-
SASL_SSLfor authentication over TLS encrypted connections -
SASL_PLAINTEXTfor authentication over unencrypted connections
Use SASL_SSL for production and SASL_PLAINTEXT for local development only.
When using SASL_SSL, additional ssl.truststore configuration is needed. The truststore configuration is required for secure connection (https://) to the OAuth 2.0 authorization server. To verify the OAuth 2.0 authorization server, add the CA certificate for the authorization server to the truststore in your client configuration. You can configure a truststore in PEM or PKCS #12 format.
Configuring SASL authentication mechanisms
Specify SASL mechanisms in the client configuration:
-
OAUTHBEARERfor credentials exchange using a bearer token -
PLAINto pass client credentials (clientId + secret) or an access token
Configuring a JAAS module
Specify a JAAS module that implements the SASL authentication mechanism as a sasl.jaas.config property value:
-
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModuleimplements theOAUTHBEARERmechanism -
org.apache.kafka.common.security.plain.PlainLoginModuleimplements thePLAINmechanism
For the OAUTHBEARER mechanism, Streams for Apache Kafka provides a callback handler for clients that use Kafka Client Java libraries to enable credentials exchange. For clients in other languages, custom code may be required to obtain the access token. For the PLAIN mechanism, Streams for Apache Kafka provides server-side callbacks to enable credentials exchange.
To be able to use the OAUTHBEARER mechanism, you must also add the custom io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler class as the callback handler. JaasClientOauthLoginCallbackHandler handles OAuth callbacks to the authorization server for access tokens during client login. This enables automatic token renewal, ensuring continuous authentication without user intervention. Additionally, it handles login credentials for clients using the OAuth 2.0 password grant method.
Configuring authentication properties
Configure the client to use credentials or access tokens for OAuth 2.0 authentication.
- Using client credentials
-
Using client credentials involves configuring the client with the necessary credentials (client ID and secret, or client ID and client assertion) to obtain a valid access token from an authorization server. This is the simplest mechanism. When you configure a client ID and secret or assertion to authenticate against the authorization server, you can specify a
grantType. The property defaults toclient_credentials. - Using access tokens
- Using access tokens, the client is configured with a valid long-lived access token or refresh token obtained from an authorization server. Using access tokens adds more complexity because there is an additional dependency on authorization server tools. If you are using long-lived access tokens, you may need to configure the client in the authorization server to increase the maximum lifetime of the token.
The only information ever sent to Kafka is the access token. The credentials used to obtain the token are never sent to Kafka. When a client obtains an access token, no further communication with the authorization server is needed.
SASL authentication properties support the following authentication methods:
- OAuth 2.0 client credentials
- Access token or Service account token
- Refresh token
- OAuth 2.0 password grant (deprecated)
Add the authentication properties as JAAS configuration (sasl.jaas.config and sasl.login.callback.handler.class).
If the client application is not configured with an access token directly, the client exchanges one of the following sets of credentials for an access token during Kafka session initiation:
- Client ID and secret
- Client ID and client assertion
- Client ID, refresh token, and (optionally) a secret
- Username and password, with client ID and (optionally) a secret
You can also specify authentication properties as environment variables, or as Java system properties. For Java system properties, you can set them using setProperty and pass them on the command line using the -D option.
Example client credentials configuration using the client secret
- 1
SASL_SSLsecurity protocol for TLS-encrypted connections. UseSASL_PLAINTEXTover unencrypted connections for local development only.- 2
- The SASL mechanism specified as
OAUTHBEARERorPLAIN. - 3
- The truststore configuration for secure access to the Kafka cluster.
- 4
- URI of the authorization server token endpoint.
- 5
- Client ID, which is the name used when creating the client in the authorization server.
- 6
- Client secret created when creating the client in the authorization server.
- 7
- (Optional) Grant type when using
clientIdandclientSecretorclientAssertion. Sets the grant type parameter sent to the token endpoint. Defaults toclient_credentialswith this configuration. - 8
- The location of the truststore for the authorization server.
- 9
- The password for accessing the truststore.
- 10
- The truststore type.
- 11
- (Optional) The
scopefor requesting the token from the token endpoint. An authorization server may require a client to specify the scope. - 12
- (Optional) The
audiencefor requesting the token from the token endpoint. An authorization server may require a client to specify the audience.
Example client credentials configuration using the client assertion
- 1
- Path to the client assertion file used for authenticating the client. This file is a private key file as an alternative to the client secret. Alternatively, use the
oauth.client.assertionoption to specify the client assertion value in clear text. - 2
- (Optional) Sometimes you may need to specify the client assertion type. In not specified, the default value is
urn:ietf:params:oauth:client-assertion-type:jwt-bearer.
Example password grants configuration
- 1
- Client ID, which is the name used when creating the client in the authorization server.
- 2
- (Optional) Client secret created when creating the client in the authorization server.
- 3
- Username for password grant authentication. OAuth password grant configuration (username and password) uses the OAuth 2.0 password grant method. To use password grants, create a user account for a client on your authorization server with limited permissions. The account should act like a service account. Use in environments where user accounts are required for authentication, but consider using a refresh token first.
- 4
- Password for password grant authentication.Note
SASL
PLAINdoes not support passing a username and password (password grants) using the OAuth 2.0 password grant method.
Example access token configuration
- 1
- Long-lived access token for Kafka clients. Alternatively,
oauth.access.token.locationcan be used to specify the file that contains the access token.
Example OpenShift service account token configuration
- 1
- Location to the service account token on the filesystem (assuming that the client is deployed as an OpenShift pod)
Example refresh token configuration
SASL extensions for custom OAUTHBEARER implementations
If your Kafka broker uses a custom OAUTHBEARER implementation, you may need to pass additional SASL extension options. These extensions can include attributes or information required as client context by the authorization server. The options are passed as key-value pairs and are sent to the Kafka broker when a new session is started.
Pass SASL extension values using oauth.sasl.extension. as a key prefix.
Example configuration to pass SASL extension values
oauth.sasl.extension.key1="value1" oauth.sasl.extension.key2="value2"
oauth.sasl.extension.key1="value1"
oauth.sasl.extension.key2="value2"17.2.3. OAuth 2.0 client authentication flows
OAuth 2.0 authentication flows depend on the underlying Kafka client and Kafka broker configuration. The flows must also be supported by the authorization server used.
The Kafka broker listener configuration determines how clients authenticate using an access token. The client can pass a client ID and secret to request an access token.
If a listener is configured to use PLAIN authentication, the client can authenticate with a client ID and secret or username and access token. These values are passed as the username and password properties of the PLAIN mechanism.
Listener configuration supports the following token validation options:
- You can use fast local token validation based on JWT signature checking and local token introspection, without contacting an authorization server. The authorization server provides a JWKS endpoint with public certificates that are used to validate signatures on the tokens.
- You can use a call to a token introspection endpoint provided by an authorization server. Each time a new Kafka broker connection is established, the broker passes the access token received from the client to the authorization server. The Kafka broker checks the response to confirm whether the token is valid.
An authorization server might only allow the use of opaque access tokens, which means that local token validation is not possible.
Kafka client credentials can also be configured for the following types of authentication:
- Direct local access using a previously generated long-lived access token
- Contact with the authorization server for a new access token to be issued (using a client ID and credentials, or a refresh token, or a username and a password)
You can use the following communication flows for Kafka authentication using the SASL OAUTHBEARER mechanism.
Client using client ID and credentials, with broker delegating validation to authorization server
- The Kafka client requests an access token from the authorization server using a client ID and credentials, and optionally a refresh token. Alternatively, the client may authenticate using a username and a password.
- The authorization server generates a new access token.
-
The Kafka client authenticates with the Kafka broker using the SASL
OAUTHBEARERmechanism to pass the access token. - The Kafka broker validates the access token by calling a token introspection endpoint on the authorization server using its own client ID and secret.
- A Kafka client session is established if the token is valid.
Client using client ID and credentials, with broker performing fast local token validation
- The Kafka client authenticates with the authorization server from the token endpoint, using a client ID and credentials, and optionally a refresh token. Alternatively, the client may authenticate using a username and a password.
- The authorization server generates a new access token.
-
The Kafka client authenticates with the Kafka broker using the SASL
OAUTHBEARERmechanism to pass the access token. - The Kafka broker validates the access token locally using a JWT token signature check, and local token introspection.
Client using long-lived access token, with broker delegating validation to authorization server
-
The Kafka client authenticates with the Kafka broker using the SASL
OAUTHBEARERmechanism to pass the long-lived access token. - The Kafka broker validates the access token by calling a token introspection endpoint on the authorization server, using its own client ID and secret.
- A Kafka client session is established if the token is valid.
Client using long-lived access token, with broker performing fast local validation
-
The Kafka client authenticates with the Kafka broker using the SASL
OAUTHBEARERmechanism to pass the long-lived access token. - The Kafka broker validates the access token locally using a JWT token signature check and local token introspection.
Fast local JWT token signature validation is suitable only for short-lived tokens as there is no check with the authorization server if a token has been revoked. Token expiration is written into the token, but revocation can happen at any time, so cannot be accounted for without contacting the authorization server. Any issued token would be considered valid until it expires.
You can use the following communication flows for Kafka authentication using the OAuth PLAIN mechanism.
Client using a client ID and secret, with the broker obtaining the access token for the client
-
The Kafka client passes a
clientIdas a username and asecretas a password. -
The Kafka broker uses a token endpoint to pass the
clientIdandsecretto the authorization server. - The authorization server returns a fresh access token or an error if the client credentials are not valid.
The Kafka broker validates the token in one of the following ways:
- If a token introspection endpoint is specified, the Kafka broker validates the access token by calling the endpoint on the authorization server. A session is established if the token validation is successful.
- If local token introspection is used, a request is not made to the authorization server. The Kafka broker validates the access token locally using a JWT token signature check.
Client using a long-lived access token without a client ID and secret
- The Kafka client passes a username and password. The password provides the value of an access token that was obtained manually and configured before running the client.
The password is passed with or without an
$accessToken:string prefix depending on whether or not the Kafka broker listener is configured with a token endpoint for authentication.-
If the token endpoint is configured, the password should be prefixed by
$accessToken:to let the broker know that the password parameter contains an access token rather than a client secret. The Kafka broker interprets the username as the account username. -
If the token endpoint is not configured on the Kafka broker listener (enforcing a
no-client-credentials mode), the password should provide the access token without the prefix. The Kafka broker interprets the username as the account username. In this mode, the client doesn’t use a client ID and secret, and thepasswordparameter is always interpreted as a raw access token.
-
If the token endpoint is configured, the password should be prefixed by
The Kafka broker validates the token in one of the following ways:
- If a token introspection endpoint is specified, the Kafka broker validates the access token by calling the endpoint on the authorization server. A session is established if token validation is successful.
- If local token introspection is used, there is no request made to the authorization server. Kafka broker validates the access token locally using a JWT token signature check.
17.2.4. Re-authenticating sessions
Configure oauth listeners to use Kafka session re-authentication for OAuth 2.0 sessions between Kafka clients and Kafka. This mechanism enforces the expiry of an authenticated session between the client and the broker after a defined period of time. When a session expires, the client immediately starts a new session by reusing the existing connection rather than dropping it.
Session re-authentication is disabled by default. To enable it, you set a time value for maxSecondsWithoutReauthentication in the oauth listener configuration. The same property is used to configure session re-authentication for OAUTHBEARER and PLAIN authentication. For an example configuration, see Section 17.2.1, “Configuring OAuth 2.0 authentication on listeners”.
Session re-authentication must be supported by the Kafka client libraries used by the client.
Session re-authentication can be used with fast local JWT or introspection endpoint token validation.
Client re-authentication
When the broker’s authenticated session expires, the client must re-authenticate to the existing session by sending a new, valid access token to the broker, without dropping the connection.
If token validation is successful, a new client session is started using the existing connection. If the client fails to re-authenticate, the broker will close the connection if further attempts are made to send or receive messages. Java clients that use Kafka client library 2.2 or later automatically re-authenticate if the re-authentication mechanism is enabled on the broker.
Session re-authentication also applies to refresh tokens, if used. When the session expires, the client refreshes the access token by using its refresh token. The client then uses the new access token to re-authenticate to the existing session.
Session expiry
When session re-authentication is configured, session expiry works differently for OAUTHBEARER and PLAIN authentication.
For OAUTHBEARER and PLAIN, using the client ID and secret method:
-
The broker’s authenticated session will expire at the configured
maxSecondsWithoutReauthentication. - The session will expire earlier if the access token expires before the configured time.
For PLAIN using the long-lived access token method:
-
The broker’s authenticated session will expire at the configured
maxSecondsWithoutReauthentication. -
Re-authentication will fail if the access token expires before the configured time. Although session re-authentication is attempted,
PLAINhas no mechanism for refreshing tokens.
If maxSecondsWithoutReauthentication is not configured, OAUTHBEARER and PLAIN clients can remain connected to brokers indefinitely, without needing to re-authenticate. Authenticated sessions do not end with access token expiry. However, this can be considered when configuring authorization, for example, by using keycloak authorization or installing a custom authorizer.
17.2.5. Example: Enabling OAuth 2.0 authentication
This example shows how to configure client access to a Kafka cluster using OAuth 2.0 authentication. The procedures describe the configuration required to set up OAuth 2.0 authentication on Kafka listeners, Kafka Java clients, and Kafka components.
Configure Kafka listeners so that they are enabled to use OAuth 2.0 authentication using an authorization server.
We advise using OAuth 2.0 over an encrypted interface through through a listener with tls: true. Plain listeners are not recommended.
If the authorization server is using certificates signed by the trusted CA and matching the OAuth 2.0 server hostname, TLS connection works using the default settings. Otherwise, you may need to configure the truststore with proper certificates or disable the certificate hostname validation.
For more information on the configuration of OAuth 2.0 authentication for Kafka broker listeners, see the KafkaListenerAuthenticationOAuth schema reference.
Prerequisites
- Streams for Apache Kafka and Kafka are running
- An OAuth 2.0 authorization server is deployed
Procedure
Specify a listener in the
Kafkaresource with anoauthauthentication type.Example listener configuration with OAuth 2.0 authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the OAuth listener depending on the authorization server and validation type:
-
Apply the changes to the
Kafkaconfiguration. Check the update in the logs or by watching the pod state transitions:
oc logs -f ${POD_NAME} -c ${CONTAINER_NAME} oc get pod -woc logs -f ${POD_NAME} -c ${CONTAINER_NAME} oc get pod -wCopy to Clipboard Copied! Toggle word wrap Toggle overflow The rolling update configures the brokers to use OAuth 2.0 authentication.
What to do next
17.2.5.2. Setting up OAuth 2.0 on Kafka Java clients
Configure Kafka producer and consumer APIs to use OAuth 2.0 for interaction with Kafka brokers. Add a callback plugin to your client pom.xml file, then configure your client for OAuth 2.0.
How you configure the authentication properties depends on the authentication method you are using to access the OAuth 2.0 authorization server. In this procedure, the properties are specified in a properties file, then loaded into the client configuration.
Prerequisites
- Streams for Apache Kafka and Kafka are running
- An OAuth 2.0 authorization server is deployed and configured for OAuth access to Kafka brokers
- Kafka brokers are configured for OAuth 2.0
Procedure
Add the client library with OAuth 2.0 support to the
pom.xmlfile for the Kafka client:<dependency> <groupId>io.strimzi</groupId> <artifactId>kafka-oauth-client</artifactId> <version>0.17.1.redhat-00002</version> </dependency>
<dependency> <groupId>io.strimzi</groupId> <artifactId>kafka-oauth-client</artifactId> <version>0.17.1.redhat-00002</version> </dependency>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the client depending on the OAuth 2.0 authentication method:
For example, specify the properties for the authentication method in a
client.propertiesfile.Input the client properties for OAUTH 2.0 authentication into the Java client code.
Example showing input of client properties
Properties props = new Properties(); try (FileReader reader = new FileReader("client.properties", StandardCharsets.UTF_8)) { props.load(reader); }Properties props = new Properties(); try (FileReader reader = new FileReader("client.properties", StandardCharsets.UTF_8)) { props.load(reader); }Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Verify that the Kafka client can access the Kafka brokers.
17.2.5.3. Setting up OAuth 2.0 on Kafka components
This procedure describes how to set up Kafka components to use OAuth 2.0 authentication using an authorization server.
You can configure OAuth 2.0 authentication for the following components:
- Kafka Connect
- Kafka MirrorMaker
- Kafka Bridge
In this scenario, the Kafka component and the authorization server are running in the same cluster.
Before you start
For more information on the configuration of OAuth 2.0 authentication for Kafka components, see the KafkaClientAuthenticationOAuth schema reference. The schema reference includes examples of configuration options.
Prerequisites
- Streams for Apache Kafka and Kafka are running
- An OAuth 2.0 authorization server is deployed and configured for OAuth access to Kafka brokers
- Kafka brokers are configured for OAuth 2.0
Procedure
Create a client secret and mount it to the component as an environment variable.
For example, here we are creating a client
Secretfor the Kafka Bridge:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The
clientSecretkey must be in base64 format.
Create or edit the resource for the Kafka component so that OAuth 2.0 authentication is configured for the authentication property.
For OAuth 2.0 authentication, you can use the following options:
- Client ID and secret
- Client ID and client assertion
- Client ID and refresh token
- Access token
- Username and password
- TLS
For example, here OAuth 2.0 is assigned to the Kafka Bridge client using a client ID and secret, and TLS:
Example OAuth 2.0 authentication configuration using the client secret
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, OAuth 2.0 is assigned to the Kafka Bridge client using a client ID and the location of a client assertion file, with TLS to connect to the authorization server:
Example OAuth 2.0 authentication configuration using client assertion
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Filesystem path to the client assertion file used for authenticating the client. This file is typically added to the deployed pod by an external operator service. Alternatively, use
clientAssertionto refer to a secret containing the client assertion value.
Here, OAuth 2.0 is assigned to the Kafka Bridge client using a service account token:
Example OAuth 2.0 authentication configuration using the service account token
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Path to the service account token file location.
Depending on how you apply OAuth 2.0 authentication, and the type of authorization server, there are additional configuration options you can use:
Additional configuration options
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- (Optional) Disable TLS hostname verification. Default is
false. - 2
- If you are using opaque tokens, you can apply
accessTokenIsJwt: falseso that access tokens are not treated as JWT tokens. - 3
- (Optional) The
scopefor requesting the token from the token endpoint. An authorization server may require a client to specify the scope. In this case it isany. - 4
- (Optional) The
audiencefor requesting the token from the token endpoint. An authorization server may require a client to specify the audience. In this case it iskafka. - 5
- (Optional) The connect timeout in seconds when connecting to the authorization server. The default value is 60.
- 6
- (Optional) The read timeout in seconds when connecting to the authorization server. The default value is 60.
- 7
- (Optional) The maximum number of times to retry a failed HTTP request to the authorization server. The default value is
0, meaning that no retries are performed. To use this option effectively, consider reducing the timeout times for theconnectTimeoutSecondsandreadTimeoutSecondsoptions. However, note that retries may prevent the current worker thread from being available to other requests, and if too many requests stall, it could make the Kafka broker unresponsive. - 8
- (Optional) The time to wait before attempting another retry of a failed HTTP request to the authorization server. By default, this time is set to zero, meaning that no pause is applied. This is because many issues that cause failed requests are per-request network glitches or proxy issues that can be resolved quickly. However, if your authorization server is under stress or experiencing high traffic, you may want to set this option to a value of 100 ms or more to reduce the load on the server and increase the likelihood of successful retries.
- 9
- (Optional) Some authorization servers have issues with client sending
Accept: application/jsonheader. By settingincludeAcceptHeader: falsethe header will not be sent. Default istrue.
- Apply the changes to the resource configuration of the component.
Check the update in the logs or by watching the pod state transitions:
oc logs -f ${POD_NAME} -c ${CONTAINER_NAME} oc get pod -woc logs -f ${POD_NAME} -c ${CONTAINER_NAME} oc get pod -wCopy to Clipboard Copied! Toggle word wrap Toggle overflow The rolling updates configure the component for interaction with Kafka brokers using OAuth 2.0 authentication.
17.3. Using OAuth 2.0 token-based authorization
Streams for Apache Kafka supports the use of OAuth 2.0 token-based authorization through Authorization Services, which lets you manage security policies and permissions centrally.
Security policies and permissions defined in Red Hat build of Keycloak grant access to Kafka resources. Users and clients are matched against policies that permit access to perform specific actions on Kafka brokers.
Kafka allows all users full access to brokers by default, but also provides the StandardAuthorizer plugin to configure authorization based on Access Control Lists (ACLs). The ACL rules managed by these plugins are used to grant or deny access to resources based on username, and these rules are stored within the Kafka cluster itself.
However, OAuth 2.0 token-based authorization with Red Hat build of Keycloak offers far greater flexibility on how you wish to implement access control to Kafka brokers. In addition, you can configure your Kafka brokers to use OAuth 2.0 authorization and ACLs.
17.3.1. Example: Enabling OAuth 2.0 authorization
This example procedure shows how to configure Kafka to use OAuth 2.0 authorization using Red Hat build of Keycloak Authorization Services. To enable OAuth 2.0 authorization using Red Hat build of Keycloak, configure the Kafka resource to use keycloak authorization and specify the properties required to access the authorization server and Red Hat build of Keycloak Authorization Services.
Red Hat build of Keycloak server Authorization Services REST endpoints extend token-based authentication with Red Hat build of Keycloak by applying defined security policies on a particular user, and providing a list of permissions granted on different resources for that user. Policies use roles and groups to match permissions to users. OAuth 2.0 authorization enforces permissions locally based on the received list of grants for the user from Red Hat build of Keycloak Authorization Services.
A Red Hat build of Keycloak authorizer (KeycloakAuthorizer) is provided with Streams for Apache Kafka. The authorizer fetches a list of granted permissions from the authorization server as needed, and enforces authorization locally on Kafka, making rapid authorization decisions for each client request.
Before you begin
Consider the access you require or want to limit for certain users. You can use a combination of Red Hat build of Keycloak groups, roles, clients, and users to configure access in Red Hat build of Keycloak.
Typically, groups are used to match users based on organizational departments or geographical locations. And roles are used to match users based on their function.
With Red Hat build of Keycloak, you can store users and groups in LDAP, whereas clients and roles cannot be stored this way. Storage and access to user data may be a factor in how you choose to configure authorization policies.
Super users always have unconstrained access to Kafka regardless of the authorization implemented.
Prerequisites
- Streams for Apache Kafka must be configured to use OAuth 2.0 with Red Hat build of Keycloak for token-based authentication. You use the same Red Hat build of Keycloak server endpoint when you set up authorization.
-
OAuth 2.0 authentication must be configured with the
maxSecondsWithoutReauthenticationoption to enable re-authentication.
Procedure
- Access the Red Hat build of Keycloak Admin Console or use the Red Hat build of Keycloak Admin CLI to enable Authorization Services for the OAuth 2.0 client for Kafka you created when setting up OAuth 2.0 authentication.
- Use Authorization Services to define resources, authorization scopes, policies, and permissions for the client.
- Bind the permissions to users and clients by assigning them roles and groups.
Configure the
kafkaresource to usekeycloakauthorization, and to be able to access the authorization server and Authorization Services.Example OAuth 2.0 authorization configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Type
keycloakenables Red Hat build of Keycloak authorization. - 2
- URI of the Red Hat build of Keycloak token endpoint. For production, always use
https://urls. When you configure token-basedoauthauthentication, you specify ajwksEndpointUrias the URI for local JWT validation. The hostname for thetokenEndpointUriURI must be the same. - 3
- The client ID of the OAuth 2.0 client definition in Red Hat build of Keycloak that has Authorization Services enabled. Typically,
kafkais used as the ID. - 4
- (Optional) Delegate authorization to the Kafka
StandardAuthorizerif access is denied by Red Hat build of Keycloak Authorization Services policies. Default isfalse. - 5
- (Optional) Disable TLS hostname verification. Default is
false. - 6
- (Optional) Designated super users.
- 7
- (Optional) Certificates stored in X.509 format within the specified secrets for TLS connection to the authorization server.
- 8
- (Optional) The time between two consecutive grants refresh runs. That is the maximum time for active sessions to detect any permissions changes for the user on Red Hat build of Keycloak. The default value is 60.
- 9
- (Optional) The number of threads to use to refresh (in parallel) the grants for the active sessions. The default value is 5.
- 10
- (Optional) The time, in seconds, after which an idle grant in the cache can be evicted. The default value is 300.
- 11
- (Optional) The time, in seconds, between consecutive runs of a job that cleans stale grants from the cache. The default value is 300.
- 12
- (Optional) Controls whether the latest grants are fetched for a new session. When enabled, grants are retrieved from Red Hat build of Keycloak and cached for the user. The default value is
false. - 13
- (Optional) The connect timeout in seconds when connecting to the Red Hat build of Keycloak token endpoint. The default value is 60.
- 14
- (Optional) The read timeout in seconds when connecting to the Red Hat build of Keycloak token endpoint. The default value is 60.
- 15
- (Optional) The maximum number of times to retry (without pausing) a failed HTTP request to the authorization server. The default value is
0, meaning that no retries are performed. To use this option effectively, consider reducing the timeout times for theconnectTimeoutSecondsandreadTimeoutSecondsoptions. However, note that retries may prevent the current worker thread from being available to other requests, and if too many requests stall, it could make Kafka unresponsive. - 16
- (Optional) Enable or disable OAuth metrics. The default value is
false. - 17
- (Optional) Some authorization servers have issues with client sending
Accept: application/jsonheader. By settingincludeAcceptHeader: falsethe header will not be sent. Default istrue.
-
Apply the changes to the
Kafkaconfiguration. Check the update in the logs or by watching the pod state transitions:
oc logs -f ${POD_NAME} -c kafka oc get pod -woc logs -f ${POD_NAME} -c kafka oc get pod -wCopy to Clipboard Copied! Toggle word wrap Toggle overflow The rolling update configures the brokers to use OAuth 2.0 authorization.
- Verify the configured permissions by accessing Kafka brokers as clients or users with specific roles, ensuring they have the necessary access and do not have unauthorized access.
When using Red Hat build of Keycloak as the OAuth 2.0 authorization server, Kafka permissions are granted to user accounts or service accounts using authorization permissions. To grant permissions to access Kafka, create an OAuth client specification in Red Hat build of Keycloak that maps the authorization models of Red Hat build of Keycloak Authorization Services and Kafka.
Kafka and Red Hat build of Keycloak use different authorization models.
Kafka authorization model
Kafka’s authorization model uses resource types and operations to describe ACLs for a user. When a Kafka client performs an action on a broker, the broker uses the configured KeycloakAuthorizer to check the client’s permissions, based on the action and resource type.
Each resource type has a set of available permissions for operations. For example, the Topic resource type has Create and Write permissions among others.
Refer to the Kafka authorization model in the Kafka documentation for the full list of resources and permissions.
Red Hat build of Keycloak authorization model
Red Hat build of Keycloak’s authorization services model has four concepts for defining and granting permissions:
- Resources
- Scopes
- Policies
- Permissions
For information on these concepts, see the guide to Authorization Services.
17.4.2. Mapping authorization models
The Kafka authorization model is used as a basis for defining the Red Hat build of Keycloak roles and resources that control access to Kafka.
To grant Kafka permissions to user accounts or service accounts, you first create an OAuth client specification in Red Hat build of Keycloak for the Kafka cluster. You then specify Red Hat build of Keycloak Authorization Services rules on the client. Typically, the client ID of the OAuth client that represents the Kafka cluster is kafka. The example configuration files provided with Streams for Apache Kafka use kafka as the OAuth client id.
If you have multiple Kafka clusters, you can use a single OAuth client (kafka) for all of them. This gives you a single, unified space in which to define and manage authorization rules. However, you can also use different OAuth client ids (for example, my-cluster-kafka or cluster-dev-kafka) and define authorization rules for each cluster within each client configuration.
The kafka client definition must have the Authorization Enabled option enabled in the Red Hat build of Keycloak Admin Console. For information on enabling authorization services, see the guide to Authorization Services.
All permissions exist within the scope of the kafka client. If you have different Kafka clusters configured with different OAuth client IDs, they each need a separate set of permissions even though they’re part of the same Red Hat build of Keycloak realm.
When the Kafka client uses OAUTHBEARER authentication, the Red Hat build of Keycloak authorizer (KeycloakAuthorizer) uses the access token of the current session to retrieve a list of grants from the Red Hat build of Keycloak server. To grant permissions, the authorizer evaluates the grants list (received and cached) from Red Hat build of Keycloak Authorization Services based on the access token owner’s policies and permissions.
Uploading authorization scopes for Kafka permissions
An initial Red Hat build of Keycloak configuration usually involves uploading authorization scopes to create a list of all possible actions that can be performed on each Kafka resource type. This step is performed once only, before defining any permissions. You can add authorization scopes manually instead of uploading them.
Authorization scopes should contain the following Kafka permissions regardless of the resource type:
-
Create -
Write -
Read -
Delete -
Describe -
Alter -
DescribeConfigs -
AlterConfigs -
ClusterAction -
IdempotentWrite
If you’re certain you won’t need a permission (for example, IdempotentWrite), you can omit it from the list of authorization scopes. However, that permission won’t be available to target on Kafka resources.
The All permission is not supported.
Resource patterns for permissions checks
Resource patterns are used for pattern matching against the targeted resources when performing permission checks. The general pattern format is <resource_type>:<pattern_name>.
The resource types mirror the Kafka authorization model. The pattern allows for two matching options:
-
Exact matching (when the pattern does not end with
*) -
Prefix matching (when the pattern ends with
*)
Example patterns for resources
Topic:my-topic Topic:orders-* Group:orders-* Cluster:*
Topic:my-topic
Topic:orders-*
Group:orders-*
Cluster:*
Additionally, the general pattern format can be prefixed by kafka-cluster:<cluster_name> followed by a comma, where <cluster_name> refers to the metadata.name in the Kafka custom resource.
Example patterns for resources with cluster prefix
kafka-cluster:my-cluster,Topic:* kafka-cluster:*,Group:b_*
kafka-cluster:my-cluster,Topic:*
kafka-cluster:*,Group:b_*
When the kafka-cluster prefix is missing, it is assumed to be kafka-cluster:*.
When defining a resource, you can associate it with a list of possible authorization scopes which are relevant to the resource. Set whatever actions make sense for the targeted resource type.
Though you may add any authorization scope to any resource, only the scopes supported by the resource type are considered for access control.
Policies for applying access permission
Policies are used to target permissions to one or more user accounts or service accounts. Targeting can refer to:
- Specific user or service accounts
- Realm roles or client roles
- User groups
A policy is given a unique name and can be reused to target multiple permissions to multiple resources.
Permissions to grant access
Use fine-grained permissions to pull together the policies, resources, and authorization scopes that grant access to users.
The name of each permission should clearly define which permissions it grants to which users. For example, Dev Team B can read from topics starting with x.
17.4.3. Permissions for common Kafka operations
The following examples demonstrate the user permissions required for performing common operations on Kafka.
Create a topic
To create a topic, the Create permission is required for the specific topic, or for Cluster:kafka-cluster.
bin/kafka-topics.sh --create --topic my-topic \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-topics.sh --create --topic my-topic \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesList topics
If a user has the Describe permission on a specified topic, the topic is listed.
bin/kafka-topics.sh --list \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-topics.sh --list \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesDisplay topic details
To display a topic’s details, Describe and DescribeConfigs permissions are required on the topic.
bin/kafka-topics.sh --describe --topic my-topic \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-topics.sh --describe --topic my-topic \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesProduce messages to a topic
To produce messages to a topic, Describe and Write permissions are required on the topic.
If the topic hasn’t been created yet, and topic auto-creation is enabled, the permissions to create a topic are required.
bin/kafka-console-producer.sh --topic my-topic \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --producer.config=/tmp/config.properties
bin/kafka-console-producer.sh --topic my-topic \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --producer.config=/tmp/config.propertiesConsume messages from a topic
To consume messages from a topic, Describe and Read permissions are required on the topic. Consuming from the topic normally relies on storing the consumer offsets in a consumer group, which requires additional Describe and Read permissions on the consumer group.
Two resources are needed for matching. For example:
Topic:my-topic Group:my-group-*
Topic:my-topic
Group:my-group-*bin/kafka-console-consumer.sh --topic my-topic --group my-group-1 --from-beginning \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --consumer.config /tmp/config.properties
bin/kafka-console-consumer.sh --topic my-topic --group my-group-1 --from-beginning \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --consumer.config /tmp/config.propertiesProduce messages to a topic using an idempotent producer
As well as the permissions for producing to a topic, an additional IdempotentWrite permission is required on the Cluster:kafka-cluster resource.
Two resources are needed for matching. For example:
Topic:my-topic Cluster:kafka-cluster
Topic:my-topic
Cluster:kafka-clusterbin/kafka-console-producer.sh --topic my-topic \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --producer.config=/tmp/config.properties --producer-property enable.idempotence=true --request-required-acks -1
bin/kafka-console-producer.sh --topic my-topic \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --producer.config=/tmp/config.properties --producer-property enable.idempotence=true --request-required-acks -1List consumer groups
When listing consumer groups, only the groups on which the user has the Describe permissions are returned. Alternatively, if the user has the Describe permission on the Cluster:kafka-cluster, all the consumer groups are returned.
bin/kafka-consumer-groups.sh --list \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-consumer-groups.sh --list \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesDisplay consumer group details
To display a consumer group’s details, the Describe permission is required on the group and the topics associated with the group.
bin/kafka-consumer-groups.sh --describe --group my-group-1 \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-consumer-groups.sh --describe --group my-group-1 \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesChange topic configuration
To change a topic’s configuration, the Describe and Alter permissions are required on the topic.
bin/kafka-topics.sh --alter --topic my-topic --partitions 2 \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-topics.sh --alter --topic my-topic --partitions 2 \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesDisplay Kafka broker configuration
In order to use kafka-configs.sh to get a broker’s configuration, the DescribeConfigs permission is required on the Cluster:kafka-cluster.
bin/kafka-configs.sh --entity-type brokers --entity-name 0 --describe --all \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-configs.sh --entity-type brokers --entity-name 0 --describe --all \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesChange Kafka broker configuration
To change a Kafka broker’s configuration, DescribeConfigs and AlterConfigs permissions are required on Cluster:kafka-cluster.
bin/kafka-configs --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-configs --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesDelete a topic
To delete a topic, the Describe and Delete permissions are required on the topic.
bin/kafka-topics.sh --delete --topic my-topic \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.properties
bin/kafka-topics.sh --delete --topic my-topic \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config=/tmp/config.propertiesSelect a lead partition
To run leader selection for topic partitions, the Alter permission is required on the Cluster:kafka-cluster.
bin/kafka-leader-election.sh --topic my-topic --partition 0 --election-type PREFERRED / --bootstrap-server my-cluster-kafka-bootstrap:9092 --admin.config /tmp/config.properties
bin/kafka-leader-election.sh --topic my-topic --partition 0 --election-type PREFERRED /
--bootstrap-server my-cluster-kafka-bootstrap:9092 --admin.config /tmp/config.propertiesReassign partitions
To generate a partition reassignment file, Describe permissions are required on the topics involved.
bin/kafka-reassign-partitions.sh --topics-to-move-json-file /tmp/topics-to-move.json --broker-list "0,1" --generate \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties > /tmp/partition-reassignment.json
bin/kafka-reassign-partitions.sh --topics-to-move-json-file /tmp/topics-to-move.json --broker-list "0,1" --generate \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties > /tmp/partition-reassignment.json
To execute the partition reassignment, Describe and Alter permissions are required on Cluster:kafka-cluster. Also, Describe permissions are required on the topics involved.
bin/kafka-reassign-partitions.sh --reassignment-json-file /tmp/partition-reassignment.json --execute \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties
bin/kafka-reassign-partitions.sh --reassignment-json-file /tmp/partition-reassignment.json --execute \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties
To verify partition reassignment, Describe, and AlterConfigs permissions are required on Cluster:kafka-cluster, and on each of the topics involved.
bin/kafka-reassign-partitions.sh --reassignment-json-file /tmp/partition-reassignment.json --verify \ --bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties
bin/kafka-reassign-partitions.sh --reassignment-json-file /tmp/partition-reassignment.json --verify \
--bootstrap-server my-cluster-kafka-bootstrap:9092 --command-config /tmp/config.properties
If you are using OAuth 2.0 with Red Hat build of Keycloak for token-based authentication, you can also use Red Hat build of Keycloak to configure authorization rules to constrain client access to Kafka brokers. This example explains how to use Red Hat build of Keycloak Authorization Services with keycloak authorization. Set up Red Hat build of Keycloak Authorization Services to enforce access restrictions on Kafka clients. Red Hat build of Keycloak Authorization Services use authorization scopes, policies and permissions to define and apply access control to resources.
Red Hat build of Keycloak Authorization Services REST endpoints provide a list of granted permissions on resources for authenticated users. The list of grants (permissions) is fetched from the Red Hat build of Keycloak server as the first action after an authenticated session is established by the Kafka client. The list is refreshed in the background so that changes to the grants are detected. Grants are cached and enforced locally on the Kafka broker for each user session to provide fast authorization decisions.
Streams for Apache Kafka provides example configuration files. These include the following example files for setting up Red Hat build of Keycloak:
kafka-ephemeral-oauth-single-keycloak-authz.yaml-
An example
Kafkacustom resource configured for OAuth 2.0 token-based authorization using Red Hat build of Keycloak. You can use the custom resource to deploy a Kafka cluster that useskeycloakauthorization and token-basedoauthauthentication. kafka-authz-realm.json- An example Red Hat build of Keycloak realm configured with sample groups, users, roles and clients. You can import the realm into a Red Hat build of Keycloak instance to set up fine-grained permissions to access Kafka.
If you want to try the example with Red Hat build of Keycloak, use these files to perform the tasks outlined in this section in the order shown.
Authentication
When you configure token-based oauth authentication, you specify a jwksEndpointUri as the URI for local JWT validation. When you configure keycloak authorization, you specify a tokenEndpointUri as the URI of the Red Hat build of Keycloak token endpoint. The hostname for both URIs must be the same.
Targeted permissions with group or role policies
In Red Hat build of Keycloak, confidential clients with service accounts enabled can authenticate to the server in their own name using a client ID and a secret. This is convenient for microservices that typically act in their own name, and not as agents of a particular user (like a web site). Service accounts can have roles assigned like regular users. They cannot, however, have groups assigned. As a consequence, if you want to target permissions to microservices using service accounts, you cannot use group policies, and should instead use role policies. Conversely, if you want to limit certain permissions only to regular user accounts where authentication with a username and password is required, you can achieve that as a side effect of using the group policies rather than the role policies. This is what is used in this example for permissions that start with ClusterManager. Performing cluster management is usually done interactively using CLI tools. It makes sense to require the user to log in before using the resulting access token to authenticate to the Kafka broker. In this case, the access token represents the specific user, rather than the client application.
Set up Red Hat build of Keycloak, then connect to its Admin Console and add the preconfigured realm. Use the example kafka-authz-realm.json file to import the realm. You can check the authorization rules defined for the realm in the Admin Console. The rules grant access to the resources on the Kafka cluster configured to use the example Red Hat build of Keycloak realm.
Prerequisites
- A running OpenShift cluster.
-
The Streams for Apache Kafka
examples/security/keycloak-authorization/kafka-authz-realm.jsonfile that contains the preconfigured realm.
Procedure
- Install the Red Hat build of Keycloak server using the Red Hat build of Keycloak Operator as described in Getting Started in the Red Hat build of Keycloak documentation.
- Wait until the Red Hat build of Keycloak instance is running.
Get the external hostname to be able to access the Admin Console.
NS=sso oc get ingress keycloak -n $NS
NS=sso oc get ingress keycloak -n $NSCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, we assume the Red Hat build of Keycloak server is running in the
ssonamespace.Get the password for the
adminuser.oc get -n $NS pod keycloak-0 -o yaml | less
oc get -n $NS pod keycloak-0 -o yaml | lessCopy to Clipboard Copied! Toggle word wrap Toggle overflow The password is stored as a secret, so get the configuration YAML file for the Red Hat build of Keycloak instance to identify the name of the secret (
secretKeyRef.name).Use the name of the secret to obtain the clear text password.
SECRET_NAME=credential-keycloak oc get -n $NS secret $SECRET_NAME -o yaml | grep PASSWORD | awk '{print $2}' | base64 -DSECRET_NAME=credential-keycloak oc get -n $NS secret $SECRET_NAME -o yaml | grep PASSWORD | awk '{print $2}' | base64 -DCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, we assume the name of the secret is
credential-keycloak.Log in to the Admin Console with the username
adminand the password you obtained.Use
https://HOSTNAMEto access the KubernetesIngress.You can now upload the example realm to Red Hat build of Keycloak using the Admin Console.
- Click Add Realm to import the example realm.
Add the
examples/security/keycloak-authorization/kafka-authz-realm.jsonfile, and then click Create.You now have
kafka-authzas your current realm in the Admin Console.The default view displays the Master realm.
In the Red Hat build of Keycloak Admin Console, go to Clients > kafka > Authorization > Settings and check that Decision Strategy is set to Affirmative.
An affirmative policy means that at least one policy must be satisfied for a client to access the Kafka cluster.
In the Red Hat build of Keycloak Admin Console, go to Groups, Users, Roles and Clients to view the realm configuration.
- Groups
-
Groupsorganize users and set permissions. Groups can be linked to an LDAP identity provider and used to compartmentalize users, such as by region or department. Users can be added to groups through a custom LDAP interface to manage permissions for Kafka resources. For more information on using KeyCloak’s LDAP provider, see the guide to Keycloak Server Administration. - Users
-
Usersdefine individual users. In this example,alice(in theClusterManagergroup) andbob(inClusterManager-my-cluster) are created. Users can also be stored in an LDAP identity provider. - Roles
-
Rolesassign specific permissions to users or clients. Roles function like tags to give users certain rights. While roles cannot be stored in LDAP, you can add Red Hat build of Keycloak roles to groups to combine both roles and permissions. - Clients
Clientsdefine configurations for Kafka interactions.-
The
kafkaclient handles OAuth 2.0 token validation for brokers and contains authorization policies (which require authorization to be enabled). -
The
kafka-cliclient is used by command line tools to obtain access or refresh tokens. -
team-a-clientandteam-b-clientrepresent services with partial access to specific Kafka topics.
-
The
In the Red Hat build of Keycloak Admin Console, go to Authorization > Permissions to see the granted permissions that use the resources and policies defined for the realm.
For example, the
kafkaclient has the following permissions:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Dev Team A
-
The Dev Team A realm role can write to topics that start with
x_on any cluster. This combines a resource calledTopic:x_*,DescribeandWritescopes, and theDev Team Apolicy. TheDev Team Apolicy matches all users that have a realm role calledDev Team A. - Dev Team B
-
The Dev Team B realm role can read from topics that start with
x_on any cluster. This combinesTopic:x_*,Group:x_*resources,DescribeandReadscopes, and theDev Team Bpolicy. TheDev Team Bpolicy matches all users that have a realm role calledDev Team B. Matching users and clients have the ability to read from topics, and update the consumed offsets for topics and consumer groups that have names starting withx_.
Deploy a Kafka cluster configured to connect to the Red Hat build of Keycloak server. Use the example kafka-ephemeral-oauth-single-keycloak-authz.yaml file to deploy the Kafka cluster as a Kafka custom resource. The example deploys a single-node Kafka cluster with keycloak authorization and oauth authentication.
Prerequisites
- The Red Hat build of Keycloak authorization server is deployed to your OpenShift cluster and loaded with the example realm.
- The Cluster Operator is deployed to your OpenShift cluster.
-
The Streams for Apache Kafka
examples/security/keycloak-authorization/kafka-ephemeral-oauth-single-keycloak-authz.yamlcustom resource.
Procedure
Use the hostname of the Red Hat build of Keycloak instance you deployed to prepare a truststore certificate for Kafka brokers to communicate with the Red Hat build of Keycloak server.
SSO_HOST=SSO-HOSTNAME SSO_HOST_PORT=$SSO_HOST:443 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $SSO_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/sso.pemSSO_HOST=SSO-HOSTNAME SSO_HOST_PORT=$SSO_HOST:443 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $SSO_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/sso.pemCopy to Clipboard Copied! Toggle word wrap Toggle overflow The certificate is required as Kubernetes
Ingressis used to make a secure (HTTPS) connection.Usually there is not one single certificate, but a certificate chain. You only have to provide the top-most issuer CA, which is listed last in the
/tmp/sso.pemfile. You can extract it manually or using the following commands:Example command to extract the top CA certificate in a certificate chain
split -p "-----BEGIN CERTIFICATE-----" sso.pem sso- for f in $(ls sso-*); do mv $f $f.pem; done cp $(ls sso-* | sort -r | head -n 1) sso-ca.crt
split -p "-----BEGIN CERTIFICATE-----" sso.pem sso- for f in $(ls sso-*); do mv $f $f.pem; done cp $(ls sso-* | sort -r | head -n 1) sso-ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteA trusted CA certificate is normally obtained from a trusted source, and not by using the
opensslcommand.Deploy the certificate to OpenShift as a secret.
oc create secret generic oauth-server-cert --from-file=/tmp/sso-ca.crt -n $NS
oc create secret generic oauth-server-cert --from-file=/tmp/sso-ca.crt -n $NSCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set the hostname as an environment variable
SSO_HOST=SSO-HOSTNAME
SSO_HOST=SSO-HOSTNAMECopy to Clipboard Copied! Toggle word wrap Toggle overflow Create and deploy the example Kafka cluster.
cat examples/security/keycloak-authorization/kafka-ephemeral-oauth-single-keycloak-authz.yaml | sed -E 's#\${SSO_HOST}'"#$SSO_HOST#" | oc create -n $NS -f -cat examples/security/keycloak-authorization/kafka-ephemeral-oauth-single-keycloak-authz.yaml | sed -E 's#\${SSO_HOST}'"#$SSO_HOST#" | oc create -n $NS -f -Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Create a new pod for an interactive CLI session. Set up a truststore with a Red Hat build of Keycloak certificate for TLS connectivity. The truststore is to connect to Red Hat build of Keycloak and the Kafka broker.
Prerequisites
The Red Hat build of Keycloak authorization server is deployed to your OpenShift cluster and loaded with the example realm.
In the Red Hat build of Keycloak Admin Console, check the roles assigned to the clients are displayed in Clients > Service Account Roles.
- The Kafka cluster configured to connect with Red Hat build of Keycloak is deployed to your OpenShift cluster.
Procedure
Run a new interactive pod container using the Kafka image to connect to a running Kafka broker.
NS=sso oc run -ti --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 kafka-cli -n $NS -- /bin/sh
NS=sso oc run -ti --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 kafka-cli -n $NS -- /bin/shCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf
octimes out waiting on the image download, subsequent attempts may result in an AlreadyExists error.Attach to the pod container.
oc attach -ti kafka-cli -n $NS
oc attach -ti kafka-cli -n $NSCopy to Clipboard Copied! Toggle word wrap Toggle overflow Use the hostname of the Red Hat build of Keycloak instance to prepare a certificate for client connection using TLS.
SSO_HOST=SSO-HOSTNAME SSO_HOST_PORT=$SSO_HOST:443 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $SSO_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/sso.pemSSO_HOST=SSO-HOSTNAME SSO_HOST_PORT=$SSO_HOST:443 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $SSO_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/sso.pemCopy to Clipboard Copied! Toggle word wrap Toggle overflow Usually there is not one single certificate, but a certificate chain. You only have to provide the top-most issuer CA, which is listed last in the
/tmp/sso.pemfile. You can extract it manually or using the following command:Example command to extract the top CA certificate in a certificate chain
split -p "-----BEGIN CERTIFICATE-----" sso.pem sso- for f in $(ls sso-*); do mv $f $f.pem; done cp $(ls sso-* | sort -r | head -n 1) sso-ca.crt
split -p "-----BEGIN CERTIFICATE-----" sso.pem sso- for f in $(ls sso-*); do mv $f $f.pem; done cp $(ls sso-* | sort -r | head -n 1) sso-ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteA trusted CA certificate is normally obtained from a trusted source, and not by using the
opensslcommand.Create a truststore for TLS connection to the Kafka brokers.
keytool -keystore /tmp/truststore.p12 -storetype pkcs12 -alias sso -storepass $STOREPASS -import -file /tmp/sso-ca.crt -noprompt
keytool -keystore /tmp/truststore.p12 -storetype pkcs12 -alias sso -storepass $STOREPASS -import -file /tmp/sso-ca.crt -nopromptCopy to Clipboard Copied! Toggle word wrap Toggle overflow Use the Kafka bootstrap address as the hostname of the Kafka broker and the
tlslistener port (9093) to prepare a certificate for the Kafka broker.KAFKA_HOST_PORT=my-cluster-kafka-bootstrap:9093 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $KAFKA_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/my-cluster-kafka.pemKAFKA_HOST_PORT=my-cluster-kafka-bootstrap:9093 STOREPASS=storepass echo "Q" | openssl s_client -showcerts -connect $KAFKA_HOST_PORT 2>/dev/null | awk ' /BEGIN CERTIFICATE/,/END CERTIFICATE/ { print $0 } ' > /tmp/my-cluster-kafka.pemCopy to Clipboard Copied! Toggle word wrap Toggle overflow The obtained
.pemfile is usually not one single certificate, but a certificate chain. You only have to provide the top-most issuer CA, which is listed last in the/tmp/my-cluster-kafka.pemfile. You can extract it manually or using the following command:Example command to extract the top CA certificate in a certificate chain
split -p "-----BEGIN CERTIFICATE-----" /tmp/my-cluster-kafka.pem kafka- for f in $(ls kafka-*); do mv $f $f.pem; done cp $(ls kafka-* | sort -r | head -n 1) my-cluster-kafka-ca.crt
split -p "-----BEGIN CERTIFICATE-----" /tmp/my-cluster-kafka.pem kafka- for f in $(ls kafka-*); do mv $f $f.pem; done cp $(ls kafka-* | sort -r | head -n 1) my-cluster-kafka-ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteA trusted CA certificate is normally obtained from a trusted source, and not by using the
opensslcommand. For this example we assume the client is running in a pod in the same namespace where the Kafka cluster was deployed. If the client is accessing the Kafka cluster from outside the OpenShift cluster, you would have to first determine the bootstrap address. In that case you can also get the cluster certificate directly from the OpenShift secret, and there is no need foropenssl. For more information, see Chapter 15, Setting up client access to a Kafka cluster.Add the certificate for the Kafka broker to the truststore.
keytool -keystore /tmp/truststore.p12 -storetype pkcs12 -alias my-cluster-kafka -storepass $STOREPASS -import -file /tmp/my-cluster-kafka-ca.crt -noprompt
keytool -keystore /tmp/truststore.p12 -storetype pkcs12 -alias my-cluster-kafka -storepass $STOREPASS -import -file /tmp/my-cluster-kafka-ca.crt -nopromptCopy to Clipboard Copied! Toggle word wrap Toggle overflow Keep the session open to check authorized access.
Check the authorization rules applied through the Red Hat build of Keycloak realm using an interactive CLI session. Apply the checks using Kafka’s example producer and consumer clients to create topics with user and service accounts that have different levels of access.
Use the team-a-client and team-b-client clients to check the authorization rules. Use the alice admin user to perform additional administrative tasks on Kafka.
The Kafka image used in this example contains Kafka producer and consumer binaries.
Prerequisites
- A Kafka cluster is running in the OpenShift cluster to be able to send and receive messages.
- The interactive CLI Kafka client session is started.
Setting up client and admin user configuration
Prepare a Kafka configuration file with authentication properties for the
team-a-clientclient.Copy to Clipboard Copied! Toggle word wrap Toggle overflow The SASL
OAUTHBEARERmechanism is used. This mechanism requires a client ID and client secret, which means the client first connects to the Red Hat build of Keycloak server to obtain an access token. The client then connects to the Kafka broker and uses the access token to authenticate.Prepare a Kafka configuration file with authentication properties for the
team-b-clientclient.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Authenticate admin user
aliceby usingcurland performing a password grant authentication to obtain a refresh token.Copy to Clipboard Copied! Toggle word wrap Toggle overflow The refresh token is an offline token that is long-lived and does not expire.
Prepare a Kafka configuration file with authentication properties for the admin user
alice.Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
kafka-clipublic client is used for theoauth.client.idin thesasl.jaas.config. Since it’s a public client it does not require a secret. The client authenticates with the refresh token that was authenticated in the previous step. The refresh token requests an access token behind the scenes, which is then sent to the Kafka broker for authentication.
Producing messages with authorized access
Use the team-a-client configuration to check that you can produce messages to topics that start with a_ or x_.
Write to topic
my-topic.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic my-topic \ --producer.config=/tmp/team-a-client.properties First message
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic my-topic \ --producer.config=/tmp/team-a-client.properties First messageCopy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [my-topic]error.team-a-clienthas aDev Team Arole that gives it permission to perform any supported actions on topics that start witha_, but can only write to topics that start withx_. The topic namedmy-topicmatches neither of those rules.Write to topic
a_messages.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --producer.config /tmp/team-a-client.properties First message Second message
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --producer.config /tmp/team-a-client.properties First message Second messageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Messages are produced to Kafka successfully.
- Press CTRL+C to exit the CLI application.
Check the Kafka container log for a debug log of
Authorization GRANTEDfor the request.oc logs my-cluster-kafka-0 -f -n $NS
oc logs my-cluster-kafka-0 -f -n $NSCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Consuming messages with authorized access
Use the team-a-client configuration to consume messages from topic a_messages.
Fetch messages from topic
a_messages.bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --from-beginning --consumer.config /tmp/team-a-client.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow The request returns an error because the
Dev Team Arole forteam-a-clientonly has access to consumer groups that have names starting witha_.Update the
team-a-clientproperties to specify the custom consumer group it is permitted to use.bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group a_consumer_group_1
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group a_consumer_group_1Copy to Clipboard Copied! Toggle word wrap Toggle overflow The consumer receives all the messages from the
a_messagestopic.
Administering Kafka with authorized access
The team-a-client is an account without any cluster-level access, but it can be used with some administrative operations.
List topics.
bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --list
bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --listCopy to Clipboard Copied! Toggle word wrap Toggle overflow The
a_messagestopic is returned.List consumer groups.
bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --list
bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --listCopy to Clipboard Copied! Toggle word wrap Toggle overflow The
a_consumer_group_1consumer group is returned.Fetch details on the cluster configuration.
bin/kafka-configs.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties \ --entity-type brokers --describe --entity-default
bin/kafka-configs.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties \ --entity-type brokers --describe --entity-defaultCopy to Clipboard Copied! Toggle word wrap Toggle overflow The request returns an error because the operation requires cluster level permissions that
team-a-clientdoes not have.
Using clients with different permissions
Use the team-b-client configuration to produce messages to topics that start with b_.
Write to topic
a_messages.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --producer.config /tmp/team-b-client.properties Message 1
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic a_messages \ --producer.config /tmp/team-b-client.properties Message 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [a_messages]error.Write to topic
b_messages.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic b_messages \ --producer.config /tmp/team-b-client.properties Message 1 Message 2 Message 3
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic b_messages \ --producer.config /tmp/team-b-client.properties Message 1 Message 2 Message 3Copy to Clipboard Copied! Toggle word wrap Toggle overflow Messages are produced to Kafka successfully.
Write to topic
x_messages.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-b-client.properties Message 1
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-b-client.properties Message 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow A
Not authorized to access topics: [x_messages]error is returned, Theteam-b-clientcan only read from topicx_messages.Write to topic
x_messagesusingteam-a-client.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-a-client.properties Message 1
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-a-client.properties Message 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [x_messages]error. Theteam-a-clientcan write to thex_messagestopic, but it does not have a permission to create a topic if it does not yet exist. Beforeteam-a-clientcan write to thex_messagestopic, an admin power user must create it with the correct configuration, such as the number of partitions and replicas.
Managing Kafka with an authorized admin user
Use admin user alice to manage Kafka. alice has full access to manage everything on any Kafka cluster.
Create the
x_messagestopic asalice.bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties \ --topic x_messages --create --replication-factor 1 --partitions 1
bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties \ --topic x_messages --create --replication-factor 1 --partitions 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow The topic is created successfully.
List all topics as
alice.bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties --list bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --list bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-b-client.properties --list
bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties --list bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-a-client.properties --list bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/team-b-client.properties --listCopy to Clipboard Copied! Toggle word wrap Toggle overflow Admin user
alicecan list all the topics, whereasteam-a-clientandteam-b-clientcan only list the topics they have access to.The
Dev Team AandDev Team Broles both haveDescribepermission on topics that start withx_, but they cannot see the other team’s topics because they do not haveDescribepermissions on them.Use the
team-a-clientto produce messages to thex_messagestopic:bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-a-client.properties Message 1 Message 2 Message 3
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-a-client.properties Message 1 Message 2 Message 3Copy to Clipboard Copied! Toggle word wrap Toggle overflow As
alicecreated thex_messagestopic, messages are produced to Kafka successfully.Use the
team-b-clientto produce messages to thex_messagestopic.bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-b-client.properties Message 4 Message 5
bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --producer.config /tmp/team-b-client.properties Message 4 Message 5Copy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [x_messages]error.Use the
team-b-clientto consume messages from thex_messagestopic:bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-b-client.properties --group x_consumer_group_b
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-b-client.properties --group x_consumer_group_bCopy to Clipboard Copied! Toggle word wrap Toggle overflow The consumer receives all the messages from the
x_messagestopic.Use the
team-a-clientto consume messages from thex_messagestopic.bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group x_consumer_group_a
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group x_consumer_group_aCopy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [x_messages]error.Use the
team-a-clientto consume messages from a consumer group that begins witha_.bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group a_consumer_group_a
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/team-a-client.properties --group a_consumer_group_aCopy to Clipboard Copied! Toggle word wrap Toggle overflow This request returns a
Not authorized to access topics: [x_messages]error.Dev Team Ahas noReadaccess on topics that start with ax_.Use
aliceto produce messages to thex_messagestopic.bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/alice.properties
bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --topic x_messages \ --from-beginning --consumer.config /tmp/alice.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Messages are produced to Kafka successfully.
alicecan read from or write to any topic.Use
aliceto read the cluster configuration.bin/kafka-configs.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties \ --entity-type brokers --describe --entity-default
bin/kafka-configs.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/alice.properties \ --entity-type brokers --describe --entity-defaultCopy to Clipboard Copied! Toggle word wrap Toggle overflow The cluster configuration for this example is empty.
Chapter 18. Managing TLS certificates
Streams for Apache Kafka supports TLS for encrypted communication between Kafka and Streams for Apache Kafka components.
Streams for Apache Kafka establishes encrypted TLS connections for communication between the following components when using Kafka in KRaft mode:
- Kafka brokers
- Kafka controllers
- Kafka brokers and controllers
- Streams for Apache Kafka operators and Kafka
- Cruise Control and Kafka brokers
- Kafka Exporter and Kafka brokers
Connections between clients and Kafka brokers use listeners that you must configure to use TLS-encrypted communication. You configure these listeners in the Kafka custom resource and each listener name and port number must be unique within the cluster. Communication between Kafka brokers and Kafka clients is encrypted according to how the tls property is configured for the listener. For more information, see Chapter 15, Setting up client access to a Kafka cluster.
The following diagram shows the connections for secure communication.
Figure 18.1. Kafka communication secured by TLS encryption
The ports shown in the diagram are used as follows:
- Control plane listener (9090)
- The internal control plane listener on port 9090 facilitates interbroker communication between Kafka controllers and broker-to-controller communication. Additionally, the Cluster Operator communicates with the controllers through the listener. This listener is not accessible to Kafka clients.
- Replication listener (9091)
- Data replication between brokers, as well as internal connections to the brokers from Streams for Apache Kafka operators, Cruise Control, and the Kafka Exporter, use the replication listener on port 9091. This listener is not accessible to Kafka clients.
- Listeners for client connections (9092 or higher)
- For TLS-encrypted communication (through configuration of the listener), internal and external clients connect to Kafka brokers. External clients (producers and consumers) connect to the Kafka brokers through the advertised listener port.
When configuring listeners for client access to brokers, you can use port 9092 or higher (9093, 9094, and so on), but with a few exceptions. The listeners cannot be configured to use the ports reserved for interbroker communication (9090 and 9091), Prometheus metrics (9404), and JMX (Java Management Extensions) monitoring (9999).
Node status monitoring using the KafkaAgent (8443)
Streams for Apache Kafka includes a component called KafkaAgent that runs inside each Kafka node. The agent is responsible for collecting and providing node-specific information, such as current state and readiness, to the Cluster Operator. It listens on port 8443 for secure HTTPS connections and exposes this information through a REST API, which the Cluster Operator uses to retrieve data from the nodes.
18.1. Internal cluster CA and clients CA
To support encryption, each Streams for Apache Kafka component needs its own private keys and public key certificates. All component certificates are signed by an internal CA (certificate authority) called the cluster CA.
CA (Certificate Authority) certificates are generated by the Cluster Operator to verify the identities of components and clients.
Similarly, each Kafka client application connecting to Streams for Apache Kafka using mTLS needs to use private keys and certificates. A second internal CA, named the clients CA, is used to sign certificates for the Kafka clients.
Both the cluster CA and clients CA have a self-signed public key certificate.
Kafka brokers are configured to trust certificates signed by either the cluster CA or clients CA. Components that clients do not need to connect to only trust certificates signed by the cluster CA. Unless TLS encryption for external listeners is disabled, client applications must trust certificates signed by the cluster CA. This is also true for client applications that perform mTLS authentication.
By default, Streams for Apache Kafka automatically generates and renews CA certificates issued by the cluster CA or clients CA. You can configure the management of these CA certificates using Kafka.spec.clusterCa and Kafka.spec.clientsCa properties.
If you don’t want to use the CAs generated by the Cluster Operator, you can install your own cluster and clients CA certificates. Any certificates you provide are not renewed by the Cluster Operator.
18.2. Secrets generated by the operators
The Cluster Operator automatically sets up and renews TLS certificates to enable encryption and authentication within a cluster. It also sets up other TLS certificates if you want to enable encryption or mTLS authentication between Kafka brokers and clients.
Secrets are created when custom resources are deployed, such as Kafka and KafkaUser. Streams for Apache Kafka uses these secrets to store private and public key certificates for Kafka clusters, clients, and users. The secrets are used for establishing TLS encrypted connections between Kafka brokers, and between brokers and clients. They are also used for mTLS authentication.
Cluster and clients secrets are always pairs: one contains the public key and one contains the private key.
- Cluster secret
- A cluster secret contains the cluster CA to sign Kafka broker certificates. Connecting clients use the certificate to establish a TLS encrypted connection with a Kafka cluster. The certificate verifies broker identity.
- Client secret
- A client secret contains the clients CA for a user to sign its own client certificate. This allows mutual authentication against the Kafka cluster. The broker validates a client’s identity through the certificate.
- User secret
- A user secret contains a private key and certificate. The secret is created and signed by the clients CA when a new user is created. The key and certificate are used to authenticate and authorize the user when accessing the cluster.
You can provide Kafka listener certificates for TLS listeners or external listeners that have TLS encryption enabled. Use Kafka listener certificates to incorporate the security infrastructure you already have in place.
The secrets created by Streams for Apache Kafka provide private keys and certificates in PEM (Privacy Enhanced Mail) and PKCS #12 (Public-Key Cryptography Standards) formats. PEM and PKCS #12 are OpenSSL-generated key formats for TLS communications using the SSL protocol.
You can configure mutual TLS (mTLS) authentication that uses the credentials contained in the secrets generated for a Kafka cluster and user.
To set up mTLS, you must first do the following:
When you deploy a Kafka cluster, a <cluster_name>-cluster-ca-cert secret is created with public key to verify the cluster. You use the public key to configure a truststore for the client.
When you create a KafkaUser, a <kafka_user_name> secret is created with the keys and certificates to verify the user (client). Use these credentials to configure a keystore for the client.
With the Kafka cluster and client set up to use mTLS, you extract credentials from the secrets and add them to your client configuration.
- PEM keys and certificates
For PEM, you add the following to your client configuration:
- Truststore
-
ca.crtfrom the<cluster_name>-cluster-ca-certsecret, which is the CA certificate for the cluster.
-
- Keystore
-
user.crtfrom the<kafka_user_name>secret, which is the public certificate of the user. -
user.keyfrom the<kafka_user_name>secret, which is the private key of the user.
-
- PKCS #12 keys and certificates
For PKCS #12, you add the following to your client configuration:
- Truststore
-
ca.p12from the<cluster_name>-cluster-ca-certsecret, which is the CA certificate for the cluster. -
ca.passwordfrom the<cluster_name>-cluster-ca-certsecret, which is the password to access the public cluster CA certificate.
-
- Keystore
-
user.p12from the<kafka_user_name>secret, which is the public key certificate of the user. -
user.passwordfrom the<kafka_user_name>secret, which is the password to access the public key certificate of the Kafka user.
-
PKCS #12 is supported by Java, so you can add the values of the certificates directly to your Java client configuration. You can also reference the certificates from a secure storage location. With PEM files, you must add the certificates directly to the client configuration in single-line format. Choose a format that’s suitable for establishing TLS connections between your Kafka cluster and client. Use PKCS #12 if you are unfamiliar with PEM.
All keys are 2048 bits in size and, by default, are valid for 365 days from the initial generation. You can change the validity period.
18.2.2. Secrets generated by the Cluster Operator
The Cluster Operator generates the following certificates, which are saved as secrets in the OpenShift cluster. Streams for Apache Kafka uses these secrets by default.
The cluster CA and clients CA have separate secrets for the private key and public key.
<cluster_name>-cluster-ca- Contains the private key of the cluster CA. Streams for Apache Kafka and Kafka components use the private key to sign server certificates.
<cluster_name>-cluster-ca-cert- Contains the public key of the cluster CA. Kafka clients use the public key to verify the identity of the Kafka brokers they are connecting to with TLS server authentication.
<cluster_name>-clients-ca- Contains the private key of the clients CA. Kafka clients use the private key to sign new user certificates for mTLS authentication when connecting to Kafka brokers.
<cluster_name>-clients-ca-cert- Contains the public key of the clients CA. Kafka brokers use the public key to verify the identity of clients accessing the Kafka brokers when mTLS authentication is used.
Secrets for communication between Streams for Apache Kafka components contain a private key and a public key certificate signed by the cluster CA.
<cluster_name>-<pool_name>-<pod_id>- Contains the private and public keys for a Kafka node. Each pod has its own secret.
<cluster_name>-cluster-operator-certs- Contains the private and public keys for encrypting communication between the Cluster Operator and Kafka.
<cluster_name>-entity-topic-operator-certs- Contains the private and public keys for encrypting communication between the Topic Operator and Kafka.
<cluster_name>-entity-user-operator-certs- Contains the private and public keys for encrypting communication between the User Operator and Kafka.
<cluster_name>-cruise-control-certs- Contains the private and public keys for encrypting communication between Cruise Control and Kafka.
<cluster_name>-kafka-exporter-certs- Contains the private and public keys for encrypting communication between Kafka Exporter and Kafka.
You can provide your own server certificates and private keys to connect to Kafka brokers using Kafka listener certificates rather than certificates signed by the cluster CA.
18.2.3. Cluster CA secrets
Cluster CA secrets are managed by the Cluster Operator in a Kafka cluster.
Only the <cluster_name>-cluster-ca-cert secret is required by clients. All other cluster secrets are accessed by Streams for Apache Kafka components. You can enforce this using OpenShift role-based access controls, if necessary.
The CA certificates in <cluster_name>-cluster-ca-cert must be trusted by Kafka client applications so that they validate the Kafka broker certificates when connecting to Kafka brokers over TLS.
| Field | Description |
|---|---|
|
| The current private key for the cluster CA. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
| The current certificate for the cluster CA. |
| Field | Description |
|---|---|
|
|
Certificate for a Kafka pod <num>. Signed by a current or former cluster CA private key in |
|
|
Private key for a Kafka pod |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
|
Certificate for mTLS communication between the Cluster Operator and Kafka. Signed by a current or former cluster CA private key in |
|
| Private key for mTLS communication between the Cluster Operator and Kafka. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
|
Certificate for mTLS communication between the Topic Operator and Kafka. Signed by a current or former cluster CA private key in |
|
| Private key for mTLS communication between the Topic Operator and Kafka. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
|
Certificate for mTLS communication between the User Operator and Kafka. Signed by a current or former cluster CA private key in |
|
| Private key for mTLS communication between the User Operator and Kafka. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
|
Certificate for mTLS communication between Cruise Control and Kafka. Signed by a current or former cluster CA private key in |
|
| Private key for mTLS communication between the Cruise Control and Kafka. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
|
Certificate for mTLS communication between Kafka Exporter and Kafka. Signed by a current or former cluster CA private key in |
|
| Private key for mTLS communication between the Kafka Exporter and Kafka. |
18.2.4. Clients CA secrets
Clients CA secrets are managed by the Cluster Operator in a Kafka cluster.
The certificates in <cluster_name>-clients-ca-cert are those which the Kafka brokers trust.
The <cluster_name>-clients-ca secret is used to sign the certificates of client applications. This secret must be accessible to the Streams for Apache Kafka components and for administrative access if you are intending to issue application certificates without using the User Operator. You can enforce this using OpenShift role-based access controls, if necessary.
| Field | Description |
|---|---|
|
| The current private key for the clients CA. |
| Field | Description |
|---|---|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. |
|
| The current certificate for the clients CA. |
18.2.5. User secrets generated by the User Operator
User secrets are managed by the User Operator.
When a user is created using the User Operator, a secret is generated using the name of the user.
| Secret name | Field within secret | Description |
|---|---|---|
|
|
| PKCS #12 store for storing certificates and keys. |
|
| Password for protecting the PKCS #12 store. | |
|
| Certificate for the user, signed by the clients CA | |
|
| Private key for the user |
By configuring the clusterCaCert template property in the Kafka custom resource, you can add custom labels and annotations to the Cluster CA secrets created by the Cluster Operator. Labels and annotations are useful for identifying objects and adding contextual information. You configure template properties in Streams for Apache Kafka custom resources.
Example template customization to add labels and annotations to secrets
18.2.7. Disabling ownerReference in the CA secrets
By default, the cluster and clients CA secrets are created with an ownerReference property that is set to the Kafka custom resource. This means that, when the Kafka custom resource is deleted, the CA secrets are also deleted (garbage collected) by OpenShift.
If you want to reuse the CA for a new cluster, you can disable the ownerReference by setting the generateSecretOwnerReference property for the cluster and clients CA secrets to false in the Kafka configuration. When the ownerReference is disabled, CA secrets are not deleted by OpenShift when the corresponding Kafka custom resource is deleted.
Example Kafka configuration with disabled ownerReference for cluster and clients CAs
18.3. Certificate renewal and validity periods
Cluster CA and clients CA certificates are valid for a limited time, known as the validity period. This is defined as the number of days from the date the certificate was generated.
For CA certificates automatically created by the Cluster Operator, configure the validity period for certificates in the kafka resource:
-
Kafka.spec.clusterCa.validityDaysfor Cluster CA certificates -
Kafka.spec.clientsCa.validityDaysfor Clients CA certificates
The default validity period for both certificates is 365 days. For manually-installed custom CA certificates, set validity through your certificate management system.
When a CA certificate expires, components and clients that still trust the old certificate do not accept connections from peers whose certificates were signed by the CA private key. The components and clients must trust the new CA certificate instead.
To prevent loss of service, the Cluster Operator initiates certificate renewal before the old CA certificates expire.
Configure the renewal period of the certificates created by the Cluster Operator in the kafka resource using the following properties:
-
Kafka.spec.clusterCa.renewalDaysfor Cluster CA certificates -
Kafka.spec.clientsCa.renewalDaysfor Clients CA certificates
The default renewal period for both certificates is 30 days from the expiry date of the current certificate. Changing the validityDays does not trigger immediate certificate renewal. The updated value is applied the next time the certificate is renewed, either automatically or through manual renewal.
Validity period against renewal period
Not Before Not After
| |
|<--------------- validityDays --------------->|
<--- renewalDays --->|
Not Before Not After
| |
|<--------------- validityDays --------------->|
<--- renewalDays --->|
Changes to renewalDays may trigger renewal earlier if the new value places the certificate within the renewal window. To schedule the renewal period at a convenient time, use maintenance time windows.
maintenanceTimeWindows apply only to certificates generated automatically by the Cluster Operator. They do not apply to custom or externally managed certificates, so restarts triggered by updates to those certificates may occur outside the defined windows. With a custom Certificate Authority (CA), the Cluster Operator still manages the validity and renewal of the server certificates it generates. In this case, validityDays and renewalDays apply to those server certificates, not to the CA itself.
To change validity and renewal periods after creating the Kafka cluster:
-
Modify the
Kafkacustom resource. - Manually renew the CA certificates.
If you do not manually renew the certificates, the new settings take effect the next time the certificate is renewed automatically.
Example Kafka configuration for certificate validity and renewal periods
Automatic certificate renewal depends on the generateCertificateAuthority setting:
-
If
true, the Cluster Operator handles renewal. -
If
false, certificates must be managed externally.
Use this option if you are installing your own certificates.
18.3.1. Cluster CA renewal
To renew the Cluster CA certificate, the Cluster Operator does the following:
Generates a new Cluster CA certificate, retaining the existing private key.
The renewed certificate replaces
ca.crtin the Cluster CA secret.Regenerates internal client certificates for the following components:
- Kafka nodes
- Entity Operator (Topic Operator and User Operator)
- Kafka Exporter
Cruise Control
These new certificates are not strictly required, because the signing key hasn’t changed. However, the Cluster Operator regenerates them to align their validity period with the new CA certificate.
- Restarts the components to trust the new Cluster CA certificate and use the renewed internal certificates.
18.3.2. Clients CA renewal
To renew the Clients CA certificate, the Cluster Operator and User Operator each perform part of the process:
The Cluster Operator generates a new Clients CA certificate, retaining the existing private key.
The renewed certificate replaces
ca.crtin the Clients CA secret.- The User Operator detects the updated Clients CA certificate and regenerates the user certificates that are signed by it.
After renewal, you must ensure client applications update their truststores and keystores with the renewed user certificates before the old ones expire to avoid connection failures.
The Streams for Apache Kafka operators do not manage external client applications. You are responsible for ensuring that clients continue to connect successfully after certificate renewal.
When the Clients CA is renewed, the User Operator automatically regenerates user certificates. Client applications must be updated to use these renewed credentials before the old certificates expire.
Client applications must be configured with the following:
-
A truststore that includes credentials from the
<cluster_name>-cluster-ca-certsecret, which is created by the Cluster Operator and contains the public key to verify the Kafka cluster. -
A keystore built from the
<kafka_user_name>secret, which is created by the User Operator and contains the user’s certificate and key.
User secrets provide credentials in PEM and PKCS #12 formats, or a password if using SCRAM-SHA authentication. The User Operator creates these secrets when a user is created. For an example of configuring secure clients, see Section 16.4.2, “Securing user access to Kafka”.
If you provision client certificates manually, generate and distribute new certificates before the current ones expire. Failure to do so can result in clients being unable to connect to the Kafka cluster.
For workloads in the same OpenShift cluster and namespace, you can mount secrets as volumes. This allows client pods to construct keystores and truststores dynamically from the current state of the secrets. For details, see Configuring internal clients to trust the cluster CA.
18.3.4. Scheduling maintenance time windows
Use maintenance time windows to control when the Cluster Operator performs certificate renewals and the related rolling restarts. This helps minimize disruption to Kafka clients by scheduling updates at convenient times.
Maintenance time windows apply only to automatic certificate renewals and rolling restarts managed by the Cluster Operator.
They apply in the following scenarios:
- Automatic time-based renewal of Streams for Apache Kafka-managed internal certificates, such as those used by Kafka nodes.
- Automatic time-based renewal of server certificates signed by a custom Certificate Authority (CA), if Streams for Apache Kafka manages the renewal process based on certificate expiry.
They do not apply in the following scenarios:
-
User-driven changes to the
Kafkacustom resource, including configuration updates. - Environment-driven changes to certificate fields, such as the Common Name (CN) or Subject Alternative Names (SANs). For example, when a new load balancer address is introduced.
- Manual updates to Cluster CA or Clients CA certificates, even if they are Streams for Apache Kafka-managed, since these updates are treated as user-initiated actions.
Externally managed certificates, such as:
- Custom listener certificates provided directly by the user.
- Certificates issued by an external certificate manager.
In these cases, restarts may occur immediately and outside any configured maintenance time window.
Use time windows in conjunction with the renewal periods for CA certificates set in the Kafka resource (Kafka.spec.clusterCa.renewalDays and Kafka.spec.clientsCa.renewalDays).
Rolling restarts can be triggered in two ways:
-
By user-driven changes to the
Kafkacustom resource - Automatically, when Cluster Operator-managed certificates near expiration
User-driven restarts are not restricted by maintenance time windows. However, automatically triggered restarts for expiring certificates are subject to the configured maintenance windows.
Although restarts typically do not affect service availability, they can temporarily impact client performance. Use maintenance time windows to schedule automatic restarts occur during periods of low client activity.
Configure maintenance time windows as follows:
-
Configure an array of strings using the
Kafka.spec.maintenanceTimeWindowsproperty of theKafkaresource. - Each string is a cron expression interpreted as being in UTC (Coordinated Universal Time)
The following example configures a single maintenance time window that starts at midnight and ends at 01:59am (UTC), on Sundays, Mondays, Tuesdays, Wednesdays, and Thursdays.
Example maintenance time window configuration
The Cluster Operator doesn’t adhere strictly to the time windows. Maintenance operations are triggered by the first reconciliation that occurs within the specified time window. If the time window is shorter than the interval between reconciliations, there’s a risk that the reconciliation may happen outside of the time window, Therefore, maintenance time windows must be at least as long as the interval between reconciliations.
Cluster and clients CA certificates generated by the Cluster Operator auto-renew at the start of their respective certificate renewal periods. However, you can use the strimzi.io/force-renew annotation to manually renew one or both of these certificates before the certificate renewal period starts. You might do this for security reasons, or if you have changed the renewal or validity periods for the certificates.
A renewed certificate uses the same private key as the old certificate.
If you are using your own CA certificates, the force-renew annotation cannot be used. Instead, follow the procedure for renewing your own CA certificates.
Prerequisites
- The Cluster Operator must be deployed.
- A Kafka cluster in which CA certificates and private keys are installed.
- The OpenSSL TLS management tool to check the period of validity for CA certificates.
In this procedure, we use a Kafka cluster named my-cluster within the my-project namespace.
Procedure
Apply the
strimzi.io/force-renewannotation to the secret that contains the CA certificate that you want to renew.Renewing the Cluster CA secret
oc annotate secret my-cluster-cluster-ca-cert -n my-project strimzi.io/force-renew="true"
oc annotate secret my-cluster-cluster-ca-cert -n my-project strimzi.io/force-renew="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Renewing the Clients CA secret
oc annotate secret my-cluster-clients-ca-cert -n my-project strimzi.io/force-renew="true"
oc annotate secret my-cluster-clients-ca-cert -n my-project strimzi.io/force-renew="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow At the next reconciliation, the Cluster Operator generates new certificates.
If maintenance time windows are configured, the Cluster Operator generates the new CA certificate at the first reconciliation within the next maintenance time window.
Check the period of validity for the new CA certificates.
Checking the period of validity for the new cluster CA certificate
oc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesoc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Checking the period of validity for the new clients CA certificate
oc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesoc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesCopy to Clipboard Copied! Toggle word wrap Toggle overflow The command returns a
notBeforeandnotAfterdate, which is the valid start and end date for the CA certificate.Update client configurations to trust the new cluster CA certificate.
See:
The Cluster Operator automatically renews the cluster and clients CA certificates when their renewal periods begin. Nevertheless, unexpected operational problems or disruptions may prevent the renewal process, such as prolonged downtime of the Cluster Operator or unavailability of the Kafka cluster. If CA certificates expire, Kafka cluster components cannot communicate with each other and the Cluster Operator cannot renew the CA certificates without manual intervention.
To promptly perform a recovery, follow the steps outlined in this procedure in the order given. You can recover from expired cluster and clients CA certificates. The process involves deleting the secrets containing the expired certificates so that new ones are generated by the Cluster Operator. For more information on the secrets managed in Streams for Apache Kafka, see Section 18.2.2, “Secrets generated by the Cluster Operator”.
If you are using your own CA certificates and they expire, the process is similar, but you need to renew the CA certificates rather than use certificates generated by the Cluster Operator.
Prerequisites
- The Cluster Operator must be deployed.
- A Kafka cluster in which CA certificates and private keys are installed.
- The OpenSSL TLS management tool to check the period of validity for CA certificates.
In this procedure, we use a Kafka cluster named my-cluster within the my-project namespace.
Procedure
Delete the secret containing the expired CA certificate.
Deleting the Cluster CA secret
oc delete secret my-cluster-cluster-ca-cert -n my-project
oc delete secret my-cluster-cluster-ca-cert -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Deleting the Clients CA secret
oc delete secret my-cluster-clients-ca-cert -n my-project
oc delete secret my-cluster-clients-ca-cert -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the Cluster Operator to generate new certificates.
-
A new CA cluster certificate to verify the identity of the Kafka brokers is created in a secret of the same name (
my-cluster-cluster-ca-cert). -
A new CA clients certificate to verify the identity of Kafka users is created in a secret of the same name (
my-cluster-clients-ca-cert).
-
A new CA cluster certificate to verify the identity of the Kafka brokers is created in a secret of the same name (
Check the period of validity for the new CA certificates.
Checking the period of validity for the new cluster CA certificate
oc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesoc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Checking the period of validity for the new clients CA certificate
oc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesoc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -datesCopy to Clipboard Copied! Toggle word wrap Toggle overflow The command returns a
notBeforeandnotAfterdate, which is the valid start and end date for the CA certificate.Delete the component pods and secrets that use the CA certificates.
- Delete the Kafka secret.
- Wait for the Cluster Operator to detect the missing Kafka secret and recreate it.
- Delete all Kafka pods.
If you are only recovering the clients CA certificate, you only need to delete the Kafka secret and pods.
You can use the following
occommand to find resources and also verify that they have been removed.oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>
oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<resource_type>with the type of the resource, such asPodorSecret.Wait for the Cluster Operator to detect the missing Kafka pods and recreate them with the updated CA certificates.
On reconciliation, the Cluster Operator automatically updates other components to trust the new CA certificates.
- Verify that there are no issues related to certificate validation in the Cluster Operator log.
Update client configurations to trust the new cluster CA certificate.
See:
You can replace the private keys used by the cluster CA and clients CA certificates generated by the Cluster Operator. When a private key is replaced, the Cluster Operator generates a new CA certificate for the new private key.
If you are using your own CA certificates, the force-replace annotation cannot be used. Instead, follow the procedure for renewing your own CA certificates.
Prerequisites
- The Cluster Operator is running.
- A Kafka cluster in which CA certificates and private keys are installed.
Procedure
Apply the
strimzi.io/force-replaceannotation to theSecretthat contains the private key that you want to renew.Expand Table 18.12. Commands for replacing private keys Private key for Secret Annotate command Cluster CA
<cluster_name>-cluster-ca
oc annotate secret <cluster_name>-cluster-ca strimzi.io/force-replace="true"Clients CA
<cluster_name>-clients-ca
oc annotate secret <cluster_name>-clients-ca strimzi.io/force-replace="true"
At the next reconciliation the Cluster Operator will:
-
Generate a new private key for the
Secretthat you annotated - Generate a new CA certificate
If maintenance time windows are configured, the Cluster Operator will generate the new private key and CA certificate at the first reconciliation within the next maintenance time window.
Client applications must reload the cluster and clients CA certificates that were renewed by the Cluster Operator.
This procedure describes how to configure a Kafka client that resides inside the OpenShift cluster — connecting to a TLS listener — to trust the cluster CA certificate.
The easiest way to achieve this for an internal client is to use a volume mount to access the Secrets containing the necessary certificates and keys.
Follow the steps to configure trust certificates that are signed by the cluster CA for Java-based Kafka Producer, Consumer, and Streams APIs.
Choose the steps to follow according to the certificate format of the cluster CA: PKCS #12 (.p12) or PEM (.crt).
The steps describe how to mount the Cluster Secret that verifies the identity of the Kafka cluster to the client pod.
Prerequisites
- The Cluster Operator must be running.
-
There needs to be a
Kafkaresource within the OpenShift cluster. - You need a Kafka client application inside the OpenShift cluster that will connect using TLS, and needs to trust the cluster CA certificate.
-
The client application must be running in the same namespace as the
Kafkaresource.
Using PKCS #12 format (.p12)
Mount the cluster Secret as a volume when defining the client pod.
For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Here we’re mounting the following:
- The PKCS #12 file into an exact path, which can be configured
- The password into an environment variable, where it can be used for Java configuration
Configure the Kafka client with the following properties:
A security protocol option:
-
security.protocol: SSLwhen using TLS for encryption (with or without mTLS authentication). -
security.protocol: SASL_SSLwhen using SCRAM-SHA authentication over TLS.
-
-
ssl.truststore.locationwith the truststore location where the certificates were imported. -
ssl.truststore.passwordwith the password for accessing the truststore. -
ssl.truststore.type=PKCS12to identify the truststore type.
Using PEM format (.crt)
Mount the cluster Secret as a volume when defining the client pod.
For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Use the extracted certificate to configure a TLS connection in clients that use certificates in X.509 format.
This procedure describes how to configure a Kafka client that resides outside the OpenShift cluster – connecting to an external listener – to trust the cluster CA certificate. Follow this procedure when setting up the client and during the renewal period, when the old clients CA certificate is replaced.
Follow the steps to configure trust certificates that are signed by the cluster CA for Java-based Kafka Producer, Consumer, and Streams APIs.
Choose the steps to follow according to the certificate format of the cluster CA: PKCS #12 (.p12) or PEM (.crt).
The steps describe how to obtain the certificate from the Cluster Secret that verifies the identity of the Kafka cluster.
The <cluster_name>-cluster-ca-cert secret contains more than one CA certificate during the CA certificate renewal period. Clients must add all of them to their truststores.
Prerequisites
- The Cluster Operator must be running.
-
There needs to be a
Kafkaresource within the OpenShift cluster. - You need a Kafka client application outside the OpenShift cluster that will connect using TLS, and needs to trust the cluster CA certificate.
Using PKCS #12 format (.p12)
Extract the cluster CA certificate and password from the
<cluster_name>-cluster-ca-certSecret of the Kafka cluster.oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.passwordoc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.passwordCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <cluster_name> with the name of the Kafka cluster.
Configure the Kafka client with the following properties:
A security protocol option:
-
security.protocol: SSLwhen using TLS. -
security.protocol: SASL_SSLwhen using SCRAM-SHA authentication over TLS.
-
-
ssl.truststore.locationwith the truststore location where the certificates were imported. -
ssl.truststore.passwordwith the password for accessing the truststore. This property can be omitted if it is not needed by the truststore. -
ssl.truststore.type=PKCS12to identify the truststore type.
Using PEM format (.crt)
Extract the cluster CA certificate from the
<cluster_name>-cluster-ca-certsecret of the Kafka cluster.oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtoc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Use the extracted certificate to configure a TLS connection in clients that use certificates in X.509 format.
18.6. Using your own CA certificates and private keys
Install and use your own CA certificates and private keys instead of using the defaults generated by the Cluster Operator. You can replace the cluster and clients CA certificates and private keys.
You can switch to using your own CA certificates and private keys in the following ways:
- Install your own CA certificates and private keys before deploying your Kafka cluster
- Replace the default CA certificates and private keys with your own after deploying a Kafka cluster
The steps to replace the default CA certificates and private keys after deploying a Kafka cluster are the same as those used to renew your own CA certificates and private keys.
If you use your own certificates, they won’t be renewed automatically. You need to renew the CA certificates and private keys before they expire.
Renewal options:
- Renew the CA certificates only
- Renew CA certificates and private keys (or replace the defaults)
Install your own CA certificates and private keys instead of using the cluster and clients CA certificates and private keys generated by the Cluster Operator.
By default, Streams for Apache Kafka uses the following cluster CA and clients CA secrets, which are renewed automatically.
Cluster CA secrets
-
<cluster_name>-cluster-ca -
<cluster_name>-cluster-ca-cert
-
Clients CA secrets
-
<cluster_name>-clients-ca -
<cluster_name>-clients-ca-cert
-
To install your own certificates, use the same names.
Prerequisites
- The Cluster Operator is running.
A Kafka cluster is not yet deployed.
If you have already deployed a Kafka cluster, you can replace the default CA certificates with your own.
Your own X.509 certificates and keys in PEM format for the cluster CA or clients CA.
If you want to use a cluster or clients CA which is not a Root CA, you have to include the whole chain in the certificate file. The chain should be in the following order:
- The cluster or clients CA
- One or more intermediate CAs
- The root CA
- All CAs in the chain should be configured using the X509v3 Basic Constraints extension. Basic Constraints limit the path length of a certificate chain.
- The OpenSSL TLS management tool for converting certificates.
Before you begin
The Cluster Operator generates keys and certificates in PEM (Privacy Enhanced Mail) and PKCS #12 (Public-Key Cryptography Standards) formats. Only the keys and certificates in the PEM format are used internally by Streams for Apache Kafka. The PKCS #12 store is there only for user applications that do not support using the PEM format directly. When using custom CA certificates, adding the PKCS #12 store and its password to the secret is optional only.
Procedure
Create a new secret that contains the CA certificate.
Client secret creation with a certificate in PEM format only
oc create secret generic <cluster_name>-clients-ca-cert --from-file=ca.crt=ca.crt
oc create secret generic <cluster_name>-clients-ca-cert --from-file=ca.crt=ca.crtCopy to Clipboard Copied! Toggle word wrap Toggle overflow Cluster secret creation with certificates in PEM and PKCS #12 format
oc create secret generic <cluster_name>-cluster-ca-cert \ --from-file=ca.crt=ca.crt \ --from-file=ca.p12=ca.p12 \ --from-literal=ca.password=P12-PASSWORD
oc create secret generic <cluster_name>-cluster-ca-cert \ --from-file=ca.crt=ca.crt \ --from-file=ca.p12=ca.p12 \ --from-literal=ca.password=P12-PASSWORDCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <cluster_name> with the name of your Kafka cluster.
Create a new secret that contains the private key.
oc create secret generic <ca_key_secret> --from-file=ca.key=ca.key
oc create secret generic <ca_key_secret> --from-file=ca.key=ca.keyCopy to Clipboard Copied! Toggle word wrap Toggle overflow Label the secrets.
oc label secret <ca_certificate_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"
oc label secret <ca_certificate_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc label secret <ca_key_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"
oc label secret <ca_key_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Label
strimzi.io/kind=Kafkaidentifies the Kafka custom resource. -
Label
strimzi.io/cluster="<cluster_name>"identifies the Kafka cluster.
-
Label
Annotate the secrets
oc annotate secret <ca_certificate_secret> strimzi.io/ca-cert-generation="<ca_certificate_generation>"
oc annotate secret <ca_certificate_secret> strimzi.io/ca-cert-generation="<ca_certificate_generation>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc annotate secret <ca_key_secret> strimzi.io/ca-key-generation="<ca_key_generation>"
oc annotate secret <ca_key_secret> strimzi.io/ca-key-generation="<ca_key_generation>"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Annotation
strimzi.io/ca-cert-generation="<ca_certificate_generation>"defines the generation of a new CA certificate. Annotation
strimzi.io/ca-key-generation="<ca_key_generation>"defines the generation of a new CA key.Start from 0 (zero) as the incremental value (
strimzi.io/ca-cert-generation=0) for your own CA certificate. Set a higher incremental value when you renew the certificates.
-
Annotation
Create the
Kafkaresource for your cluster, configuring either theKafka.spec.clusterCaor theKafka.spec.clientsCaobject to not use generated CAs.Example fragment
Kafkaresource configuring the cluster CA to use certificates you supply for yourselfCopy to Clipboard Copied! Toggle word wrap Toggle overflow
18.6.2. Renewing your own CA certificates
If you are using your own CA certificates, you need to renew them manually. The Cluster Operator will not renew them automatically. Renew the CA certificates in the renewal period before they expire.
Perform the steps in this procedure when you are renewing CA certificates and continuing with the same private key. If you are renewing your own CA certificates and private keys, see Section 18.6.3, “Renewing or replacing CA certificates and private keys with your own”.
The procedure describes the renewal of CA certificates in PEM format.
Prerequisites
- The Cluster Operator is running.
- You have new cluster or clients X.509 certificates in PEM format.
Procedure
Update the
Secretfor the CA certificate.Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.
oc edit secret <ca_certificate_secret_name>
oc edit secret <ca_certificate_secret_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <ca_certificate_secret_name> is the name of the
Secret, which is<kafka_cluster_name>-cluster-ca-certfor the cluster CA certificate and<kafka_cluster_name>-clients-ca-certfor the clients CA certificate.The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named
my-cluster.Example secret configuration for a cluster CA certificate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Encode your new CA certificate into base64.
cat <path_to_new_certificate> | base64
cat <path_to_new_certificate> | base64Copy to Clipboard Copied! Toggle word wrap Toggle overflow Update the CA certificate.
Copy the base64-encoded CA certificate from the previous step as the value for the
ca.crtproperty underdata.Increase the value of the CA certificate generation annotation.
Update the
strimzi.io/ca-cert-generationannotation with a higher incremental value. For example, changestrimzi.io/ca-cert-generation=0tostrimzi.io/ca-cert-generation=1. If theSecretis missing the annotation, the value is treated as0, so add the annotation with a value of1.When Streams for Apache Kafka generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The
strimzi.io/ca-cert-generationhas to be incremented on each CA certificate renewal.Save the secret with the new CA certificate and certificate generation annotation value.
Example secret configuration updated with a new CA certificate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
On the next reconciliation, the Cluster Operator performs a rolling update of Kafka and other components to trust the new CA certificate.
If maintenance time windows are configured, the Cluster Operator will roll the pods at the first reconciliation within the next maintenance time window.
If you are using your own CA certificates and private keys, you need to renew them manually. The Cluster Operator will not renew them automatically. Renew the CA certificates in the renewal period before they expire. You can also use the same procedure to replace the CA certificates and private keys generated by the Streams for Apache Kafka operators with your own.
Perform the steps in this procedure when you are renewing or replacing CA certificates and private keys. If you are only renewing your own CA certificates, see Section 18.6.2, “Renewing your own CA certificates”.
The procedure describes the renewal of CA certificates and private keys in PEM format.
Before going through the following steps, make sure that the CN (Common Name) of the new CA certificate is different from the current one. For example, when the Cluster Operator renews certificates automatically it adds a v<version_number> suffix to identify a version. Do the same with your own CA certificate by adding a different suffix on each renewal. By using a different key to generate a new CA certificate, you retain the current CA certificate stored in the Secret.
Prerequisites
- The Cluster Operator is running.
- You have new cluster or clients X.509 certificates and keys in PEM format.
Procedure
Pause the reconciliation of the
Kafkacustom resource.Annotate the custom resource in OpenShift, setting the
pause-reconciliationannotation totrue:oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="true"
oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example, for a
Kafkacustom resource namedmy-cluster:oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"
oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the status conditions of the custom resource show a change to
ReconciliationPaused:oc describe Kafka <name_of_custom_resource>
oc describe Kafka <name_of_custom_resource>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
typecondition changes toReconciliationPausedat thelastTransitionTime.
Check the settings for the
generateCertificateAuthorityproperties in yourKafkacustom resource.If a property is set to
false, a CA certificate is not generated by the Cluster Operator. You require this setting if you are using your own certificates.If needed, edit the existing
Kafkacustom resource and set thegenerateCertificateAuthorityproperties tofalse.oc edit Kafka <name_of_custom_resource>
oc edit Kafka <name_of_custom_resource>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The following example shows a
Kafkacustom resource with both cluster and clients CA certificates generation delegated to the user.Example
Kafkaconfiguration using your own CA certificatesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Update the
Secretfor the CA certificate.Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.
oc edit secret <ca_certificate_secret_name>
oc edit secret <ca_certificate_secret_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <ca_certificate_secret_name> is the name of the
Secret, which is<kafka_cluster_name>-cluster-ca-certfor the cluster CA certificate and<kafka_cluster_name>-clients-ca-certfor the clients CA certificate.The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named
my-cluster.Example secret configuration for a cluster CA certificate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Rename the current CA certificate to retain it.
Rename the current
ca.crtproperty underdataasca-<date>.crt, where <date> is the certificate expiry date in the format YEAR-MONTH-DAYTHOUR-MINUTE-SECONDZ. For exampleca-2023-01-26T17-32-00Z.crt:. Leave the value for the property as it is to retain the current CA certificate.Encode your new CA certificate into base64.
cat <path_to_new_certificate> | base64
cat <path_to_new_certificate> | base64Copy to Clipboard Copied! Toggle word wrap Toggle overflow Update the CA certificate.
Create a new
ca.crtproperty underdataand copy the base64-encoded CA certificate from the previous step as the value forca.crtproperty.Increase the value of the CA certificate generation annotation.
Update the
strimzi.io/ca-cert-generationannotation with a higher incremental value. For example, changestrimzi.io/ca-cert-generation=0tostrimzi.io/ca-cert-generation=1. If theSecretis missing the annotation, the value is treated as0, so add the annotation with a value of1.When Streams for Apache Kafka generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The
strimzi.io/ca-cert-generationhas to be incremented on each CA certificate renewal.Save the secret with the new CA certificate and certificate generation annotation value.
Example secret configuration updated with a new CA certificate
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Update the
Secretfor the CA key used to sign your new CA certificate.Edit the existing secret to add the new CA key and update the key generation annotation value.
oc edit secret <ca_key_name>
oc edit secret <ca_key_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow <ca_key_name> is the name of CA key, which is
<kafka_cluster_name>-cluster-cafor the cluster CA key and<kafka_cluster_name>-clients-cafor the clients CA key.The following example shows a secret for a cluster CA key that’s associated with a Kafka cluster named
my-cluster.Example secret configuration for a cluster CA key
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Encode the CA key into base64.
cat <path_to_new_key> | base64
cat <path_to_new_key> | base64Copy to Clipboard Copied! Toggle word wrap Toggle overflow Update the CA key.
Copy the base64-encoded CA key from the previous step as the value for the
ca.keyproperty underdata.Increase the value of the CA key generation annotation.
Update the
strimzi.io/ca-key-generationannotation with a higher incremental value. For example, changestrimzi.io/ca-key-generation=0tostrimzi.io/ca-key-generation=1. If theSecretis missing the annotation, it is treated as0, so add the annotation with a value of1.When Streams for Apache Kafka generates certificates, the key generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates together with a new CA key, set the annotation with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates and keys. The
strimzi.io/ca-key-generationhas to be incremented on each CA certificate renewal.Save the secret with the new CA key and key generation annotation value.
Example secret configuration updated with a new CA key
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Resume from the pause.
To resume the
Kafkacustom resource reconciliation, set thepause-reconciliationannotation tofalse.oc annotate --overwrite Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="false"
oc annotate --overwrite Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="false"Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can also do the same by removing the
pause-reconciliationannotation.oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation-
oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation-Copy to Clipboard Copied! Toggle word wrap Toggle overflow On the next reconciliation, the Cluster Operator performs a rolling update of Kafka and other components to trust the new CA certificate. When the rolling update is complete, the Cluster Operator will start a new one to generate new server certificates signed by the new CA key.
If maintenance time windows are configured, the Cluster Operator will roll the pods at the first reconciliation within the next maintenance time window.
- Wait until the rolling updates to move to the new CA certificate are complete.
Remove any outdated certificates from the secret configuration to ensure that the cluster no longer trusts them.
oc edit secret <ca_certificate_secret_name>
oc edit secret <ca_certificate_secret_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example secret configuration with the old certificate removed
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Start a manual rolling update of your cluster to pick up the changes made to the secret configuration.
Security context defines constraints on pods and containers. By specifying a security context, pods and containers only have the permissions they need. For example, permissions can control runtime operations or access to resources.
19.1. Handling of security context by OpenShift platform
Handling of security context depends on the tooling of the OpenShift platform you are using.
For example, OpenShift uses built-in security context constraints (SCCs) to control permissions. SCCs are the settings and strategies that control the security features a pod has access to.
By default, OpenShift injects security context configuration automatically. In most cases, this means you don’t need to configure security context for the pods and containers created by the Cluster Operator. Although you can still create and manage your own SCCs.
For more information, see the OpenShift documentation.
Scaling Kafka clusters by adding brokers can improve performance and reliability. Increasing the number of brokers provides more resources, enabling the cluster to handle larger workloads and process more messages. It also enhances fault tolerance by providing additional replicas. Conversely, removing underutilized brokers can reduce resource consumption and increase efficiency. Scaling must be done carefully to avoid disruption or data loss. Redistributing partitions across brokers reduces the load on individual brokers, increasing the overall throughput of the cluster.
Adjusting the replicas configuration changes the number of brokers in a cluster. A replication factor of 3 means each partition is replicated across three brokers, ensuring fault tolerance in case of broker failure:
Example node pool configuration for the number of replicas
The actual replication factor for topics depends on the number of available brokers and how many brokers store replicas of each topic partition (configured by default.replication.factor). The minimum number of replicas that must acknowledge a write for it to be considered successful is defined by min.insync.replicas:
Example configuration for topic replication
When adding brokers by changing the number of replicas, node IDs start at 0, and the Cluster Operator assigns the next lowest available ID to new brokers. Removing brokers starts with the pod that has the highest node ID. Additionally, when scaling clusters with node pools, you can assign node IDs for scaling operations.
Streams for Apache Kafka can automatically reassign partitions when brokers are added or removed if Cruise Control is deployed and auto-rebalancing is enabled in the Kafka resource. If auto-rebalancing is disabled, you can use Cruise Control to generate optimization proposals before manually rebalancing the cluster.
Cruise Control provides add-brokers and remove-brokers modes for scaling:
-
Use the
add-brokersmode after scaling up to move partition replicas to the new brokers. -
Use the
remove-brokersmode before scaling down to move partition replicas off the brokers being removed.
With auto-rebalancing, these modes run automatically using the default Cruise Control configuration or custom settings from a rebalancing template.
To increase the throughput of a Kafka topic, you can increase the number of partitions for that topic, distributing the load across multiple brokers. However, if all brokers are constrained by a resource (such as I/O), adding more partitions won’t improve throughput, and adding more brokers is necessary.
20.1. Triggering auto-rebalances when scaling clusters
Set up auto-rebalancing to automatically redistribute topic partitions when scaling a cluster. You can scale a Kafka cluster by adjusting the number of brokers using the spec.replicas property in the Kafka or KafkaNodePool custom resource used in deployment. When auto-rebalancing is enabled, the cluster is rebalanced without further intervention.
- After adding brokers, topic partitions are redistributed across the new brokers.
- Before removing brokers, partitions are moved off the brokers being removed.
Auto-rebalancing helps maintain balanced load distribution across Kafka brokers during scaling operations, depending on how the rebalancing configuration is set up.
Scaling operates in two modes: add-brokers and remove-brokers. Each mode can have its own auto-rebalancing configuration specified in the Kafka resource under spec.cruiseControl.autoRebalance properties. Use the template property to specify a predefined KafkaRebalance resource, which serves as a rebalance configuration template. If a template is not specified in the autorebalance configuration, the default Cruise Control rebalancing configuration is used. You can apply the same template configuration for both scaling modes, use different configurations for each, or enable auto-rebalancing for only one mode. If autorebalance configuration is not set for a mode, auto-rebalancing will not occur for that mode.
The template KafkaRebalance resource must include the strimzi.io/rebalance-template: "true" annotation. The template does not represent an actual rebalance request but holds the rebalancing configuration. During scaling, the Cluster Operator creates a KafkaRebalance resource based on this template, named <cluster_name>-auto-rebalancing-<mode>, where <mode> is either add-brokers or remove-brokers. The Cluster Operator applies a finalizer (strimzi.io/auto-rebalancing) to prevent the resource’s deletion during the rebalancing process.
Progress is reflected in the status of the Kafka resource. The status.autoRebalance property indicates the state of the rebalance. A modes property lists the brokers being added or removed during the operation to help track progress across reconciliations.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
- You have configured optimization goals and, optionally, capacity limits on broker resources.
Procedure
Create a rebalancing template for the auto-rebalancing operation (if required).
Configure a
KafkaRebalanceresource with thestrimzi.io/rebalance-template: "true"annotation. The rebalance configuration template does not requiremodeandbrokersproperties unlike when when generating an optimization proposal for rebalancing.Example rebalancing template configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The annotation designates the resource as a rebalance configuration template.
- Apply the configuration to create the template.
Add auto-rebalancing configuration to the
Kafkaresource.In this example, the same template is used for adding and removing brokers.
Example using template specifications for auto-rebalancing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To use default Cruise Control configuration for rebalancing, omit the template configuration. In this example, the default configuration is used when adding brokers.
Example using default specifications for auto-rebalancing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the changes to the
Kafkaconfiguration.
Wait for the Cluster Operator to update the cluster. Scale the cluster by adjusting the
spec.replicasproperty representing the number of brokers in the cluster.The following example shows a node pool configuration for a cluster using three brokers (
replicas: 3).Example node pool configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For more information on scaling through node pools, see the following:
Check the rebalance status.
The status is visible in theKafkaresource.Example status for auto-rebalancing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The state of the rebalance, which shows
RebalanceOnScaleUpwhen adding brokers, andRebalanceOnScaleDownwhen removing brokers. Scale-down operations take precedence. Initial and final state (failed or successful) shows asIdle. - 2
- Rebalance operations grouped by mode, with a list of nodes to be added or removed.
During a rebalance, the status of the KafkaRebalance resource used for the rebalance is checked, and the auto-rebalance state is adjusted accordingly.
20.2. Skipping checks on scale-down operations
By default, Streams for Apache Kafka performs a check to ensure that there are no partition replicas on brokers before initiating a scale-down operation on a Kafka cluster. The check applies to nodes in node pools that perform the role of broker only or a dual role of broker and controller.
If replicas are found, the scale-down is not done in order to prevent potential data loss. To scale-down the cluster, no replicas must be left on the broker before trying to scale it down again.
However, there may be scenarios where you want to bypass this mechanism. Disabling the check might be necessary on busy clusters, for example, because new topics keep generating replicas for the broker. This situation can indefinitely block the scale-down, even when brokers are nearly empty. Overriding the blocking mechanism in this way has an impact: the presence of topics on the broker being scaled down will likely cause a reconciliation failure for the Kafka cluster.
You can bypass the blocking mechanism by annotating the Kafka resource for the Kafka cluster. Annotate the resource by setting the strimzi.io/skip-broker-scaledown-check annotation to true:
Adding the annotation to skip checks on scale-down operations
oc annotate Kafka my-kafka-cluster strimzi.io/skip-broker-scaledown-check="true"
oc annotate Kafka my-kafka-cluster strimzi.io/skip-broker-scaledown-check="true"
This annotation instructs Streams for Apache Kafka to skip the scale-down check. Replace my-kafka-cluster with the name of your specific Kafka resource.
To restore the check for scale-down operations, remove the annotation:
Removing the annotation to skip checks on scale-down operations
oc annotate Kafka my-kafka-cluster strimzi.io/skip-broker-scaledown-check-
oc annotate Kafka my-kafka-cluster strimzi.io/skip-broker-scaledown-check-Chapter 21. Using Cruise Control for cluster rebalancing
Cruise Control is an open-source application designed to run alongside Kafka to help optimize use of cluster resources by doing the following:
- Monitoring cluster workload
- Rebalancing partitions based on predefined constraints
Cruise Control operations help with running a more balanced Kafka cluster that uses brokers more efficiently.
As Kafka clusters evolve, some brokers may become overloaded while others remain underutilized. Cruise Control addresses this imbalance by modeling resource utilization at the replica level—including, CPU, disk, network load—and generating optimization proposals (which you can approve or reject) for balanced partition assignments based on configurable optimization goals.
Optimization proposals are configured and generated using a KafkaRebalance resource. You can configure the resource using an annotation so that optimization proposals are approved automatically or manually.
Streams for Apache Kafka provides example configuration files for Cruise Control.
21.1. Cruise Control components and features
Cruise Control comprises four main components:
- Load Monitor
- Load Monitor collects the metrics and analyzes cluster workload data.
- Analyzer
- Analyzer generates optimization proposals based on collected data and configured goals.
- Anomaly Detector
- Anomaly Detector identifies and reports irregularities in cluster behavior.
- Executor
- Executor applies approved optimization proposals to the cluster.
- REST API
Cruise Control provides a REST API for client interactions, which Streams for Apache Kafka uses to support these features:
- Generating optimization proposals from optimization goals
- Rebalancing a Kafka cluster based on an optimization proposal
- Changing topic replication factor
- Reassigning partitions between JBOD disks
Cruise Control self-healing is not supported. Notifications and custom goals may be introduced through customization. For example, an anomaly notifier or custom goals can be added by including the appropriate JAR files in a custom image based on the Streams for Apache Kafka Kafka image.
21.1.1. Optimization goals
Optimization goals define objectives for rebalancing, such as distributing topic replicas evenly across brokers.
They are categorized as follows:
- Supported goals are a list of goals supported by the Cruise Control instance that can be used in its operations. By default, this list includes all goals included with Cruise Control. For a goal to be used in other categories, such as default or hard goals, it must first be listed in supported goals. To prevent a goal’s usage, remove it from this list.
- Hard goals are preset and must be satisfied for a proposal to succeed.
- Soft goals are preset goals with objectives that are prioritized during optimization as much as possible, without preventing a proposal from being created if all hard goals are satisfied.
- Default goals refer to the goals used by default when generating proposals. They match the supported goals unless specifically set by the user.
- Proposal-specific goals are a subset of supported goals configured for specific proposals.
Configure optimization goals in the Kafka and KafkaRebalance custom resources.
Kafkaresource for supported, hard, and default goals.-
Supported goals:
Kafka.spec.cruiseControl.config.goals -
Hard goals:
Kafka.spec.cruiseControl.config.hard.goals -
Default goals:
Kafka.spec.cruiseControl.config.default.goals
-
Supported goals:
KafkaRebalanceresource for proposal-specific goals.-
Proposal-specific goals:
KafkaRebalance.spec.goals
-
Proposal-specific goals:
21.1.1.1. Supported goals
Supported goals are predefined and available to use for generating Cruise Control optimization proposals. Goals not listed as supported goals cannot be used in Cruise Control operations. Some supported goals are preset as hard goals.
Configure supported goals in Kafka.spec.cruiseControl.config.goals:
-
To accept inherited supported goals, omit the
goalsproperty. -
To modify supported goals, specify the goals in descending priority order in the
goalsproperty.
21.1.1.2. Hard and soft goals
Hard goals must be satisfied for optimization proposals to be generated. Soft goals are best-effort objectives that Cruise Control tries to meet after all hard goals are satisfied. The classification of hard and soft goals is fixed in Cruise Control code and cannot be changed.
Cruise Control first prioritizes satisfying hard goals, and then addresses soft goals in the order they are listed. A proposal meeting all hard goals is valid, even if it violates some soft goals.
For example, a soft goal might be to evenly distribute a topic’s replicas. Cruise Control continues to generate an optimization proposal even if the soft goal isn’t completely satisfied.
Configure hard goals in your Cruise Control deployment using Kafka.spec.cruiseControl.config.hard.goals:
-
To enforce all hard goals, omit the
hard.goalsproperty. -
To specify hard goals, list them in
hard.goals. -
To exclude a hard goal, ensure it’s not in either
default.goalsorhard.goals.
Increasing the number of configured hard goals will reduce the likelihood of Cruise Control generating optimization proposals.
21.1.1.3. Default goals
Cruise Control uses default goals to generate an optimization proposal.
If default.goals is not specified in the Cruise Control deployment configuration, Streams for Apache Kafka configures default.goals to the list of supported goals specified in goals.
The optimization proposal based on this supported goals list is then generated and cached.
Configure default goals in Kafka.spec.cruiseControl.config.default.goals:
-
To use supported goals as default, omit the
default.goalsproperty. -
To modify default goals, specify a subset of supported goals in the
default.goalsproperty.
You can adjust the priority order in the default goals configuration.
21.1.1.4. Proposal-specific goals
Proposal-specific optimization goals support the creation of optimization proposals based on a specific list of goals. If proposal-specific goals are not set in the KafkaRebalance resource, then default goals are used
Configure proposal-specific goals in KafkaRebalance.spec.goals, specifying a subset of supported optimization goals for customization.
For example, you can optimize topic leader replica distribution across the Kafka cluster without considering disk capacity or utilization by defining a single proposal-specific goal.
21.1.1.5. Goals order of priority
Unless you change the Cruise Control deployment configuration, Streams for Apache Kafka inherits goals from Cruise Control, in descending priority order.
The following list shows supported goals inherited by Streams for Apache Kafka from Cruise Control in descending priority order. Goals labeled as hard are mandatory constraints that must be satisfied for optimization proposals.
-
RackAwareGoal(hard) -
MinTopicLeadersPerBrokerGoal(hard) -
ReplicaCapacityGoal(hard) -
DiskCapacityGoal(hard) -
NetworkInboundCapacityGoal(hard) -
NetworkOutboundCapacityGoal(hard) -
CpuCapacityGoal(hard) -
ReplicaDistributionGoal -
PotentialNwOutGoal -
DiskUsageDistributionGoal -
NetworkInboundUsageDistributionGoal -
NetworkOutboundUsageDistributionGoal -
CpuUsageDistributionGoal -
TopicReplicaDistributionGoal -
LeaderReplicaDistributionGoal -
LeaderBytesInDistributionGoal -
PreferredLeaderElectionGoal -
IntraBrokerDiskCapacityGoal(hard) -
IntraBrokerDiskUsageDistributionGoal
Resource distribution goals are subject to capacity limits on broker resources.
For more information on each optimization goal, see Goals in the Cruise Control Wiki.
Example Kafka configuration for default and hard goals
Ensure that the supported goals, default.goals, and (unless skipHardGoalCheck is set to true) proposal-specific spec.goals include all hard goals specified in hard.goals to avoid errors when generating optimization proposals. Hard goals must be included as a subset in the supported, default, and proposal-specific goals.
Example KafkaRebalance configuration for proposal-specific goals
21.1.1.6. Skipping hard goal checks
If skipHardGoalCheck: true is specified in the KafkaRebalance custom resource, Cruise Control does not verify that the proposal-specific goals include all the configured hard goals. This allows for more flexibility in generating optimization proposals, but may lead to proposals that do not satisfy all hard goals.
However, any hard goals included in the proposal-specific goals will still be treated as hard goals by Cruise Control, even with skipHardGoalCheck: true.
21.1.2. Optimization proposals
Optimization proposals are summaries of proposed changes based on the defined optimization goals, assessed in a specific order of priority. You can approve or reject proposals and rerun them with adjusted goals if needed.
With Cruise Control deployed for use in Streams for Apache Kafka, the process to generate and approve an optimization proposal is as follows:
-
Create a
KafkaRebalanceresource specifying optimization goals and any specific configurations. This resource triggers Cruise Control to initiate the optimization proposal generation process. -
A Cruise Control Metrics Reporter runs in every Kafka broker, collecting raw metrics and publishing them to a dedicated Kafka topic (
strimzi.cruisecontrol.metrics). Metrics for brokers, topics, and partitions are aggregated, sampled, and stored in other topics automatically created when Cruise Control is deployed. - Load Monitor collects, processes, and stores the metrics as a workload model--including CPU, disk, and network utilization data—which is used by the Analyzer and Anomaly Detector.
- Anomaly Detector continuously monitors the health and performance of the Kafka cluster, checking for things like broker failures or disk capacity issues, that could impact cluster stability.
-
Analyzer creates optimization proposals based on the workload model from the Load Monitor. Based on configured goals and capacities, it generates an optimization proposal for balancing partitions across brokers. Through the REST API, a summary of the proposal is reflected in the status of the
KafkaRebalanceresource. - The optimization proposal is approved or rejected (manually or automatically) based on its alignment with cluster management goals.
- If approved, the Executor applies the optimization proposal to rebalance the Kafka cluster. This involves reassigning partitions and redistributing workload across brokers according to the approved proposal.
Figure 21.1. Cruise Control optimization process
Optimization proposals comprise a list of partition reassignment mappings. When you approve a proposal, the Cruise Control server applies these partition reassignments to the Kafka cluster.
A partition reassignment consists of either of the following types of operations:
Partition movement: Involves transferring the partition replica and its data to a new location. Partition movements can take one of two forms:
- Inter-broker movement: The partition replica is moved to a log directory on a different broker.
- Intra-broker movement: The partition replica is moved to a different log directory on the same broker.
- Leadership movement: Involves switching the leader of the partition’s replicas.
Cruise Control issues partition reassignments to the Kafka cluster in batches. The performance of the cluster during the rebalance is affected by the number and magnitude of each type of movement contained in each batch.
21.1.2.1. Rebalancing modes
Proposals for rebalances can be generated in four modes, which are specified using the spec.mode property of the KafkaRebalance custom resource.
fullmode-
The
fullmode runs a full rebalance by moving replicas across all the brokers in the cluster. This is the default mode if thespec.modeproperty is not defined in theKafkaRebalancecustom resource. add-brokersmode-
The
add-brokersmode is used after scaling up a Kafka cluster by adding one or more brokers. Normally, after scaling up a Kafka cluster, new brokers are used to host only the partitions of newly created topics. If no new topics are created, the newly added brokers are not used and the existing brokers remain under the same load. By using theadd-brokersmode immediately after adding brokers to the cluster, the rebalancing operation moves replicas from existing brokers to the newly added brokers. You specify the new brokers as a list using thespec.brokersproperty of theKafkaRebalancecustom resource. remove-brokersmode-
The
remove-brokersmode is used before scaling down a Kafka cluster by removing one or more brokers. Theremove-brokersmode moves replicas off the brokers that are going to be removed. When these brokers are not hosting replicas anymore, you can safely run the scaling down operation. You specify the brokers you’re removing as a list in thespec.brokersproperty in theKafkaRebalancecustom resource. remove-disksmode-
The
remove-disksmode is used specifically to reassign partitions between JBOD disks used for storage on the same broker. You specify a list of broker IDs with corresponding volume IDs for partition reassignment.
Brokers are shut down even if they host replicas when checks are skipped on scale-down operations.
In general, use the full rebalance mode to rebalance a Kafka cluster by spreading the load across brokers. Use the add-brokers and remove-brokers modes only if you want to scale your cluster up or down and rebalance the replicas accordingly.
The procedure to run a rebalance is actually the same across the three different modes. The only difference is with specifying a mode through the spec.mode property and, if needed, listing brokers that have been added or will be removed through the spec.brokers property.
21.1.2.2. The results of an optimization proposal
When an optimization proposal is generated, a summary and broker load is returned.
- Summary
-
The summary is contained in the
KafkaRebalanceresource. The summary provides an overview of the proposed cluster rebalance and indicates the scale of the changes involved. A summary of a successfully generated optimization proposal is contained in theStatus.optimizationResultproperty of theKafkaRebalanceresource. The information provided is a summary of the full optimization proposal. - Broker load
- The broker load is stored in a ConfigMap that contains data as a JSON string. The broker load shows before and after values for the proposed rebalance, so you can see the impact on each of the brokers in the cluster.
An optimization proposal summary shows the proposed scope of changes.
You can use the name of the KafkaRebalance resource to return a summary from the command line.
Returning an optimization proposal summary
oc describe kafkarebalance <kafka_rebalance_resource_name> -n <namespace>
oc describe kafkarebalance <kafka_rebalance_resource_name> -n <namespace>
You can also use the jq command line JSON parser tool.
Returning an optimization proposal result using jq
oc get kafkarebalance <kafka_rebalance_resource_name> -n <namespace> -o json | jq '.status.optimizationResult'
oc get kafkarebalance <kafka_rebalance_resource_name> -n <namespace> -o json | jq '.status.optimizationResult'Use the summary to decide whether to approve or reject an optimization proposal.
- Approving an optimization proposal
-
You approve the optimization proposal by setting the
strimzi.io/rebalanceannotation of theKafkaRebalanceresource toapprove. Cruise Control applies the proposal to the Kafka cluster and starts a cluster rebalance operation. - Rejecting an optimization proposal
-
If you choose not to approve an optimization proposal, you can change the optimization goals or update any of the rebalance performance tuning options, and then generate another proposal. You can generate a new optimization proposal for a
KafkaRebalanceresource by setting thestrimzi.io/rebalanceannotation torefresh.
Use optimization proposals to assess the movements required for a rebalance. For example, a summary describes inter-broker and intra-broker movements. Inter-broker rebalancing moves data between separate brokers. Intra-broker rebalancing moves data between disks on the same broker when you are using a JBOD storage configuration. Such information can be useful even if you don’t go ahead and approve the proposal.
You might reject an optimization proposal, or delay its approval, because of the additional load on a Kafka cluster when rebalancing. If the proposal is delayed for too long, the cluster load may change significantly, so it may be better to request a new proposal.
In the following example, the proposal suggests the rebalancing of data between separate brokers. The rebalance involves the movement of 55 partition replicas, totaling 12MB of data, across the brokers. The proposal will also move 24 partition leaders to different brokers. This requires a change to the cluster metadata, which has a low impact on performance.
The balancedness scores are measurements of the overall balance of the Kafka cluster before and after the optimization proposal is approved. A balancedness score is based on optimization goals. If all goals are satisfied, the score is 100. The score is reduced for each goal that will not be met. Compare the balancedness scores to see whether the Kafka cluster is less balanced than it could be following a rebalance.
Example optimization proposal summary
Though the inter-broker movement of partition replicas has a high impact on performance, the total amount of data is not large. If the total data was much larger, you could reject the proposal, or time when to approve the rebalance to limit the impact on the performance of the Kafka cluster.
Rebalance performance tuning options can help reduce the impact of data movement. If you can extend the rebalance period, you can divide the rebalance into smaller batches. Fewer data movements at a single time reduces the load on the cluster.
21.1.2.4. Optimization proposal summary properties
The following table explains the properties contained in the optimization proposal’s summary.
| JSON property | Description |
|---|---|
|
| The total number of partition replicas that will be transferred between the disks of the cluster’s brokers.
Performance impact during rebalance operation: Relatively high, but lower than |
|
| Not yet supported. An empty list is returned. |
|
| The number of partition replicas that will be moved between separate brokers. Performance impact during rebalance operation: Relatively high. |
|
| A measurement of the overall balancedness of a Kafka Cluster, before and after the optimization proposal was generated.
The score is calculated by subtracting the sum of the
The |
|
|
The sum of the size of each partition replica that will be moved between disks on the same broker (see also
Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see |
|
| The number of metrics windows upon which the optimization proposal is based. |
|
|
The sum of the size of each partition replica that will be moved to a separate broker (see also Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. |
|
|
The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of |
|
|
If you specified a regular expression in the |
|
| The number of partitions whose leaders will be switched to different replicas. Performance impact during rebalance operation: Relatively low. |
|
| Not yet supported. An empty list is returned. |
To save time, you can automate the process of approving optimization proposals. With automation, when you generate an optimization proposal it goes straight into a cluster rebalance.
To enable the optimization proposal auto-approval mechanism, create the KafkaRebalance resource with the strimzi.io/rebalance-auto-approval annotation set to true. If the annotation is not set or set to false, the optimization proposal requires manual approval.
Example rebalance request with auto-approval mechanism enabled
You can still check the status when automatically approving an optimization proposal. The status of the KafkaRebalance resource moves to Ready when the rebalance is complete.
21.1.2.6. Comparing broker load data
Broker load data provides insights into current and anticipated usage of resources following a rebalance. The data is stored in a ConfigMap (with the same name as the KafkaRebalance resource) as a JSON formatted string
When a Kafka rebalance proposal reaches the ProposalReady state, Streams for Apache Kafka creates a ConfigMap (named after the KafkaRebalance custom resource) containing a JSON string of broker metrics generated from Cruise Control. Each broker has a set of key metrics represented by three values:
- The current metric value before the optimization proposal is applied
- The expected metric value after applying the proposal
- The difference between the two values (after minus before)
This ConfigMap remains accessible even after the rebalance completes.
To view this data from the command line, use the ConfigMap name.
Returning ConfigMap data
oc describe configmaps <my_rebalance_configmap_name> -n <namespace>
oc describe configmaps <my_rebalance_configmap_name> -n <namespace>
You can also use the jq command line JSON parser tool to extract the JSON string.
Extracting the JSON string from the ConfigMap using jq
oc get configmaps <my_rebalance_configmap_name> -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.'
oc get configmaps <my_rebalance_configmap_name> -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.'| JSON property | Description |
|---|---|
|
| The number of replicas on this broker that are partition leaders. |
|
| The number of replicas on this broker. |
|
| The CPU utilization as a percentage of the defined capacity. |
|
| The disk utilization as a percentage of the defined capacity. |
|
| The absolute disk usage in MB. |
|
| The total network output rate for the broker. |
|
| The network input rate for all partition leader replicas on this broker. |
|
| The network input rate for all follower replicas on this broker. |
|
| The hypothetical maximum network output rate that would be realized if this broker became the leader of all the replicas it currently hosts. |
21.1.2.7. Adjusting the cached proposal refresh rate
Cruise Control maintains a cached optimization proposal based on the configured default optimization goals. This proposal is generated from the workload model and updated every 15 minutes to reflect the current state of the Kafka cluster. When you generate an optimization proposal using the default goals, Cruise Control returns the latest cached version.
For clusters with rapidly changing workloads, you may want to shorten the refresh interval to ensure the optimization proposal reflects the most recent state. However, reducing the interval increases the load on the Cruise Control server. To adjust the refresh rate, modify the proposal.expiration.ms setting in the Cruise Control deployment configuration.
21.1.3. Tuning options for rebalances
Configuration options allow you to fine-tune cluster rebalance performance. These settings control the movement of partition replicas and leadership, as well as the bandwidth allocated for rebalances.
21.1.3.1. Selecting replica movement strategies
Cluster rebalance performance is also influenced by the replica movement strategy that is applied to the batches of partition reassignment commands. By default, Cruise Control uses the BaseReplicaMovementStrategy, which applies the reassignments in the order they were generated. However, this strategy could lead to the delay of other partition reassignments if large partition reassignments are generated then ordered first.
Cruise Control provides four alternative replica movement strategies that can be applied to optimization proposals:
-
PrioritizeSmallReplicaMovementStrategy: Reassign smaller partitions first. -
PrioritizeLargeReplicaMovementStrategy: Reassign larger partitions first. -
PostponeUrpReplicaMovementStrategy: Prioritize partitions without out-of-sync replicas. -
PrioritizeMinIsrWithOfflineReplicasStrategy: Prioritize reassignments for partitions at or below their minimum in-sync replicas (MinISR) with offline replicas.
SetcruiseControl.config.concurrency.adjuster.min.isr.check.enabledtotruein theKafkaresource to enable this strategy.
These strategies can be configured as a sequence. The first strategy attempts to compare two partition reassignments using its internal logic. If the reassignments are equivalent, then it passes them to the next strategy in the sequence to decide the order, and so on.
21.1.3.2. Intra-broker disk balancing
Intra-broker balancing shifts data between disks on the same broker, useful for deployments with JBOD storage and multiple disks. This type of balancing incurs less network overhead than inter-broker balancing.
If you are using JBOD storage with a single disk, intra-broker disk balancing will result in a proposal with 0 partition movements since there are no disks to balance.
To enable intra-broker balancing, set rebalanceDisk to true in KafkaRebalance.spec. When this is enabled, do not specify a goals field, as Cruise Control will automatically configure intra-broker goals and disregard inter-broker goals. Cruise Control does not perform inter-broker and intra-broker balancing at the same time.
21.1.3.3. Rebalance tuning
You can set the following rebalance tuning options when configuring Cruise Control or individual rebalances:
-
Set Cruise Control server configurations in
Kafka.spec.cruiseControl.configin theKafkaresource. -
Set proposal-specific configurations in
KafkaRebalance.specin theKafkaRebalanceresource.
| Cruise Control properties | KafkaRebalance properties | Default | Description |
|---|---|---|---|
|
|
| 5 | The maximum number of inter-broker partition movements in each partition reassignment batch |
|
|
| 2 | The maximum number of intra-broker partition movements in each partition reassignment batch |
|
|
| 1000 | The maximum number of partition leadership changes in each partition reassignment batch |
|
|
| Null (no limit) | The bandwidth (in bytes per second) to assign to partition reassignment |
|
|
|
|
The list of strategies (in priority order) used to determine the order in which partition reassignment commands are executed for generated proposals. For the server setting, use a comma separated string with the fully qualified names of the strategy class (add |
| - |
| false | Enables intra-broker disk balancing, which balances disk space utilization between disks on the same broker. Only applies to Kafka deployments that use JBOD storage with multiple disks. |
Changing the default settings affects the length of time that the rebalance takes to complete, as well as the load placed on the Kafka cluster during the rebalance. Using lower values reduces the load but increases the amount of time taken, and vice versa.
21.2. Deploying Cruise Control with Kafka
Configure a Kafka resource to deploy Cruise Control alongside a Kafka cluster. You can use the cruiseControl properties of the Kafka resource to configure the deployment. Deploy one instance of Cruise Control per Kafka cluster.
Use goals configuration in the Cruise Control config to specify optimization goals for generating optimization proposals. You can use brokerCapacity to change the default capacity limits for goals related to resource distribution. If brokers are running on nodes with heterogeneous network resources, you can use overrides to set network capacity limits for each broker.
If an empty object ({}) is used for the cruiseControl configuration, all properties use their default values.
Streams for Apache Kafka provides example configuration files, which include Kafka custom resources with Cruise Control configuration. For more information on the configuration options for Cruise Control, see the Streams for Apache Kafka Custom Resource API Reference.
Prerequisites
Procedure
Edit the
cruiseControlproperty for theKafkaresource.The properties you can configure are shown in this example configuration:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Capacity limits for broker resources.
- 2
- Overrides set network capacity limits for specific brokers when running on nodes with heterogeneous network resources.
- 3
- Cruise Control configuration. Standard Cruise Control configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka.
- 4
- Optimization goals configuration, which can include configuration for default optimization goals (
default.goals), supported optimization goals (goals), and hard goals (hard.goals). - 5
- CORS enabled and configured for read-only access to the Cruise Control API.
- 6
- Requests for reservation of supported resources, currently
cpuandmemory, and limits to specify the maximum resources that can be consumed. - 7
- Cruise Control loggers and log levels added directly (
inline) or indirectly (external) through aConfigMap. Custom Log4j configuration must be placed under thelog4j2.propertieskey in theConfigMap. You can set log levels toINFO,ERROR,WARN,TRACE,DEBUG,FATALorOFF. - 8
- Template customization. Here a pod is scheduled with additional security attributes.
- 9
- Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).
- 10
- Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter (the default metrics exporter).
Create or update the resource:
oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the deployment:
oc get deployments -n <my_cluster_operator_namespace>
oc get deployments -n <my_cluster_operator_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Output shows the deployment name and readiness
NAME READY UP-TO-DATE AVAILABLE my-cluster-cruise-control 1/1 1 1
NAME READY UP-TO-DATE AVAILABLE my-cluster-cruise-control 1/1 1 1Copy to Clipboard Copied! Toggle word wrap Toggle overflow my-clusteris the name of the Kafka cluster.READYshows the number of replicas that are ready/expected. The deployment is successful when theAVAILABLEoutput shows1.
21.2.1. Auto-created Cruise Control topics
The following table shows the three topics that are automatically created when Cruise Control is deployed. These topics are required for Cruise Control to work properly and must not be deleted or changed. You can change the name of the topic using the specified configuration option.
| Auto-created topic configuration | Default topic name | Created by | Function |
|---|---|---|---|
|
|
| Streams for Apache Kafka Metrics Reporter | Stores the raw metrics from the Metrics Reporter in each Kafka broker. |
|
|
| Cruise Control | Stores the derived metrics for each partition. These are created by the Metric Sample Aggregator. |
|
|
| Cruise Control | Stores the metrics samples used to create the Cluster Workload Model. |
To prevent the removal of records that are needed by Cruise Control, log compaction is disabled in the auto-created topics.
If the names of the auto-created topics are changed in a Kafka cluster that already has Cruise Control enabled, the old topics will not be deleted and should be manually removed.
What to do next
After configuring and deploying Cruise Control, you can generate optimization proposals.
21.3. Generating optimization proposals
When you create or update a KafkaRebalance resource, Cruise Control generates an optimization proposal for the Kafka cluster based on a set of optimization goals. Analyze the information in the optimization proposal and decide whether to approve it. You can use the results of the optimization proposal to rebalance your Kafka cluster.
This procedure covers using the following modes for generating optimization proposals related to rebalances:
-
full(default) -
add-brokers -
remove-brokers
The mode you use depends on whether you are rebalancing across all the brokers already running in the Kafka cluster; or you want to rebalance after scaling up or before scaling down your Kafka cluster. For more information, see Rebalancing modes with broker scaling.
Prerequisites
- You have deployed Cruise Control to your Streams for Apache Kafka cluster.
- You have configured optimization goals and, optionally, capacity limits on broker resources.
For more information on configuring Cruise Control, see Section 21.2, “Deploying Cruise Control with Kafka”.
Procedure
Create a
KafkaRebalanceresource and specify the appropriate mode.fullmode (default)To use the default optimization goals defined in the
Kafkaresource, leave thespecproperty empty. Cruise Control rebalances a Kafka cluster infullmode by default.Example configuration with full rebalancing by default
Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can also run a full rebalance by specifying the
fullmode through thespec.modeproperty.Example configuration specifying
fullmodeCopy to Clipboard Copied! Toggle word wrap Toggle overflow add-brokersmodeIf you want to rebalance a Kafka cluster after scaling up, specify the
add-brokersmode.In this mode, existing replicas are moved to the newly added brokers. You need to specify the brokers as a list.
Example configuration specifying
add-brokersmodeCopy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- List of newly added brokers added by the scale up operation. This property is mandatory.
remove-brokersmodeIf you want to rebalance a Kafka cluster before scaling down, specify the
remove-brokersmode.In this mode, replicas are moved off the brokers that are going to be removed. You need to specify the brokers that are being removed as a list.
Example configuration specifying
remove-brokersmodeCopy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- List of brokers to be removed by the scale down operation. This property is mandatory.
NoteThe following steps and the steps to approve or stop a rebalance are the same regardless of the rebalance mode you are using.
To configure proposal-specific optimization goals instead of using the default goals, add the
goalsproperty and enter one or more goals.In the following example, rack awareness and replica capacity are configured as proposal-specific optimization goals:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To ignore the configured hard goals, add the
skipHardGoalCheck: trueproperty:Copy to Clipboard Copied! Toggle word wrap Toggle overflow (Optional) To approve the optimization proposal automatically, set the
strimzi.io/rebalance-auto-approvalannotation totrue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create or update the resource:
oc apply -f <kafka_rebalance_configuration_file>
oc apply -f <kafka_rebalance_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow The Cluster Operator requests the optimization proposal from Cruise Control. This might take a few minutes depending on the size of the Kafka cluster.
If you used the automatic approval mechanism, wait for the status of the optimization proposal to change to
Ready. If you haven’t enabled the automatic approval mechanism, wait for the status of the optimization proposal to change toProposalReady:oc get kafkarebalance -o wide -w -n <namespace>
oc get kafkarebalance -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow PendingProposal-
A
PendingProposalstatus means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready. ProposalReady-
A
ProposalReadystatus means the optimization proposal is ready for review and approval.
When the status changes to
ProposalReady, the optimization proposal is ready to approve.Review the optimization proposal.
The optimization proposal is contained in the
Status.Optimization Resultproperty of theKafkaRebalanceresource.oc describe kafkarebalance <kafka_rebalance_resource_name>
oc describe kafkarebalance <kafka_rebalance_resource_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example optimization proposal
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The properties in the
Optimization Resultsection describe the pending cluster rebalance operation. For descriptions of each property, see Contents of optimization proposals.
Insufficient CPU capacity
If a Kafka cluster is overloaded in terms of CPU utilization, you might see an insufficient CPU capacity error in the KafkaRebalance status. It’s worth noting that this utilization value is unaffected by the excludedTopics configuration. Although optimization proposals will not reassign replicas of excluded topics, their load is still considered in the utilization calculation.
Example CPU utilization error
com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [CpuCapacityGoal] Insufficient capacity for cpu (Utilization 615.21, Allowed Capacity 420.00, Threshold: 0.70). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0.
com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException:
[CpuCapacityGoal] Insufficient capacity for cpu (Utilization 615.21, Allowed Capacity 420.00, Threshold: 0.70). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0.
The error shows CPU capacity as a percentage rather than the number of CPU cores. For this reason, it does not directly map to the number of CPUs configured in the Kafka custom resource. It is like having a single virtual CPU per broker, which has the cycles of the CPUs configured in Kafka.spec.kafka.resources.limits.cpu. This has no effect on the rebalance behavior, since the ratio between CPU utilization and capacity remains the same.
What to do next
21.4. Approving optimization proposals
You can approve an optimization proposal generated by Cruise Control, if its status is ProposalReady. Cruise Control will then apply the optimization proposal to the Kafka cluster, reassigning partitions to brokers and changing partition leadership.
This is not a dry run. Before you approve an optimization proposal, you must:
- Refresh the proposal in case it has become out of date.
- Carefully review the contents of the proposal.
Prerequisites
- You have generated an optimization proposal from Cruise Control.
-
The
KafkaRebalancecustom resource status isProposalReady.
Procedure
Perform these steps for the optimization proposal that you want to approve.
Unless the optimization proposal is newly generated, check that it is based on current information about the state of the Kafka cluster. To do so, annotate the
KafkaRebalanceresource to refresh the optimization proposal and make sure it uses the latest cluster metrics:oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"
oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the status of the optimization proposal to change to
ProposalReady:oc get kafkarebalance -o wide -w -n <namespace>
oc get kafkarebalance -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow PendingProposal-
A
PendingProposalstatus means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready. ProposalReady-
A
ProposalReadystatus means the optimization proposal is ready for review and approval.
When the status changes to
ProposalReady, the optimization proposal is ready to approve.Annotate the
KafkaRebalanceresource to approve the optimization proposal:oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="approve"
oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="approve"Copy to Clipboard Copied! Toggle word wrap Toggle overflow - The Cluster Operator detects the annotated resource and instructs Cruise Control to rebalance the Kafka cluster.
Wait for the status of the optimization proposal to change to
Ready:oc get kafkarebalance -o wide -w -n <namespace>
oc get kafkarebalance -o wide -w -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Rebalancing-
A
Rebalancingstatus means the rebalancing is in progress. Ready-
A
Readystatus means the rebalance is complete. NotReady-
A
NotReadystatus means an error occurred—see Fixing problems with aKafkaRebalanceresource.
When the status changes to
Ready, the rebalance is complete.To use the same
KafkaRebalancecustom resource to generate another optimization proposal, apply therefreshannotation to the custom resource. This moves the custom resource to thePendingProposalorProposalReadystate. You can then review the optimization proposal and approve it, if desired.
21.5. Tracking rebalances
You can track the progress of a partition rebalance using status information in the KafkaRebalance resource. Tracking an active partition rebalance supports planning of cluster operations, including worker node maintenance, scaling, and broker upgrades. Understanding key details, such as duration and the remaining data to transfer, helps with scheduling maintenance windows and assessing the impact of proceeding or canceling the rebalance.
When a rebalance is in progress, Streams for Apache Kafka stores progress information in a ConfigMap. The ConfigMap is referenced by the status.progress.rebalanceProgressConfigMap property of the KafkaRebalance resource and has the same name as the KafkaRebalance resource. The ConfigMap includes the following fields:
-
estimatedTimeToCompletionInMinutes: The estimated time it will take in minutes until the partition rebalance is complete. -
completedByteMovementPercentage: The percentage of data movement completed (in bytes), as a rounded down integer from 0-100. -
executorState.json: The “non-verbose” JSON payload from the /kafkacruisecontrol/state?substates=executor endpoint, providing details about the executor’s current status, including partition movement progress, concurrency limits, and total data to move.
The information is updated on each reconciliation of the KafkaRebalance resource.
Prerequisites
-
The
KafkaRebalancecustom resource is in aRebalancingstate.
Procedure
Accessing
estimatedTimeToCompletionInMinutesfield.:oc get configmaps <my_rebalance> -o jsonpath="{['data']['estimatedTimeToCompletionInMinutes']}"oc get configmaps <my_rebalance> -o jsonpath="{['data']['estimatedTimeToCompletionInMinutes']}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Accessing
completedByteMovementPercentagefield.:oc get configmaps <my_rebalance> -o jsonpath="{['data']['completedByteMovementPercentage']}"oc get configmaps <my_rebalance> -o jsonpath="{['data']['completedByteMovementPercentage']}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Accessing
executorState.jsonfield.oc get configmaps <my_rebalance> -o jsonpath="{['data']['executorState.json']}"oc get configmaps <my_rebalance> -o jsonpath="{['data']['executorState.json']}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow
21.6. Stopping rebalances
Once started, a cluster rebalance operation might take some time to complete and affect the overall performance of the Kafka cluster.
If you want to stop a cluster rebalance operation that is in progress, apply the stop annotation to the KafkaRebalance custom resource. This instructs Cruise Control to finish the current batch of partition reassignments and then stop the rebalance. When the rebalance has stopped, completed partition reassignments have already been applied; therefore, the state of the Kafka cluster is different when compared to prior to the start of the rebalance operation. If further rebalancing is required, you should generate a new optimization proposal.
The performance of the Kafka cluster in the intermediate (stopped) state might be worse than in the initial state.
Prerequisites
-
The status of the
KafkaRebalancecustom resource isRebalancing.
Procedure
Annotate the
KafkaRebalanceresource to stop the rebalance:oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="stop"
oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="stop"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the
KafkaRebalanceresource:oc describe kafkarebalance <kafka_rebalance_resource_name>
oc describe kafkarebalance <kafka_rebalance_resource_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Wait until the status changes to
Stopped.
21.7. Troubleshooting and refreshing rebalances
When creating a KafkaRebalance resource or interacting with Cruise Control, errors are reported in the resource status, along with guidance on how to fix them. In such cases, the resource transitions to the NotReady state.
To continue with a cluster rebalance operation, you must rectify any configuration issues in the KafkaRebalance resource or address any problems with the Cruise Control deployment.
Common issues include the following:
-
Misconfigured parameters in the
KafkaRebalanceresource. -
The
strimzi.io/clusterlabel for specifying the Kafka cluster in theKafkaRebalanceresource is missing. -
The Cruise Control server is not deployed as the
cruiseControlproperty in theKafkaresource is missing. - The Cruise Control server is not reachable.
After fixing any issues, you need to add the refresh annotation to the KafkaRebalance resource. During a “refresh”, a new optimization proposal is requested from the Cruise Control server.
Prerequisites
- You have approved an optimization proposal.
-
The status of the
KafkaRebalancecustom resource for the rebalance operation isNotReady.
Procedure
Get information about the error from the
KafkaRebalancestatus:oc describe kafkarebalance <kafka_rebalance_resource_name>
oc describe kafkarebalance <kafka_rebalance_resource_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Attempt to resolve the issue by annotating the
KafkaRebalanceresource to refresh the proposal:oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"
oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check the status of the
KafkaRebalanceresource:oc describe kafkarebalance <kafka_rebalance_resource_name>
oc describe kafkarebalance <kafka_rebalance_resource_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Wait until the status changes to
PendingProposal, or directly toProposalReady.
Change the replication factor of topics by updating the KafkaTopic resource managed by the Topic Operator. You can adjust the replication factor for specific purposes, such as:
- Setting a lower replication factor for non-critical topics or because of resource shortages
- Setting a higher replication factor to improve data durability and fault tolerance
The Topic Operator uses Cruise Control to make the necessary changes, so Cruise Control must be deployed with Streams for Apache Kafka.
The Topic Operator watches and periodically reconciles all managed and unpaused KafkaTopic resources to detect changes to .spec.replicas configuration by comparing the replication factor of the topic in Kafka. One or more replication factor updates are then sent to Cruise Control for processing in a single request.
Progress is reflected in the status of the KafkaTopic resource.
Prerequisites
- The Cluster Operator must be deployed.
-
The Topic Operator must be deployed to manage topics through the
KafkaTopiccustom resource. - Cruise Control is deployed with Kafka.
Procedure
Edit the
KafkaTopicresource to change thereplicasvalue.In this procedure, we change the
replicasvalue formy-topicfrom 1 to 3.Kafka topic replication factor configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the change to the
KafkaTopicconfiguration and wait for the Topic Operator to update the topic. Check the status of the
KafkaTopicresource to make sure the request was successful:oc get kafkatopics my-topic -o yaml
oc get kafkatopics my-topic -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Status for the replication factor change
Copy to Clipboard Copied! Toggle word wrap Toggle overflow An error message is shown in the status if the request fails before completion. The request is periodically retried if it enters a failed state.
Changing topic replication factor using the standalone Topic Operator
If you are using the standalone Topic Operator and aim to change the topic replication factor through configuration, you still need to use the Topic Operator in unidirectional mode alongside a Cruise Control deployment. You also need to include the following environment variables in the standalone Topic Operator deployment so that it can integrate with Cruise Control.
Example standalone Topic Operator deployment configuration
- 1
- Integrates Cruise Control with the Topic Operator.
- 2
- Flag to indicate whether rack awareness is enabled on the Kafka cluster. If so, replicas can be spread across different racks, data centers, or availability zones.
- 3
- Cruise Control hostname.
- 4
- Cruise control port.
- 5
- Enables TLS authentication and encryption for accessing the Kafka cluster.
- 6
- Enables basic authorization for accessing the Cruise Control API.
If you enable TLS authentication and authorization, mount the required certificates as follows:
-
Public certificates of the Cluster CA (certificate authority) in
/etc/tls-sidecar/cluster-ca-certs/ca.crt -
Basic authorization credentials (user name and password) in
/etc/eto-cc-api/topic-operator.apiAdminNameand/etc/eto-cc-api/topic-operator.apiAdminPassword
If you are using JBOD storage and have Cruise Control installed with Streams for Apache Kafka, you can reassign partitions between the JBOD disks used for storage on the same broker. This capability also allows you to remove JBOD disks without data loss.
To reassign partitions, configure a KafkaRebalance resource in remove-disks mode and specify a list of broker IDs with corresponding volume IDs for partition reassignment. Cruise Control generates an optimization proposal based on the configuration and reassigns the partitions when approved manually or automatically.
Use the Kafka kafka-log-dirs.sh tool to check information about Kafka topic partitions and their location on brokers before and after moving them. Use an interactive pod to avoid running the tool within the broker container and causing any disruptions.
Prerequisites
- The Cluster Operator must be deployed.
- Cruise Control is deployed with Kafka.
- Kafka operates in KRaft mode and brokers use JBOD storage.
- More than one JBOD disk must be configured on the broker. For more information on configuring Kafka storage, see Section 10.4, “Configuring Kafka storage”.
In this procedure, we use a Kafka cluster named my-cluster, which is deployed to the my-project namespace with node pools and Cruise Control enabled.
Example Kafka cluster configuration
A node pool named pool-a is configured with three broker replicas that use three JBOD storage volumes. In the procedure, we show how partitions are reassigned from volume 1 and 2 to volume 0.
Example node pool configuration with JBOD storage
Procedure
Run a new interactive pod container using the Kafka image to connect to a running Kafka broker.
oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 helper-pod -- /bin/sh -c "sleep 3600"
oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 helper-pod -- /bin/sh -c "sleep 3600"Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this procedure, we use a pod named
helper-pod.(Optional) Check the partition replica data on broker 0 by opening a terminal inside the interactive pod and running the Kafka
kafka-log-dirs.shtool:oc exec -n myproject -ti my-cluster-pool-a-0 bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 0,1,2 --topic-list my-topic
oc exec -n myproject -ti my-cluster-pool-a-0 bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 0,1,2 --topic-list my-topicCopy to Clipboard Copied! Toggle word wrap Toggle overflow my-cluster-pool-a-0is the pod name for broker 0. The tool returns topic information for each log directory. In this example, we are restricting the information tomy-topicto show the steps against a single topic. The JBOD volumes used for log directories are mounted at/var/lib/kafka/data-<volume_id>/kafka-log<pod_id>.Example output data for each log directory
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
KafkaRebalanceresource inremove-disksmode, listing the brokers and volume IDs to reassign partitions from. Without specific configuration, the default rebalance goals are used.Example Cruise Control configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example,
my-rebalancereassigns partitions from volumes with IDs 1 and 2 on broker 0.(Optional) To approve the optimization proposal automatically, set the
strimzi.io/rebalance-auto-approvalannotation totrue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Apply the
KafkaRebalanceconfiguration. If manually approving, wait for the status of the proposal to move to
ProposalReadybefore approving the changes.Check the summary of the changes in the
KafkaRebalancestatus:oc get kafkarebalance my-rebalance -n my-project -o yaml
oc get kafkarebalance my-rebalance -n my-project -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example summary of changes
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe summary only shows the changes after optimization, not the load before optimization.
Annotate the
KafkaRebalanceresource to approve the changes:oc annotate kafkarebalance my-rebalance strimzi.io/rebalance="approve"
oc annotate kafkarebalance my-rebalance strimzi.io/rebalance="approve"Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Wait for the status of the proposal to change to
Ready. Use the Kafka
kafka-log-dirs.shtool again to verify data movement.In this example, the log directories for volumes 1 and 2 no longer have partitions assigned to them and volume 0 holds 6 partitions for
my-topic, indicating that the partitions have been successfully reassigned.Example output data following reassignment of partitions
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To prevent empty volumes from being used in future rebalances or topic allocations, update the configuration and remove the associated persistent volume claims (PVCs).
Update the node pool configuration to exclude the volumes.
WarningBefore making changes, verify that all partitions have been successfully moved using
kafka-log-dirs.sh. Removing volumes prematurely can cause data loss.In this example, volumes 1 and 2 are removed, and only volume 0 is retained:
Updated node pool configuration with single volume JBOD storage
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the unused PVCs.
PVCs are named using the format
data-<id>-<kafka_cluster_name>-kafka-<pod_id>. You can list them using:oc get pvc -n my-project
oc get pvc -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Then delete the unused PVCs:
oc delete pvc data-<id>-<kafka_cluster_name>-kafka-<pod_id> -n my-project
oc delete pvc data-<id>-<kafka_cluster_name>-kafka-<pod_id> -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteDeleting a PVC removes the underlying storage unless
deleteClaim: falseis set in the volume configuration.(Optional) Delete the helper pod used earlier:
oc delete pod helper-pod -n my-project
oc delete pod helper-pod -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 24. Using the partition reassignment tool
You can use the kafka-reassign-partitions.sh tool for the following:
- Adding or removing brokers
- Reassigning partitions across brokers
- Changing the replication factor of topics
However, while kafka-reassign-partitions.sh supports these operations, it is generally easier with Cruise Control. Cruise Control can move topics from one broker to another without any downtime, and it is the most efficient way to reassign partitions.
To use the kafka-reassign-partitions.sh tool, run it as a separate interactive pod rather than within the broker container. Running the Kafka bin/ scripts within the broker container may cause a JVM to start with the same settings as the Kafka broker, which can potentially cause disruptions. By running the kafka-reassign-partitions.sh tool in a separate pod, you can avoid this issue. Running a pod with the -ti option creates an interactive pod with a terminal for running shell commands inside the pod.
Running an interactive pod with a terminal
oc run helper-pod -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bash
oc run helper-pod -ti --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 --rm=true --restart=Never -- bash24.1. Partition reassignment tool overview
The partition reassignment tool provides the following capabilities for managing Kafka partitions and brokers:
- Redistributing partition replicas
- Scale your cluster up and down by adding or removing brokers, and move Kafka partitions from heavily loaded brokers to under-utilized brokers. To do this, you must create a partition reassignment plan that identifies which topics and partitions to move and where to move them. Cruise Control is recommended for this type of operation as it automates the cluster rebalancing process.
- Scaling topic replication factor up and down
- Increase or decrease the replication factor of your Kafka topics. To do this, you must create a partition reassignment plan that identifies the existing replication assignment across partitions and an updated assignment with the replication factor changes.
- Changing the preferred leader
Change the preferred leader of a Kafka partition. In Kafka, the partition leader is the only partition that accepts writes from message producers, and therefore has the most complete log across all replicas.
Changing the preferred leader can be useful if you want to redistribute load across the brokers in the cluster. If the preferred leader is unavailable, another in-sync replica is automatically elected as leader, or the partition goes offline if there are no in-sync replicas. A background thread moves the leader role to the preferred replica when it is in sync. Therefore, changing the preferred replicas only makes sense in the context of a cluster rebalancing.
To do this, you must create a partition reassignment plan that specifies the new preferred leader for each partition by changing the order of replicas. In Kafka’s leader election process, the preferred leader is prioritized by the order of replicas. The first broker in the order of replicas is designated as the preferred leader. This designation is important for load balancing by distributing partition leaders across the Kafka cluster. However, this alone might not be sufficient for optimal load balancing, as some partitions may have higher usage than others. Cruise Control can help address this by providing more comprehensive cluster rebalancing.
- Changing the log directories to use a specific JBOD volume
- Change the log directories of your Kafka brokers to use a specific JBOD volume. This can be useful if you want to move your Kafka data to a different disk or storage device. To do this, you must create a partition reassignment plan that specifies the new log directory for each topic.
24.1.1. Generating a partition reassignment plan
The partition reassignment tool (kafka-reassign-partitions.sh) works by generating a partition assignment plan that specifies which partitions should be moved from their current broker to a new broker.
If you are satisfied with the plan, you can execute it. The tool then does the following:
- Migrates the partition data to the new broker
- Updates the metadata on the Kafka brokers to reflect the new partition assignments
- Triggers a rolling restart of the Kafka brokers to ensure that the new assignments take effect
The partition reassignment tool has three different modes:
--generate- Takes a set of topics and brokers and generates a reassignment JSON file which will result in the partitions of those topics being assigned to those brokers. Because this operates on whole topics, it cannot be used when you only want to reassign some partitions of some topics.
--execute- Takes a reassignment JSON file and applies it to the partitions and brokers in the cluster. Brokers that gain partitions as a result become followers of the partition leader. For a given partition, once the new broker has caught up and joined the ISR (in-sync replicas) the old broker will stop being a follower and will delete its replica.
--verify-
Using the same reassignment JSON file as the
--executestep,--verifychecks whether all the partitions in the file have been moved to their intended brokers. If the reassignment is complete,--verifyalso removes any traffic throttles (--throttle) that are in effect. Unless removed, throttles will continue to affect the cluster even after the reassignment has finished.
It is only possible to have one reassignment running in a cluster at any given time, and it is not possible to cancel a running reassignment. If you must cancel a reassignment, wait for it to complete and then perform another reassignment to revert the effects of the first reassignment. The kafka-reassign-partitions.sh will print the reassignment JSON for this reversion as part of its output. Very large reassignments should be broken down into a number of smaller reassignments in case there is a need to stop in-progress reassignment.
The kafka-reassign-partitions.sh tool uses a reassignment JSON file that specifies the topics to reassign. You can generate a reassignment JSON file or create a file manually if you want to move specific partitions.
A basic reassignment JSON file has the structure presented in the following example, which describes three partitions belonging to two Kafka topics. Each partition is reassigned to a new set of replicas, which are identified by their broker IDs. The version, topic, partition, and replicas properties are all required.
Example partition reassignment JSON file structure
- 1
- The version of the reassignment JSON file format. Currently, only version 1 is supported, so this should always be 1.
- 2
- An array that specifies the partitions to be reassigned.
- 3
- The name of the Kafka topic that the partition belongs to.
- 4
- The ID of the partition being reassigned.
- 5
- An ordered array of the IDs of the brokers that should be assigned as replicas for this partition. The first broker in the list is the leader replica.
Partitions not included in the JSON are not changed.
If you specify only topics using a topics array, the partition reassignment tool reassigns all the partitions belonging to the specified topics.
Example reassignment JSON file structure for reassigning all partitions for a topic
24.1.3. Reassigning partitions between JBOD volumes
When using JBOD storage in your Kafka cluster, you can reassign the partitions between specific volumes and their log directories (each volume has a single log directory).
To reassign a partition to a specific volume, add log_dirs values for each partition in the reassignment JSON file. Each log_dirs array contains the same number of entries as the replicas array, since each replica should be assigned to a specific log directory. The log_dirs array contains either an absolute path to a log directory or the special value any. The any value indicates that Kafka can choose any available log directory for that replica, which can be useful when reassigning partitions between JBOD volumes.
Example reassignment JSON file structure with log directories
24.1.4. Throttling partition reassignment
Partition reassignment can be a slow process because it involves transferring large amounts of data between brokers. To avoid a detrimental impact on clients, you can throttle the reassignment process. Use the --throttle parameter with the kafka-reassign-partitions.sh tool to throttle a reassignment. You specify a maximum threshold in bytes per second for the movement of partitions between brokers. For example, --throttle 5000000 sets a maximum threshold for moving partitions of 50 MBps.
Throttling might cause the reassignment to take longer to complete.
- If the throttle is too low, the newly assigned brokers will not be able to keep up with records being published and the reassignment will never complete.
- If the throttle is too high, clients will be impacted.
For example, for producers, this could manifest as higher than normal latency waiting for acknowledgment. For consumers, this could manifest as a drop in throughput caused by higher latency between polls.
Generate a reassignment JSON file with the kafka-reassign-partitions.sh tool to reassign partitions after scaling a Kafka cluster. Adding or removing brokers does not automatically redistribute the existing partitions. To balance the partition distribution and take full advantage of the new brokers, you can reassign the partitions using the kafka-reassign-partitions.sh tool.
You run the tool from an interactive pod container connected to the Kafka cluster.
The following procedure describes a secure reassignment process that uses mTLS. You’ll need a Kafka cluster that uses TLS encryption and mTLS authentication.
You’ll need the following to establish a connection:
- The cluster CA certificate and password generated by the Cluster Operator when the Kafka cluster is created
- The user CA certificate and password generated by the User Operator when a user is created for client access to the Kafka cluster
In this procedure, the CA certificates and corresponding passwords are extracted from the cluster and user secrets that contain them in PKCS #12 (.p12 and .password) format. The passwords allow access to the .p12 stores that contain the certificates. You use the .p12 stores to specify a truststore and keystore to authenticate connection to the Kafka cluster.
Prerequisites
- You have a running Cluster Operator.
You have a running Kafka cluster based on a
Kafkaresource configured with internal TLS encryption and mTLS authentication.Kafka configuration with TLS encryption and mTLS authentication
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The running Kafka cluster contains a set of topics and partitions to reassign.
Example topic configuration for
my-topicCopy to Clipboard Copied! Toggle word wrap Toggle overflow You have a
KafkaUserconfigured with ACL rules that specify permission to produce and consume topics from the Kafka brokers.Example Kafka user configuration with ACL rules to allow operations on
my-topicandmy-clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Procedure
Extract the cluster CA certificate and password from the
<cluster_name>-cluster-ca-certsecret of the Kafka cluster.oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.passwordoc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.passwordCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <cluster_name> with the name of the Kafka cluster. When you deploy Kafka using the
Kafkaresource, a secret with the cluster CA certificate is created with the Kafka cluster name (<cluster_name>-cluster-ca-cert). For example,my-cluster-cluster-ca-cert.Run a new interactive pod container using the Kafka image to connect to a running Kafka broker.
oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 <interactive_pod_name> -- /bin/sh -c "sleep 3600"
oc run --restart=Never --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 <interactive_pod_name> -- /bin/sh -c "sleep 3600"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <interactive_pod_name> with the name of the pod.
Copy the cluster CA certificate to the interactive pod container.
oc cp ca.p12 <interactive_pod_name>:/tmp
oc cp ca.p12 <interactive_pod_name>:/tmpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the user CA certificate and password from the secret of the Kafka user that has permission to access the Kafka brokers.
oc get secret <kafka_user> -o jsonpath='{.data.user\.p12}' | base64 -d > user.p12oc get secret <kafka_user> -o jsonpath='{.data.user\.p12}' | base64 -d > user.p12Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secret <kafka_user> -o jsonpath='{.data.user\.password}' | base64 -d > user.passwordoc get secret <kafka_user> -o jsonpath='{.data.user\.password}' | base64 -d > user.passwordCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <kafka_user> with the name of the Kafka user. When you create a Kafka user using the
KafkaUserresource, a secret with the user CA certificate is created with the Kafka user name. For example,my-user.Copy the user CA certificate to the interactive pod container.
oc cp user.p12 <interactive_pod_name>:/tmp
oc cp user.p12 <interactive_pod_name>:/tmpCopy to Clipboard Copied! Toggle word wrap Toggle overflow The CA certificates allow the interactive pod container to connect to the Kafka broker using TLS.
Create a
config.propertiesfile to specify the truststore and keystore used to authenticate connection to the Kafka cluster.Use the certificates and passwords you extracted in the previous steps.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The bootstrap server address to connect to the Kafka cluster. Use your own Kafka cluster name to replace <kafka_cluster_name>.
- 2
- The security protocol option when using TLS for encryption.
- 3
- The truststore location contains the public key certificate (
ca.p12) for the Kafka cluster. - 4
- The password (
ca.password) for accessing the truststore. - 5
- The keystore location contains the public key certificate (
user.p12) for the Kafka user. - 6
- The password (
user.password) for accessing the keystore.
Copy the
config.propertiesfile to the interactive pod container.oc cp config.properties <interactive_pod_name>:/tmp/config.properties
oc cp config.properties <interactive_pod_name>:/tmp/config.propertiesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Prepare a JSON file named
topics.jsonthat specifies the topics to move.Specify topic names as a comma-separated list.
Example JSON file to reassign all the partitions of
my-topicCopy to Clipboard Copied! Toggle word wrap Toggle overflow You can also use this file to change the replication factor of a topic.
Copy the
topics.jsonfile to the interactive pod container.oc cp topics.json <interactive_pod_name>:/tmp/topics.json
oc cp topics.json <interactive_pod_name>:/tmp/topics.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Start a shell process in the interactive pod container.
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bashCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <namespace> with the OpenShift namespace where the pod is running.
Use the
kafka-reassign-partitions.shcommand to generate the reassignment JSON.Example command to move the partitions of
my-topicto specified brokersbin/kafka-reassign-partitions.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --topics-to-move-json-file /tmp/topics.json \ --broker-list 0,1,2,3,4 \ --generate
bin/kafka-reassign-partitions.sh --bootstrap-server my-cluster-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --topics-to-move-json-file /tmp/topics.json \ --broker-list 0,1,2,3,4 \ --generateCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Use a reassignment file generated by the kafka-reassign-partitions.sh tool to reassign partitions after increasing the number of brokers in a Kafka cluster. The reassignment file should describe how partitions are reassigned to brokers in the enlarged Kafka cluster. You apply the reassignment specified in the file to the brokers and then verify the new partition assignments.
This procedure describes a secure scaling process that uses TLS. You’ll need a Kafka cluster that uses TLS encryption and mTLS authentication.
The kafka-reassign-partitions.sh tool can be used to reassign partitions within a Kafka cluster, regardless of whether you are managing all nodes through the cluster or using the node pools to manage groups of nodes within the cluster.
Though you can use the kafka-reassign-partitions.sh tool for this operation, Cruise Control is recommended for automated partition reassignments and cluster rebalancing. Cruise Control can move topics from one broker to another without any downtime, and it is the most efficient way to reassign partitions.
Prerequisites
-
You have a running Kafka cluster based on a
Kafkaresource configured with internal TLS encryption and mTLS authentication. -
You have generated a reassignment JSON file named
reassignment.json. - You are running an interactive pod container that is connected to the running Kafka broker.
-
You are connected as a
KafkaUserconfigured with ACL rules that specify permission to manage the Kafka cluster and its topics.
Procedure
-
Add as many new brokers as you need by increasing the
Kafka.spec.kafka.replicasconfiguration option. - Verify that the new broker pods have started.
-
If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named
reassignment.json. Copy the
reassignment.jsonfile to the interactive pod container.oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json
oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <interactive_pod_name> with the name of the pod.
Start a shell process in the interactive pod container.
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bashCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <namespace> with the OpenShift namespace where the pod is running.
Run the partition reassignment using the
kafka-reassign-partitions.shscript from the interactive pod container.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --execute
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --executeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <cluster_name> with the name of your Kafka cluster. For example,
my-cluster-kafka-bootstrap:9093If you are going to throttle replication, you can also pass the
--throttleoption with an inter-broker throttled rate in bytes per second. For example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow This command will print out two reassignment JSON objects. The first records the current assignment for the partitions being moved. You should save this to a local file (not a file in the pod) in case you need to revert the reassignment later on. The second JSON object is the target reassignment you have passed in your reassignment JSON file.
If you need to change the throttle during reassignment, you can use the same command with a different throttled rate. For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the reassignment has completed using the
kafka-reassign-partitions.shcommand line tool from any of the broker pods. This is the same command as the previous step, but with the--verifyoption instead of the--executeoption.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verify
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verifyCopy to Clipboard Copied! Toggle word wrap Toggle overflow The reassignment has finished when the
--verifycommand reports that each of the partitions being moved has completed successfully. This final--verifywill also have the effect of removing any reassignment throttles.- You can now delete the revert file if you saved the JSON for reverting the assignment to their original brokers.
Use a reassignment file generated by the kafka-reassign-partitions.sh tool to reassign partitions before decreasing the number of brokers in a Kafka cluster. The reassignment file must describe how partitions are reassigned to the remaining brokers in the Kafka cluster. You apply the reassignment specified in the file to the brokers and then verify the new partition assignments. Brokers in the highest numbered pods are removed first.
This procedure describes a secure scaling process that uses TLS. You’ll need a Kafka cluster that uses TLS encryption and mTLS authentication.
The kafka-reassign-partitions.sh tool can be used to reassign partitions within a Kafka cluster, regardless of whether you are managing all nodes through the cluster or using the node pools to manage groups of nodes within the cluster.
Though you can use the kafka-reassign-partitions.sh tool for this operation, Cruise Control is recommended for automated partition reassignments and cluster rebalancing. Cruise Control can move topics from one broker to another without any downtime, and it is the most efficient way to reassign partitions.
Prerequisites
-
You have a running Kafka cluster based on a
Kafkaresource configured with internal TLS encryption and mTLS authentication. -
You have generated a reassignment JSON file named
reassignment.json. - You are running an interactive pod container that is connected to the running Kafka broker.
-
You are connected as a
KafkaUserconfigured with ACL rules that specify permission to manage the Kafka cluster and its topics.
Procedure
-
If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named
reassignment.json. Copy the
reassignment.jsonfile to the interactive pod container.oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json
oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <interactive_pod_name> with the name of the pod.
Start a shell process in the interactive pod container.
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bashCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <namespace> with the OpenShift namespace where the pod is running.
Run the partition reassignment using the
kafka-reassign-partitions.shscript from the interactive pod container.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --execute
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --executeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <cluster_name> with the name of your Kafka cluster. For example,
my-cluster-kafka-bootstrap:9093If you are going to throttle replication, you can also pass the
--throttleoption with an inter-broker throttled rate in bytes per second. For example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow This command will print out two reassignment JSON objects. The first records the current assignment for the partitions being moved. You should save this to a local file (not a file in the pod) in case you need to revert the reassignment later on. The second JSON object is the target reassignment you have passed in your reassignment JSON file.
If you need to change the throttle during reassignment, you can use the same command with a different throttled rate. For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the reassignment has completed using the
kafka-reassign-partitions.shcommand line tool from any of the broker pods. This is the same command as the previous step, but with the--verifyoption instead of the--executeoption.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verify
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verifyCopy to Clipboard Copied! Toggle word wrap Toggle overflow The reassignment has finished when the
--verifycommand reports that each of the partitions being moved has completed successfully. This final--verifywill also have the effect of removing any reassignment throttles.- You can now delete the revert file if you saved the JSON for reverting the assignment to their original brokers.
When all the partition reassignments have finished, the brokers being removed should not have responsibility for any of the partitions in the cluster. You can verify this by checking that the broker’s data log directory does not contain any live partition logs. If the log directory on the broker contains a directory that does not match the extended regular expression
[a-zA-Z0-9.-]+\.[a-z0-9]+-delete$, the broker still has live partitions and should not be stopped.You can check this by executing the command:
oc exec my-cluster-kafka-0 -c kafka -it -- \ /bin/bash -c \ "ls -l /var/lib/kafka/kafka-log_<n>_ | grep -E '^d' | grep -vE '[a-zA-Z0-9.-]+\.[a-z0-9]+-delete$'"
oc exec my-cluster-kafka-0 -c kafka -it -- \ /bin/bash -c \ "ls -l /var/lib/kafka/kafka-log_<n>_ | grep -E '^d' | grep -vE '[a-zA-Z0-9.-]+\.[a-z0-9]+-delete$'"Copy to Clipboard Copied! Toggle word wrap Toggle overflow where n is the number of the pods being deleted.
If the above command prints any output then the broker still has live partitions. In this case, either the reassignment has not finished or the reassignment JSON file was incorrect.
-
When you have confirmed that the broker has no live partitions, you can edit the
Kafka.spec.kafka.replicasproperty of yourKafkaresource to reduce the number of brokers.
Use a reassignment file generated by the kafka-reassign-partitions.sh tool to change the replication factor of topics.
This procedure describes a secure process that uses TLS. You’ll need a Kafka cluster that uses TLS encryption and mTLS authentication.
Though you can use the kafka-reassign-partitions.sh tool for this operation, the simplest approach is to enable Cruise Control and make the change through the KafkaTopic configuration. For more information, see Chapter 22, Using Cruise Control to modify topic replication factor.
Prerequisites
-
You have a running Kafka cluster based on a
Kafkaresource configured with internal TLS encryption and mTLS authentication. - You are running an interactive pod container that is connected to the running Kafka broker.
-
You have generated a reassignment JSON file named
reassignment.json. -
You are connected as a
KafkaUserconfigured with ACL rules that specify permission to manage the Kafka cluster and its topics.
See Generating reassignment JSON files.
In this procedure, a topic called my-topic has 4 replicas and we want to reduce it to 3. A JSON file named topics.json specifies the topic, and was used to generate the reassignment.json file.
Example JSON file specifies my-topic
Procedure
If you haven’t done so, run an interactive pod container to generate a reassignment JSON file named
reassignment.json.Example reassignment JSON file showing the current and proposed replica assignment
Current partition replica assignment {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[3,4,2,0],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[0,2,3,1],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[1,3,0,4],"log_dirs":["any","any","any","any"]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3,4],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4,0],"log_dirs":["any","any","any","any"]}]}Current partition replica assignment {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[3,4,2,0],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[0,2,3,1],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[1,3,0,4],"log_dirs":["any","any","any","any"]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3,4],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4,0],"log_dirs":["any","any","any","any"]}]}Copy to Clipboard Copied! Toggle word wrap Toggle overflow Save a copy of this file locally in case you need to revert the changes later on.
Edit the
reassignment.jsonto remove a replica from each partition.For example use the
jqcommand line JSON parser tool to remove the last replica in the list for each partition of the topic:Removing the last topic replica for each partition
jq '.partitions[].replicas |= del(.[-1])' reassignment.json > reassignment.json
jq '.partitions[].replicas |= del(.[-1])' reassignment.json > reassignment.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example reassignment file showing the updated replicas
{"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4],"log_dirs":["any","any","any","any"]}]}{"version":1,"partitions":[{"topic":"my-topic","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any","any"]},{"topic":"my-topic","partition":2,"replicas":[2,3,4],"log_dirs":["any","any","any","any"]}]}Copy to Clipboard Copied! Toggle word wrap Toggle overflow Copy the
reassignment.jsonfile to the interactive pod container.oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.json
oc cp reassignment.json <interactive_pod_name>:/tmp/reassignment.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <interactive_pod_name> with the name of the pod.
Start a shell process in the interactive pod container.
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bash
oc exec -n <namespace> -ti <interactive_pod_name> /bin/bashCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <namespace> with the OpenShift namespace where the pod is running.
Make the topic replica change using the
kafka-reassign-partitions.shscript from the interactive pod container.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --execute
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --executeCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteRemoving replicas from a broker does not require any inter-broker data movement, so there is no need to throttle replication. If you are adding replicas, then you may want to change the throttle rate.
Verify that the change to the topic replicas has completed using the
kafka-reassign-partitions.shcommand line tool from any of the broker pods. This is the same command as the previous step, but with the--verifyoption instead of the--executeoption.bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verify
bin/kafka-reassign-partitions.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --reassignment-json-file /tmp/reassignment.json \ --verifyCopy to Clipboard Copied! Toggle word wrap Toggle overflow The reassignment has finished when the
--verifycommand reports that each of the partitions being moved has completed successfully. This final--verifywill also have the effect of removing any reassignment throttles.Run the
bin/kafka-topics.shcommand with the--describeoption to see the results of the change to the topics.bin/kafka-topics.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --describe
bin/kafka-topics.sh --bootstrap-server <cluster_name>-kafka-bootstrap:9093 \ --command-config /tmp/config.properties \ --describeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Results of reducing the number of replicas for a topic
my-topic Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 my-topic Partition: 1 Leader: 2 Replicas: 1,2,3 Isr: 1,2,3 my-topic Partition: 2 Leader: 3 Replicas: 2,3,4 Isr: 2,3,4
my-topic Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 my-topic Partition: 1 Leader: 2 Replicas: 1,2,3 Isr: 1,2,3 my-topic Partition: 2 Leader: 3 Replicas: 2,3,4 Isr: 2,3,4Copy to Clipboard Copied! Toggle word wrap Toggle overflow Finally, edit the
KafkaTopiccustom resource to change.spec.replicasto 3, and then wait the reconciliation.Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 25. Introducing metrics
Metrics provide insight into the health, performance, and capacity of your Kafka deployment. By observing these metrics, you can detect potential issues early, optimize resource usage, and improve reliability.
Metrics from Streams for Apache Kafka components can be collected using Prometheus and visualized in Grafana. Prometheus collects metrics from running pods in your cluster, while Grafana provides an intuitive interface for exploring and visualizing these metrics.
The Streams for Apache Kafka operators expose Prometheus metrics by default, including reconciliation counts, durations, JVM data, and resource processing statistics. Other components and features require configuration to expose their metrics.
In addition, you can use kube-state-metrics (KSM) to monitor the state and readiness of Streams for Apache Kafka custom resources, and to raise alerts for warning conditions.
Streams for Apache Kafka provides example Prometheus rules, Grafana dashboards, and kube-state-metrics configuration files that you can adapt to your environment. You can also track message flow across components by setting up distributed tracing.
Streams for Apache Kafka provides example installation files for Prometheus, Grafana, and kube-state-metrics to help get you started. For further information, refer to the resources of those projects.
25.1. Example metrics files
You can find example Grafana dashboards and other metrics configuration files in the example configuration files provided by Streams for Apache Kafka.
Example metrics files provided with Streams for Apache Kafka
- 1
- Grafana dashboards for components using the JMX Exporter.
- 2
- Installation file for the Grafana image.
- 3
- Kube-state-metrics deployment and configuration files for custom resource monitoring.
- 4
- Additional configuration to scrape metrics for CPU, memory and disk volume usage, which comes directly from the OpenShift cAdvisor agent and kubelet on the nodes.
- 5
- Hook definitions for sending notifications through Alertmanager.
- 6
- Resources for deploying and configuring Alertmanager.
- 7
- PodMonitor definitions translated by the Prometheus Operator into jobs for the Prometheus server to be able to scrape metrics data directly from pods.
- 8
- Alerting rules examples for use with Prometheus Alertmanager (deployed with Prometheus).
- 9
- Installation resource file for the Prometheus image.
- 10
- Grafana dashboards for components using the Streams for Apache Kafka Metrics Reporter.
- 11
KafkaBridgeresource for deploying Kafka Bridge with Streams for Apache Kafka Metrics Reporter.- 12
KafkaConnectresource for deploying Kafka Connect with Streams for Apache Kafka Metrics Reporter.- 13
Kafkaresource for deploying Kafka with Streams for Apache Kafka Metrics Reporter.- 14
KafkaMirrorMaker2resource for deploying MirrorMaker 2 with Streams for Apache Kafka Metrics Reporter.- 15
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka Bridge.
- 16
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka Connect.
- 17
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Cruise Control.
- 18
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka.
- 19
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for MirrorMaker 2.
- 20
- Metrics configuration that defines Prometheus JMX Exporter relabeling rules for OAuth 2.0.
The Prometheus JMX Exporter collects Java Management Extensions (JMX) from Kafka components and converts them into Prometheus metrics. You do not require jmxOptions configuration in the custom resource of the component for the Prometheus JMX Exporter to function. jmxOptions is only required if you need direct access to JMX from Kafka components.
Streams for Apache Kafka uses the Streams for Apache Kafka Metrics Reporter or the Prometheus JMX Exporter to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server.
Streams for Apache Kafka provides example custom resource configuration YAML files with relabeling rules. When deploying Prometheus metrics configuration, you can deploy the example custom resource or copy the metrics configuration to your own custom resource definition.
Grafana dashboards with Prometheus JMX Exporter are dependent on relabeling rules, which are defined for Streams for Apache Kafka components in the custom resource configuration.
A label is a name-value pair. Relabeling is the process of writing a label dynamically. For example, the value of a label may be derived from the name of a Kafka server and client ID.
| Component | Custom resource | Example YAML file |
|---|---|---|
| Kafka nodes |
|
|
| Kafka Connect |
|
|
| Kafka MirrorMaker 2 |
|
|
| Kafka Bridge |
|
|
| Cruise Control |
|
|
25.1.2. Example Metrics Reporter metrics configuration
| Component | Custom resource | Example YAML file |
|---|---|---|
| Kafka nodes |
|
|
| Kafka Connect |
|
|
| Kafka MirrorMaker2 |
|
|
| Kafka Bridge |
|
|
25.1.3. Example Prometheus rules for alert notifications
Example Prometheus rules for alert notifications are provided with the example metrics configuration files provided by Streams for Apache Kafka. The rules are specified in the example prometheus-rules directory for use in a Prometheus deployment.
The prometheus-rules directory contains example rules for the following components:
- Kafka
- Entity Operator
- Kafka Connect
- Kafka Bridge
- MirrorMaker2
- Kafka Exporter
A description of each of the example rules is provided in the file.
Alerting rules provide notifications about specific conditions observed in metrics. Rules are declared on the Prometheus server, but Prometheus Alertmanager is responsible for alert notifications.
Prometheus alerting rules describe conditions using PromQL expressions that are continuously evaluated.
When an alert expression becomes true, the condition is met and the Prometheus server sends alert data to the Alertmanager. Alertmanager then sends out a notification using the communication method configured for its deployment.
General points about the alerting rule definitions:
-
A
forproperty is used with the rules to determine the period of time a condition must persist before an alert is triggered. -
The availability of the
KafkaRunningOutOfSpacemetric and alert is dependent on the OpenShift configuration and storage implementation used. Storage implementations for certain platforms may not be able to supply the information on available space required for the metric to provide an alert.
Alertmanager can be configured to use email, chat messages or other notification methods. Adapt the default configuration of the example rules according to your specific needs.
25.1.4. Example Grafana dashboards
If you deploy Prometheus to provide metrics, you can use the example Grafana dashboards provided with Streams for Apache Kafka to monitor Streams for Apache Kafka components.
Example dashboards are provided in the examples/metrics/grafana-dashboards directory as JSON files.
All dashboards provide JVM metrics, as well as metrics specific to the component. For example, the Grafana dashboard for Streams for Apache Kafka operators provides information on the number of reconciliations or custom resources they are processing.
The example dashboards don’t show all the metrics supported by Kafka. The dashboards are populated with a representative set of metrics for monitoring.
| Component | Example JSON file |
|---|---|
| Streams for Apache Kafka operators |
|
| Kafka |
|
| Kafka KRaft controllers |
|
| Kafka Connect |
|
| Kafka MirrorMaker 2 |
|
| Kafka Bridge |
|
| Cruise Control |
|
| Kafka Exporter |
|
When metrics are not available to the Kafka Exporter, because there is no traffic in the cluster yet, the Kafka Exporter Grafana dashboard will show N/A for numeric fields and No data to show for graphs.
25.2. Using Prometheus with Streams for Apache Kafka
Use Prometheus to provide monitoring data for the example Grafana dashboards provided with Streams for Apache Kafka.
To expose metrics in Prometheus format, you add configuration to a custom resource. You must also make sure that the metrics are scraped by your monitoring stack. Prometheus and Prometheus Alertmanager are used in the examples provided by Streams for Apache Kafka, but you can use also other compatible tools.
You can expose metrics using one of the following approaches:
After enabling metrics, you can integrate with Prometheus:
Streams for Apache Kafka provides an example Grafana dashboards to display visualizations of metrics. The exposed metrics provide the monitoring data when you enable the Grafana dashboard.
25.2.1. Enabling Prometheus JMX Exporter
To enable and expose metrics in Streams for Apache Kafka for Prometheus, configure the appropriate properties in the custom resources for the components you want to monitor.
Use metricsConfig to expose metrics for these components:
- Kafka
- Kafka Connect
- Kafka MirrorMaker 2
- Kafka Bridge
- Cruise Control
This enables the Prometheus JMX Exporter, which exposes metrics on port 9404 through an HTTP endpoint. Prometheus scrapes this endpoint to collect Kafka metrics.
Set enableMetrics to true to expose metrics for the following:
OAuth 2.0
-
Configure in the
Kafkaresource foroauthorkeycloakcluster authorization, oroauthlistener authentication. -
Configure in the
KafkaBridge,KafkaConnect, orKafkaMirrorMaker2resources foroauthauthentication.
-
Configure in the
You can create your own Prometheus configuration or use the example custom resource files provided with Streams for Apache Kafka:
-
kafka-metrics.yaml -
kafka-connect-metrics.yaml -
kafka-mirror-maker-2-metrics.yaml -
kafka-bridge-metrics.yaml -
kafka-cruise-control-metrics.yaml -
oauth-metrics.yaml
These files include relabeling rules and example metrics configuration, and are a good starting point for trying Prometheus with Streams for Apache Kafka.
This procedure shows how to deploy example Prometheus metrics configuration to the Kafka resource. The same steps apply when deploying the example files for other resources.
Procedure
Deploy the example custom resource with the Prometheus configuration.
For example, for each
Kafkaresource you can apply thekafka-metrics.yamlfile.Deploying the example configuration
oc apply -f kafka-metrics.yaml
oc apply -f kafka-metrics.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Alternatively, copy the example configuration in
kafka-metrics.yamlto your ownKafkaresource.Copying the example configuration
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Copy the
metricsConfigproperty and theConfigMapit references to yourKafkaresource.Example metrics configuration for Kafka
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Enabling metrics for OAuth 2.0
To expose metrics for OAuth 2.0, set the enableMetrics property to true in the appropriate custom resource.
In the Kafka resource for:
-
Cluster authorization (
oauthorkeycloak) -
Listener authentication (
oauthonly)
-
Cluster authorization (
-
In the
KafkaBridge,KafkaConnect, orKafkaMirrorMaker2resources foroauthauthentication
In the following example, metrics are enabled for OAuth 2.0 listener authentication and OAuth 2.0 (keycloak) cluster authorization.
Example configuration with OAuth 2.0 metrics enabled
To use OAuth 2.0 metrics with Prometheus, copy the ConfigMap configuration from the oauth-metrics.yaml file to the same Kafka resource configuration file where you enabled metrics for OAuth 2.0 and then apply the configuration.
You can also enable metrics for the type: opa authorization option in the same way as for OAuth 2.0 authorization. However, type: opa is deprecated and will be removed in a future release. To continue using the Open Policy Agent authorizer, use the type: custom authorization configuration.
This feature is a technology preview and not intended for a production environment. For more information see the release notes.
To enable and expose metrics in Streams for Apache Kafka for Prometheus, configure the appropriate properties in the custom resources for the components you want to monitor.
Use metricsConfig configuration to expose metrics for these components:
- Kafka
- Kafka Bridge
- Kafka Connect
- Kafka MirrorMaker 2
This enables the Streams for Apache Kafka Metrics Reporter, which exposes metrics on port 9404 through an HTTP endpoint. Prometheus scrapes this endpoint to collect Kafka metrics.
You can create your own Prometheus configuration or use the example custom resource file provided with Streams for Apache Kafka:
-
kafka-metrics.yaml -
kafka-connect-metrics.yaml -
kafka-mirror-maker-2-metrics.yaml -
kafka-bridge-metrics.yaml
These files contain the necessary configuration, and are a good starting point for trying Prometheus with Streams for Apache Kafka.
This procedure shows how to deploy example Prometheus metrics configuration to the Kafka resource.
Procedure
Deploy the example custom resource with the Prometheus configuration.
For example, for each
Kafkaresource you can apply thekafka-metrics.yamlfile.Deploying the example configuration
oc apply -f kafka-metrics.yaml
oc apply -f kafka-metrics.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Alternatively, copy the example configuration in
kafka-metrics.yamlto your ownKafkaresource.Copying the example configuration
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Copy the
metricsConfigproperty to yourKafkaresource.Example metrics configuration for Kafka
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Set up monitoring for your Streams for Apache Kafka cluster on OpenShift Container Platform. Use OpenShift’s feature for monitoring user-defined projects, which provides a dedicated Prometheus instance to collect metrics from your Kafka components. Deploy a Grafana instance to visualize these metrics using pre-built dashboards.
The monitoring stack in the openshift-monitoring project is for core platform components only. Do not use the Prometheus and Grafana instances from this project for your Kafka monitoring. You must use the user workload monitoring stack described here.
With the prerequisites in place, set up Streams for Apache Kafka monitoring in OpenShift by following these procedures in order:
25.2.3.1. Prerequisites
-
Monitoring for user-defined projects is enabled. A cluster administrator has created a
cluster-monitoring-configConfigMapresource in your OpenShift cluster. -
You have been granted permissions to create monitoring resources in your project. This is typically done by assigning your user the
monitoring-rules-editormonitoring-editrole. Metrics exposed using one of the following approaches:
For more information on creating a cluster-monitoring-config config map and granting users permission to monitor user-defined projects, see the OpenShift documentation.
25.2.3.2. Deploying the Prometheus resources
Use Prometheus to obtain monitoring data in your Kafka cluster.
You can use your own Prometheus deployment or deploy Prometheus using the example metrics configuration files provided by Streams for Apache Kafka. To use the example files, you configure and deploy the PodMonitor resources. The PodMonitors scrape data directly from pods for Apache Kafka, Operators, the Kafka Bridge, and Cruise Control.
You then deploy the example alerting rules for Alertmanager.
Prerequisites
- A running Kafka cluster.
- Check the example alerting rules provided with Streams for Apache Kafka.
Procedure
Check that monitoring for user-defined projects is enabled:
oc get pods -n openshift-user-workload-monitoring
oc get pods -n openshift-user-workload-monitoringCopy to Clipboard Copied! Toggle word wrap Toggle overflow If enabled, pods for the monitoring components are returned. For example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If no pods are returned, monitoring for user-defined projects is disabled. See the Prerequisites in xref:assembly-metrics-str.
Multiple
PodMonitorresources are defined inexamples/metrics/prometheus-install/strimzi-pod-monitor.yaml.For each
PodMonitorresource, edit thespec.namespaceSelector.matchNamesproperty:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The project where the pods to scrape the metrics from are running, for example,
Kafka.
Deploy the
strimzi-pod-monitor.yamlfile to the project where your Kafka cluster is running:oc apply -f strimzi-pod-monitor.yaml -n <my-project>
oc apply -f strimzi-pod-monitor.yaml -n <my-project>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the example Prometheus rules to the same project:
oc apply -f prometheus-rules.yaml -n <my-project>
oc apply -f prometheus-rules.yaml -n <my-project>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
25.2.3.3. Creating a service account for Grafana
A Grafana instance for Streams for Apache Kafka needs to run with a service account that is assigned the cluster-monitoring-view role.
Create a service account if you are using Grafana to present metrics for monitoring.
Prerequisites
Procedure
Create a
ServiceAccountfor Grafana in the project containing your Kafka cluster:oc create sa grafana-service-account -n my-project
oc create sa grafana-service-account -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, a service account named
grafana-service-accountis created in themy-projectnamespace.Create a
ClusterRoleBindingresource that assigns thecluster-monitoring-viewrole to the GrafanaServiceAccount. Here the resource is namedgrafana-cluster-monitoring-binding.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the
ClusterRoleBindingto the same project:oc apply -f grafana-cluster-monitoring-binding.yaml -n my-project
oc apply -f grafana-cluster-monitoring-binding.yaml -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a token secret for the service account:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
Secretobject and access token:oc create -f <secret_configuration>.yaml
oc create -f <secret_configuration>.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow You need the access token when deploying Grafana.
25.2.3.4. Deploying Grafana with a Prometheus datasource
Deploy Grafana to present Prometheus metrics. A Grafana application requires configuration for the OpenShift Container Platform monitoring stack.
OpenShift Container Platform includes a Thanos Querier instance in the openshift-monitoring project. Thanos Querier is used to aggregate platform metrics.
To consume the required platform metrics, your Grafana instance requires a Prometheus data source that can connect to Thanos Querier. To configure this connection, you create a config map that authenticates, by using a token, to the oauth-proxy sidecar that runs alongside Thanos Querier. A datasource.yaml file is used as the source of the config map.
Finally, you deploy the Grafana application with the config map mounted as a volume to the project containing your Kafka cluster.
Prerequisites
Procedure
Get the access token of the Grafana
ServiceAccount:oc describe sa/grafana-service-account | grep Tokens: oc describe secret grafana-service-account-token-mmlp9 | grep token:
oc describe sa/grafana-service-account | grep Tokens: oc describe secret grafana-service-account-token-mmlp9 | grep token:Copy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, the service account is named
grafana-service-account. Copy the access token to use in the next step.Create a
datasource.yamlfile containing the Thanos Querier configuration for Grafana.Paste the access token into the
httpHeaderValue1property as indicated.Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
GRAFANA-ACCESS-TOKEN: The value of the access token for the GrafanaServiceAccount.
Create a config map named
grafana-configfrom thedatasource.yamlfile:oc create configmap grafana-config --from-file=datasource.yaml -n my-project
oc create configmap grafana-config --from-file=datasource.yaml -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow In this example, a config map named
grafana-configis created in themy-projectnamespace.Create a Grafana application consisting of a
Deploymentand aService.The
grafana-configconfig map is mounted as a volume for the datasource configuration.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Grafana application to the project containing your Kafka cluster:
oc apply -f <grafana-application> -n my-project
oc apply -f <grafana-application> -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow
25.2.3.5. Creating a route to the Grafana Service
You can access the Grafana user interface through a Route that exposes the Grafana service.
Prerequisites
Procedure
Create an edge route to the
grafanaservice:oc create route edge <my-grafana-route> --service=grafana --namespace=KAFKA-NAMESPACE
oc create route edge <my-grafana-route> --service=grafana --namespace=KAFKA-NAMESPACECopy to Clipboard Copied! Toggle word wrap Toggle overflow
25.2.3.6. Importing the example Grafana dashboards
Use Grafana to provide visualizations of Prometheus metrics on customizable dashboards.
Streams for Apache Kafka provides example dashboard configuration files for Grafana in JSON format.
-
examples/metrics/grafana-dashboards
This procedure uses the example Grafana dashboards.
The example dashboards are a good starting point for monitoring key metrics, but they don’t show all the metrics supported by Kafka. You can modify the example dashboards or add other metrics, depending on your infrastructure.
Prerequisites
Procedure
Get the details of the Route to the Grafana Service. For example:
oc get routes NAME HOST/PORT PATH SERVICES MY-GRAFANA-ROUTE MY-GRAFANA-ROUTE-amq-streams.net grafana
oc get routes NAME HOST/PORT PATH SERVICES MY-GRAFANA-ROUTE MY-GRAFANA-ROUTE-amq-streams.net grafanaCopy to Clipboard Copied! Toggle word wrap Toggle overflow - In a web browser, access the Grafana login screen using the URL for the Route host and port.
Enter your user name and password, and then click Log In.
The default Grafana user name and password are both
admin. After logging in for the first time, you can change the password.- In Configuration > Data Sources, check that the Prometheus data source was created. The data source was created in Section 25.2.3.4, “Deploying Grafana with a Prometheus datasource”.
- Click the + icon and then click Import.
-
In
examples/metrics/grafana-dashboards, copy the JSON of the dashboard to import. - Paste the JSON into the text box, and then click Load.
- Repeat steps 5-7 for the other example Grafana dashboards.
The imported Grafana dashboards are available to view from the Dashboards home page.
Use kube-state-metrics (KSM) to monitor the state and readiness of custom resources managed by Streams for Apache Kafka. KSM is a scalable OpenShift native service that listens to the Kubernetes API server and exports metrics about the state of the objects. Monitoring through KSM allows you to track the health of Streams for Apache Kafka components, identify resources that are not ready, and configure alerts for warning conditions.
You can use your own KSM deployment or deploy KSM using the example metrics configuration files provided by Streams for Apache Kafka. The example files include a configuration file for a KSM deployment:
-
examples/metrics/kube-state-metrics/ksm.yaml
If you are using the example deployment file, you can update the namespace by replacing the example myproject with your own:
On Linux, use:
sed -E -i '/[[:space:]]*namespace: [a-zA-Z0-9-]*$/s/namespace:[[:space:]]*[a-zA-Z0-9-]*$/namespace: myproject/' examples/metrics/kube-state-metrics/ksm.yaml
sed -E -i '/[[:space:]]*namespace: [a-zA-Z0-9-]*$/s/namespace:[[:space:]]*[a-zA-Z0-9-]*$/namespace: myproject/' examples/metrics/kube-state-metrics/ksm.yamlOn MacOS, use:
sed -i '' -e '/[[:space:]]*namespace: [a-zA-Z0-9-]*$/s/namespace:[[:space:]]*[a-zA-Z0-9-]*$/namespace: myproject/' examples/metrics/kube-state-metrics/ksm.yaml
sed -i '' -e '/[[:space:]]*namespace: [a-zA-Z0-9-]*$/s/namespace:[[:space:]]*[a-zA-Z0-9-]*$/namespace: myproject/' examples/metrics/kube-state-metrics/ksm.yaml
Streams for Apache Kafka also provides an example configuration ConfigMap for KSM:
-
examples/metrics/kube-state-metrics/configmap.yaml
This procedure uses the example KSM deployment and configuration file.
Prerequisites
Procedure
Deploy KSM:
oc apply -f configmap.yaml oc apply -f ksm.yaml
oc apply -f configmap.yaml oc apply -f ksm.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that Prometheus is scraping KSM metrics.
Metrics are scraped from the
strimzi-kube-state-metricsservice using theServiceMonitorconfigured for the KSM deployment.For alerting on these metrics, check the provided
PrometheusRuleresources:-
examples/metrics/kube-state-metrics/prometheus-rules.yaml
-
25.4. Consumer lag monitoring
Kafka Exporter is an open source project that enhances the monitoring of Apache Kafka brokers and clients. Kafka Exporter extracts additional metrics data from Kafka brokers related to consumer groups, consumer lag, topic offsets, and partitions. The metrics are exposed in Prometheus format and can be collected by Prometheus, then visualized in Grafana.
Kafka Exporter relies on data from the __consumer_offsets topic to report consumer lag metrics. This topic only contains information if consumer groups are actively committing offsets. Consumer groups must therefore be in use for Kafka Exporter to function correctly.
Consumer lag indicates the difference in the rate of production and consumption of messages. Specifically, consumer lag for a given consumer group indicates the delay between the last message in the partition and the message being currently picked up by that consumer.
The lag reflects the position of the consumer offset in relation to the end of the partition log.
Consumer lag between the producer and consumer offset
This difference is sometimes referred to as the delta between the producer offset and consumer offset: the read and write positions in the Kafka broker topic partitions.
Suppose a topic streams 100 messages a second. A lag of 1000 messages between the producer offset (the topic partition head) and the last offset the consumer has read means a 10-second delay.
Why monitor consumer lag?
For applications that rely on the processing of (near) real-time data, it is critical to monitor consumer lag to check that it does not become too big. The greater the lag becomes, the further the process moves from the real-time processing objective.
Consumer lag, for example, might be a result of consuming too much old data that has not been purged, or through unplanned shutdowns.
Reducing consumer lag
Use the Grafana charts to analyze lag and to check if actions to reduce lag are having an impact on an affected consumer group. If, for example, Kafka brokers are adjusted to reduce lag, the dashboard will show the Lag by consumer group chart going down and the Messages consumed per minute chart going up.
Typical actions to reduce lag include:
- Scaling-up consumer groups by adding new consumers
- Increasing the retention time for a message to remain in a topic
- Adding more disk capacity to increase the message buffer
Actions to reduce consumer lag depend on the underlying infrastructure and the use cases Streams for Apache Kafka is supporting. For instance, a lagging consumer is less likely to benefit from the broker being able to service a fetch request from its disk cache. And in certain cases, it might be acceptable to automatically drop messages until a consumer has caught up.
25.4.1. Deploying Kafka Exporter
To monitor consumer lag in your Kafka cluster, configure Kafka Exporter in the Kafka custom resource. Kafka Exporter exposes lag data as Prometheus metrics, which can be visualized in Grafana. A Grafana dashboard for Kafka Exporter is included in the example Grafana dashboards provided by Streams for Apache Kafka.
Kafka Exporter provides only metrics related to consumer groups and lag. To collect general Kafka metrics, configure metrics on the Kafka brokers. For more information, see Section 25.2, “Using Prometheus with Streams for Apache Kafka”.
Prerequisites
Consumer groups must be in use.
Kafka Exporter relies on data from the
__consumer_offsetstopic to report consumer lag metrics. This topic only contains information if consumer groups are actively committing offsets.
Procedure
Add
kafkaExporterconfiguration to thespecsection of theKafkaresource.Example configuration for deploying Kafka Exporter
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Regular expression to specify consumer groups to include in metrics.
- 2
- Regular expression to specify topics to include in metrics.
- 3
- Regular expression to specify consumer groups to exclude from metrics.
- 4
- Regular expression to specify topics to exclude from metrics.
- 6
- By default, metrics are collected for all consumers regardless of connection status. Setting
showAllOffsetstofalsestops collecting metrics for disconnected consumers. - 5
- CPU and memory resources to reserve.
- 7
- Logging configuration, to log messages with a given severity (debug, info, warn, error, fatal) or above.
- 8
- Boolean to enable Sarama logging, which provides detailed logs from the Go client library used by Kafka Exporter. Useful for debugging Kafka client interactions.
- 9
- Readiness probe to check when Kafka Exporter is ready to serve metrics.
- 10
- Liveness probe to detect and restart Kafka Exporter if it becomes unresponsive.
- 11
- Template customization. Here a pod is scheduled with additional security attributes.
- 12
- ADVANCED OPTION: Container image configuration, which is recommended only in special situations.
Apply the changes to the
Kafkaconfiguration.Resources, including a
ServiceandPod, are created for the Kafka Exporter with the naming convention<kafka_cluster_name>-kafka-exporter.Configure Prometheus to scrape metrics from the Kafka Exporter endpoint.
If you are using the example Prometheus deployment, it is already set up to discover and scrape Kafka Exporter metrics. The
PodMonitorresource namedkafka-resources-metricsmatches thestrimzi.io/kind: Kafkalabel used to identify the Kafka Exporter. For more information, see Section 25.2.3.2, “Deploying the Prometheus resources”.Import the Kafka Exporter dashboard into Grafana to visualize consumer lag.
For more information, see Section 25.2.3.6, “Importing the example Grafana dashboards”.
TipUse the Lag by consumer group and Messages consumed per second panels to evaluate lag and the impact of tuning actions.
25.5. Cruise Control operations monitoring
Cruise Control monitors Kafka brokers in order to track the utilization of brokers, topics, and partitions. Cruise Control also provides a set of metrics for monitoring its own performance.
The Cruise Control metrics reporter collects raw metrics data from Kafka brokers. The data is produced to topics that are automatically created by Cruise Control. The metrics are used to generate optimization proposals for Kafka clusters.
Cruise Control metrics are available for real-time monitoring of Cruise Control operations. For example, you can use Cruise Control metrics to monitor the status of rebalancing operations that are running or provide alerts on any anomalies that are detected in an operation’s performance.
For more information on exposing Cruise Control metrics, see Using Prometheus with Strimzi.
For a full list of available Cruise Control metrics, which are known as sensors, see the Cruise Control documentation
25.5.1. Monitoring balancedness scores
Cruise Control metrics include a balancedness score. Balancedness is the measure of how evenly a workload is distributed in a Kafka cluster.
The Cruise Control metric for balancedness score (balancedness-score) might differ from the balancedness score in the KafkaRebalance resource. Cruise Control calculates each score using anomaly.detection.goals which might not be the same as the default.goals used in the KafkaRebalance resource. The anomaly.detection.goals are specified in the spec.cruiseControl.config of the Kafka custom resource.
Refreshing the KafkaRebalance resource fetches an optimization proposal. The latest cached optimization proposal is fetched if one of the following conditions applies:
-
KafkaRebalance
goalsmatch the goals configured in thedefault.goalssection of theKafkaresource -
KafkaRebalance
goalsare not specified
Otherwise, Cruise Control generates a new optimization proposal based on KafkaRebalance goals. If new proposals are generated with each refresh, this can impact performance monitoring.
25.5.2. Setting up alerts for anomaly detection
Cruise control’s anomaly detector provides metrics data for conditions that block the generation of optimization goals, such as broker failures. If you want more visibility, you can use the metrics provided by the anomaly detector to set up alerts and send out notifications. You can set up Cruise Control’s anomaly notifier to route alerts based on these metrics through a specified notification channel. Alternatively, you can set up Prometheus to scrape the metrics data provided by the anomaly detector and generate alerts. Prometheus Alertmanager can then route the alerts generated by Prometheus.
The Cruise Control configurations documentation provides information on AnomalyDetector metrics and the anomaly notifier.
Chapter 26. Introducing distributed tracing
Distributed tracing allows you to track the progress of transactions between applications in a distributed system. In a microservices architecture, tracing tracks the progress of transactions between services. Trace data is useful for monitoring application performance and investigating issues with target systems and end-user applications.
In Streams for Apache Kafka, distributed tracing facilitates end-to-end tracking of messages: from source systems to Kafka, and then from Kafka to target systems and applications. This complements the monitoring of metrics in Grafana dashboards and component loggers.
Streams for Apache Kafka provides built-in support for tracing for the following Kafka components:
- MirrorMaker to trace messages from a source cluster to a target cluster
- Kafka Connect to trace messages consumed and produced by Kafka Connect
- Kafka Bridge to trace messages between Kafka and HTTP client applications
Tracing is not supported for Kafka brokers.
26.1. Tracing options
Distributed traces consist of spans, which represent individual units of work performed over a specific time period. When instrumented with tracers, applications generate traces that follow requests as they move through the system, making it easier to identify delays or issues.
OpenTelemetry, a telemetry framework, provides APIs for tracing that are independent of any specific backend tracing system. In Streams for Apache Kafka, the default protocol for transmitting traces between Kafka components and tracing systems is OpenTelemetry’s OTLP (OpenTelemetry Protocol), a vendor-neutral protocol.
While OTLP is the default, Streams for Apache Kafka also supports other tracing systems, such as Jaeger. Jaeger is a distributed tracing system designed for monitoring microservices, and its user interface allows you to query, filter, and analyze trace data in detail.
The Jaeger user interface showing a simple query
26.2. Environment variables for tracing
Use environment variables to enable tracing for Kafka components or to initialize a tracer for Kafka clients.
Tracing environment variables are subject to change. For the latest information, see the OpenTelemetry documentation.
The following table describes the key environment variables for setting up tracing with OpenTelemetry.
| Property | Required | Description |
|---|---|---|
|
| Yes | The name of the tracing service for OpenTelemetry, such as OTLP or Jaeger. |
|
| Yes (if using OTLP exporter) |
The OTLP endpoint for exporting trace data to the tracing system. For Jaeger tracing, specify the |
|
| No (unless using a non-OTLP exporter) |
The exporter used for tracing. The default is |
|
| No (required if using TLS with OTLP) |
The path to the file containing trusted certificates for TLS authentication. Required to secure communication between Kafka components and the OpenTelemetry endpoint when using TLS with the |
26.3. Setting up distributed tracing
Enable distributed tracing in Kafka components by specifying a tracing type in the custom resource. Instrument tracers in Kafka clients for end-to-end tracking of messages.
To set up distributed tracing, follow these procedures in order:
- Enable tracing for supported Kafka components
- Initialize a tracer for Kafka clients
Instrument clients with tracers, embedding telemetry-gathering functionality into the code:
26.3.1. Prerequisites
Before setting up distributed tracing, make sure backend components are deployed to your OpenShift cluster.
We recommend using the Jaeger operator for deploying Jaeger on your OpenShift cluster. For deployment instructions, see the Jaeger documentation.
Setting up tracing systems is outside the scope of this content.
26.3.2. Enabling tracing in supported Kafka components
Distributed tracing is supported for MirrorMaker, MirrorMaker 2, Kafka Connect, and the Kafka Bridge. Enable tracing using OpenTelemetry by setting the spec.tracing.type property to opentelemetry. Configure the custom resource of the component to specify and enable a tracing system using spec.template properties.
By default, OpenTelemetry uses the OTLP (OpenTelemetry Protocol) exporter and endpoint to gather trace data. This procedure shows the configuration to use OTLP as the tracing system.
If you prefer to use a different tracing system supported by OpenTelemetry, such as Jaeger, you can modify the exporter and endpoint settings in the tracing configuration.
Streams for Apache Kafka no longer supports OpenTracing. If you were previously using OpenTracing with the type: jaeger option, we encourage you to transition to using OpenTelemetry instead.
Enabling tracing in a resource triggers the following events:
- Interceptor classes are updated in the integrated consumers and producers of the component.
- For MirrorMaker, MirrorMaker 2, and Kafka Connect, the tracing agent initializes a tracer based on the tracing configuration defined in the resource.
- For the Kafka Bridge, a tracer based on the tracing configuration defined in the resource is initialized by the Kafka Bridge itself.
Tracing in MirrorMaker 2
For MirrorMaker 2, messages are traced from the source cluster to the target cluster. The trace data records messages entering and leaving the MirrorMaker 2 component.
Tracing in Kafka Connect
For Kafka Connect, only messages produced and consumed by Kafka Connect are traced. To trace messages sent between Kafka Connect and external systems, you must configure tracing in the connectors for those systems.
Tracing in the Kafka Bridge
For the Kafka Bridge, messages produced and consumed by the Kafka Bridge are traced. Incoming HTTP requests from client applications to send and receive messages through the Kafka Bridge are also traced. To have end-to-end tracing, you must configure tracing in your HTTP clients.
Procedure
Perform these steps for each KafkaMirrorMaker2, KafkaConnect, and KafkaBridge resource.
In the
spec.templateproperty, configure the tracer service.- Use the tracing environment variables as template configuration properties.
-
For OpenTelemetry, set the
spec.tracing.typeproperty toopentelemetry.
Example tracing configuration for Kafka Connect using OpenTelemetry
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example tracing configuration for MirrorMaker 2 using OpenTelemetry
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example tracing configuration for the Kafka Bridge using OpenTelemetry
Copy to Clipboard Copied! Toggle word wrap Toggle overflow (Optional) If TLS authentication is configured on the OTLP endpoint, use the
OTEL_EXPORTER_OTLP_CERTIFICATEenvironment variable to specify the path to a trusted certificate This secures communication between Kafka components and the OpenTelemetry endpoint.To provide the certificate, mount a volume containing the secret that holds the trusted certificate. Unless the endpoint address is redirected from
http, usehttps.Example configuration for TLS
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Apply the changes to the custom resource configuration.
26.3.3. Initializing tracing for Kafka clients
Initialize a tracer for OpenTelemetry, then instrument your client applications for distributed tracing. You can instrument Kafka producer and consumer clients, and Kafka Streams API applications.
Configure and initialize a tracer using a set of tracing environment variables.
Procedure
In each client application add the dependencies for the tracer:
Add the Maven dependencies to the
pom.xmlfile for the client application:Dependencies for OpenTelemetry
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Define the configuration of the tracer using the tracing environment variables.
Create a tracer, which is initialized with the environment variables:
Creating a tracer for OpenTelemetry
OpenTelemetry ot = GlobalOpenTelemetry.get();
OpenTelemetry ot = GlobalOpenTelemetry.get();Copy to Clipboard Copied! Toggle word wrap Toggle overflow Register the tracer as a global tracer:
GlobalTracer.register(tracer);
GlobalTracer.register(tracer);Copy to Clipboard Copied! Toggle word wrap Toggle overflow Instrument your client:
Instrument application code to enable tracing in Kafka producers and consumers. Use a decorator pattern or interceptors to instrument your Java producer and consumer application code for tracing. You can then record traces when messages are produced or retrieved from a topic.
OpenTelemetry instrumentation project provides classes that support instrumentation of producers and consumers.
- Decorator instrumentation
- For decorator instrumentation, create a modified producer or consumer instance for tracing.
- Interceptor instrumentation
- For interceptor instrumentation, add the tracing capability to the consumer or producer configuration.
Prerequisites
You have initialized tracing for the client.
You enable instrumentation in producer and consumer applications by adding the tracing JARs as dependencies to your project.
Procedure
Perform these steps in the application code of each producer and consumer application. Instrument your client application code using either a decorator pattern or interceptors.
To use a decorator pattern, create a modified producer or consumer instance to send or receive messages.
You pass the original
KafkaProducerorKafkaConsumerclass.Example decorator instrumentation for OpenTelemetry
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To use interceptors, set the interceptor class in the producer or consumer configuration.
You use the
KafkaProducerandKafkaConsumerclasses in the usual way. TheTracingProducerInterceptorandTracingConsumerInterceptorinterceptor classes take care of the tracing capability.Example producer configuration using interceptors
senderProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingProducerInterceptor.class.getName()); KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps); producer.send(...);senderProps.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingProducerInterceptor.class.getName()); KafkaProducer<Integer, String> producer = new KafkaProducer<>(senderProps); producer.send(...);Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example consumer configuration using interceptors
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Instrument application code to enable tracing in Kafka Streams API applications. Use a decorator pattern or interceptors to instrument your Kafka Streams API applications for tracing. You can then record traces when messages are produced or retrieved from a topic.
- Decorator instrumentation
-
For decorator instrumentation, create a modified Kafka Streams instance for tracing. For OpenTelemetry, you need to create a custom
TracingKafkaClientSupplierclass to provide tracing instrumentation for Kafka Streams. - Interceptor instrumentation
- For interceptor instrumentation, add the tracing capability to the Kafka Streams producer and consumer configuration.
Prerequisites
You have initialized tracing for the client.
You enable instrumentation in Kafka Streams applications by adding the tracing JARs as dependencies to your project.
-
To instrument Kafka Streams with OpenTelemetry, you’ll need to write a custom
TracingKafkaClientSupplier. The custom
TracingKafkaClientSuppliercan extend Kafka’sDefaultKafkaClientSupplier, overriding the producer and consumer creation methods to wrap the instances with the telemetry-related code.Example custom
TracingKafkaClientSupplierCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Procedure
Perform these steps for each Kafka Streams API application.
To use a decorator pattern, create an instance of the
TracingKafkaClientSuppliersupplier interface, then provide the supplier interface toKafkaStreams.Example decorator instrumentation
KafkaClientSupplier supplier = new TracingKafkaClientSupplier(tracer); KafkaStreams streams = new KafkaStreams(builder.build(), new StreamsConfig(config), supplier); streams.start();
KafkaClientSupplier supplier = new TracingKafkaClientSupplier(tracer); KafkaStreams streams = new KafkaStreams(builder.build(), new StreamsConfig(config), supplier); streams.start();Copy to Clipboard Copied! Toggle word wrap Toggle overflow To use interceptors, set the interceptor class in the Kafka Streams producer and consumer configuration.
The
TracingProducerInterceptorandTracingConsumerInterceptorinterceptor classes take care of the tracing capability.Example producer and consumer configuration using interceptors
props.put(StreamsConfig.PRODUCER_PREFIX + ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingProducerInterceptor.class.getName()); props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingConsumerInterceptor.class.getName());
props.put(StreamsConfig.PRODUCER_PREFIX + ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingProducerInterceptor.class.getName()); props.put(StreamsConfig.CONSUMER_PREFIX + ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, TracingConsumerInterceptor.class.getName());Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Instead of the default OTLP system, you can specify other tracing systems that are supported by OpenTelemetry. You do this by adding the required artifacts to the Kafka image provided with Streams for Apache Kafka. Any required implementation specific environment variables must also be set. You then enable the new tracing implementation using the OTEL_TRACES_EXPORTER environment variable.
This procedure shows how to implement Zipkin tracing.
Procedure
Add the tracing artifacts to the
/opt/kafka/libs/directory of the Kafka image.You can use the Kafka container image on the Red Hat Ecosystem Catalog as a base image for creating a new custom image.
OpenTelemetry artifact for Zipkin
io.opentelemetry:opentelemetry-exporter-zipkin
io.opentelemetry:opentelemetry-exporter-zipkinCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set the tracing exporter and endpoint for the new tracing implementation.
Example Zikpin tracer configuration
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
26.3.7. Specifying custom span names for OpenTelemetry
A tracing span is a logical unit of work in Jaeger, with an operation name, start time, and duration. Spans have built-in names, but you can specify custom span names in your Kafka client instrumentation where used.
Specifying custom span names is optional and only applies when using a decorator pattern in producer and consumer client instrumentation or Kafka Streams instrumentation.
Custom span names cannot be specified directly with OpenTelemetry. Instead, you retrieve span names by adding code to your client application to extract additional tags and attributes.
Example code to extract attributes
Kafka pods might be evicted during OpenShift upgrades, maintenance, or pod rescheduling. If your Kafka pods were deployed by Streams for Apache Kafka, you can use the Streams for Apache Kafka Drain Cleaner tool to handle the pod evictions. The Streams for Apache Kafka Drain Cleaner handles the eviction instead of OpenShift.
By deploying the Streams for Apache Kafka Drain Cleaner, you can use the Cluster Operator to move Kafka pods instead of OpenShift. The Cluster Operator ensures that the number of in sync replicas for topics are at or above the configured min.insync.replicas and Kafka can remain operational during the eviction process. The Cluster Operator waits for topics to synchronize, as the OpenShift worker nodes drain consecutively.
An admission webhook notifies the Streams for Apache Kafka Drain Cleaner of pod eviction requests to the Kubernetes API. The Streams for Apache Kafka Drain Cleaner then adds a rolling update annotation to the pods to be drained. This informs the Cluster Operator to perform a rolling update of an evicted pod.
If you are not using the Streams for Apache Kafka Drain Cleaner, you can add pod annotations to perform rolling updates manually.
Webhook configuration
The Streams for Apache Kafka Drain Cleaner deployment files include a ValidatingWebhookConfiguration resource file. The resource provides the configuration for registering the webhook with the Kubernetes API.
The configuration defines the rules for the Kubernetes API to follow in the event of a pod eviction request. The rules specify that only CREATE operations related to pods/eviction sub-resources are intercepted. If these rules are met, the API forwards the notification.
The clientConfig points to the Streams for Apache Kafka Drain Cleaner service and /drainer endpoint that exposes the webhook. The webhook uses a secure TLS connection, which requires authentication. The caBundle property specifies the certificate chain to validate HTTPS communication. Certificates are encoded in Base64.
Webhook configuration for pod eviction notifications
Deploy the Streams for Apache Kafka Drain Cleaner to the OpenShift cluster where the Cluster Operator and Kafka cluster are running.
Streams for Apache Kafka Drain Cleaner can run in two different modes. By default, the Drain Cleaner denies (blocks) the OpenShift eviction request to prevent OpenShift from evicting the pods and instead uses the Cluster Operator to move the pod. This mode has better compatibility with various cluster autoscaling tools and does not require any specific PodDisuptionBudget configuration. Alternatively, you can enable the legacy mode where it allows the eviction request while also instructing the Cluster Operator to move the pod. For the legacy mode to work, you have to configure the PodDisruptionBudget to not allow any pod evictions by setting the maxUnavailable option to 0.
Prerequisites
- The Drain Cleaner deployment files, which are included in the Streams for Apache Kafka deployment files.
- You have a highly available Kafka cluster deployment running with OpenShift worker nodes that you would like to update.
Topics are replicated for high availability.
Topic configuration specifies a replication factor of at least 3 and a minimum number of in-sync replicas to 1 less than the replication factor.
Kafka topic replicated for high availability
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Using Drain Cleaner in legacy mode
To use the Drain Cleaner in legacy mode, change the default environment variables in the Drain Cleaner Deployment configuration file:
-
Set
STRIMZI_DENY_EVICTIONtofalseto use the legacy mode relying on thePodDisruptionBudgetconfiguration.
Example configuration to use legacy mode
Procedure
If you are using the legacy mode activated by setting the
STRIMZI_DENY_EVICTIONenvironment variable tofalse, you must also configure thePodDisruptionBudgetresource. SetmaxUnavailableto0(zero) in the Kafka section of theKafkaresource usingtemplatesettings.Specifying a pod disruption budget
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This setting prevents the automatic eviction of pods in case of planned disruptions, leaving the Streams for Apache Kafka Drain Cleaner and Cluster Operator to roll the pods on different worker nodes.
Update the
Kafkaresource:oc apply -f <kafka_configuration_file>
oc apply -f <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Streams for Apache Kafka Drain Cleaner.
To run the Drain Cleaner on OpenShift, apply the resources in the
/install/drain-cleaner/openshiftdirectory.oc apply -f ./install/drain-cleaner/openshift
oc apply -f ./install/drain-cleaner/openshiftCopy to Clipboard Copied! Toggle word wrap Toggle overflow
27.2. Using the Streams for Apache Kafka Drain Cleaner
Use the Streams for Apache Kafka Drain Cleaner in combination with the Cluster Operator to move Kafka broker pods from nodes that are being drained. When you run the Streams for Apache Kafka Drain Cleaner, it annotates pods with a rolling update pod annotation. The Cluster Operator performs rolling updates based on the annotation.
Prerequisites
Considerations when using anti-affinity configuration
When using anti-affinity with your Kafka pods, consider adding a spare worker node to your cluster. Including spare nodes ensures that your cluster has the capacity to reschedule pods during node draining or temporary unavailability of other nodes. When a worker node is drained, and anti-affinity rules restrict pod rescheduling on alternative nodes, spare nodes help prevent restarted pods from becoming unschedulable. This mitigates the risk of the draining operation failing.
Procedure
Drain a specified OpenShift node hosting the Kafka broker.
oc get nodes oc drain <name_of_node> --delete-emptydir-data --ignore-daemonsets --timeout=6000s --force
oc get nodes oc drain <name_of_node> --delete-emptydir-data --ignore-daemonsets --timeout=6000s --forceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the eviction events in the Streams for Apache Kafka Drain Cleaner log to verify that the pods have been annotated for restart.
Streams for Apache Kafka Drain Cleaner log show annotations of pods
INFO ... Received eviction webhook for Pod my-cluster-kafka-0 in namespace my-project INFO ... Pod my-cluster-kafka-0 in namespace my-project will be annotated for restart INFO ... Pod my-cluster-kafka-0 in namespace my-project found and annotated for restart
INFO ... Received eviction webhook for Pod my-cluster-kafka-0 in namespace my-project INFO ... Pod my-cluster-kafka-0 in namespace my-project will be annotated for restart INFO ... Pod my-cluster-kafka-0 in namespace my-project found and annotated for restartCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the reconciliation events in the Cluster Operator log to verify the rolling updates.
Cluster Operator log shows rolling updates
INFO PodOperator:68 - Reconciliation #13(timer) Kafka(my-project/my-cluster): Rolling Pod my-cluster-kafka-0 INFO AbstractOperator:500 - Reconciliation #13(timer) Kafka(my-project/my-cluster): reconciled
INFO PodOperator:68 - Reconciliation #13(timer) Kafka(my-project/my-cluster): Rolling Pod my-cluster-kafka-0 INFO AbstractOperator:500 - Reconciliation #13(timer) Kafka(my-project/my-cluster): reconciledCopy to Clipboard Copied! Toggle word wrap Toggle overflow
By default, the Drain Cleaner deployment watches the secret containing the TLS certificates its uses for authentication. The Drain Cleaner watches for changes, such as certificate renewals. If it detects a change, it restarts to reload the TLS certificates. The Drain Cleaner installation files enable this behavior by default. But you can disable the watching of certificates by setting the STRIMZI_CERTIFICATE_WATCH_ENABLED environment variable to false in the Deployment configuration (060-Deployment.yaml) of the Drain Cleaner installation files.
With STRIMZI_CERTIFICATE_WATCH_ENABLED enabled, you can also use the following environment variables for watching TLS certificates.
| Environment Variable | Description | Default |
|---|---|---|
|
| Enables or disables the certificate watch |
|
|
| The namespace where the Drain Cleaner is deployed and where the certificate secret exists |
|
|
| The Drain Cleaner pod name | - |
|
| The name of the secret containing TLS certificates |
|
|
| The list of fields inside the secret that contain the TLS certificates |
|
Example environment variable configuration to control watch operations
Use the Downward API (as described in the Openshift Nodes guide) mechanism to configure STRIMZI_CERTIFICATE_WATCH_NAMESPACE and STRIMZI_CERTIFICATE_WATCH_POD_NAME.
The report.sh diagnostics tool is a script provided by Red Hat to gather essential data for troubleshooting Streams for Apache Kafka deployments on OpenShift. It collects relevant logs, configuration files, and other diagnostic data to assist in identifying and resolving issues. When you run the script, you can specify additional parameters to retrieve specific data.
Prerequisites
- Bash 4 or newer to run the script.
-
The OpenShift
occommand-line tool is installed and configured to connect to the running cluster.
This establishes the necessary authentication for the oc command-line tool to interact with your cluster and retrieve the required diagnostic data.
Procedure
Download and extract the tool.
The diagnostics tool is available from Streams for Apache Kafka software downloads page.
From the directory where you extracted the tool, open a terminal and run the reporting tool:
./report.sh --namespace=<cluster_namespace> --cluster=<cluster_name> --out-dir=<local_output_directory>
./report.sh --namespace=<cluster_namespace> --cluster=<cluster_name> --out-dir=<local_output_directory>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<cluster_namespace>with the actual OpenShift namespace of your Streams for Apache Kafka deployment,<cluster_name>with the name of your Kafka cluster, and<local_output_directory>with the path to the local directory where you want to save the generated report. If you don’t specify a directory, a temporary directory is created.Include other optional reporting options, as necessary:
- --bridge=<string>
- Specify the name of the Kafka Bridge cluster to get data on its pods and logs.
- --connect=<string>
- Specify the name of the Kafka Connect cluster to get data on its pods and logs.
- --mm2=<string>
- Specify the name of the Mirror Maker 2 cluster to get data on its pods and logs.
- --secrets=(off|hidden|all)
Specify the secret verbosity level. The default is
hidden. The available options are as follows:-
all: Secret keys and data values are reported. -
hidden: Secrets with only keys are reported. Data values, such as passwords, are removed. -
off: Secrets are not reported at all.
-
Example request with data collection options
./report.sh --namespace=my-amq-streams-namespace --cluster=my-kafka-cluster --bridge=my-bridge-component --secrets=all --out-dir=~/reports
./report.sh --namespace=my-amq-streams-namespace --cluster=my-kafka-cluster --bridge=my-bridge-component --secrets=all --out-dir=~/reportsCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf required, assign execute permissions on the script to your user with the
chmodcommand. For example,chmod +x report.sh.
After the script has finished executing, the output directory contains files and directories of logs, configurations, and other diagnostic data collected for each component of your Streams for Apache Kafka deployment.
Data collected by the reporting diagnostics tool
Data on the following components is returned if present:
Cluster Operator
- Deployment YAML and logs
- All related pods and their logs
- YAML files for resources related to the cluster operator (ClusterRoles, ClusterRoleBindings)
Drain Cleaner (if present)
- Deployment YAML and logs
- Pod logs
Custom Resources
- Custom Resource Definitions (CRD) YAML
- YAML files for all related Custom Resources (CR)
Events
- Events related to the specified namespace
Configurations
-
Kafka pod logs and configuration file (
strimzi.properties) - Entity Operator (Topic Operator, User Operator) pod logs
- Cruise Control pod logs
- Kafka Exporter pod logs
- Bridge pod logs if specified in the options
- Connect pod logs if specified in the options
- MirrorMaker 2 pod logs if specified in the options
Secrets (if requested in the options)
- YAML files for all secrets related to the specified Kafka cluster
Chapter 29. Managing rolling updates
Use annotations to manually trigger a rolling update of Kafka and other operands through the Cluster Operator. Initiate rolling updates of Kafka, Kafka Connect, and MirrorMaker 2 clusters.
Manually performing a rolling update on a specific pod or set of pods is usually only required in exceptional circumstances. However, rather than deleting the pods directly, if you perform the rolling update through the Cluster Operator you ensure the following:
- The manual deletion of the pod does not conflict with simultaneous Cluster Operator operations, such as deleting other pods in parallel.
- The Cluster Operator logic handles the Kafka configuration specifications, such as the number of in-sync replicas.
This procedure describes how to trigger a rolling update of Kafka, Kafka Connect, or MirrorMaker 2 clusters. To trigger the update, you add an annotation to the StrimziPodSet that manages the pods running on the cluster.
Prerequisites
To perform a manual rolling update, you need a running Cluster Operator. The cluster for the component you are updating, whether it’s Kafka, Kafka Connect, or MirrorMaker 2 must also be running.
Procedure
Find the name of the resource that controls the pods you want to manually update.
For example, if your Kafka cluster is named my-cluster, the corresponding name is my-cluster-kafka. For a Kafka Connect cluster named my-connect-cluster, the corresponding name is my-connect-cluster-connect. And for a MirrorMaker 2 cluster named my-mm2-cluster, the corresponding name is my-mm2-cluster-mirrormaker2.
Use
oc annotateto annotate the appropriate resource in OpenShift.Annotating a StrimziPodSet
oc annotate strimzipodset <cluster_name>-kafka strimzi.io/manual-rolling-update="true" oc annotate strimzipodset <cluster_name>-connect strimzi.io/manual-rolling-update="true" oc annotate strimzipodset <cluster_name>-mirrormaker2 strimzi.io/manual-rolling-update="true"
oc annotate strimzipodset <cluster_name>-kafka strimzi.io/manual-rolling-update="true" oc annotate strimzipodset <cluster_name>-connect strimzi.io/manual-rolling-update="true" oc annotate strimzipodset <cluster_name>-mirrormaker2 strimzi.io/manual-rolling-update="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Wait for the next reconciliation to occur (every two minutes by default). A rolling update of all pods within the annotated resource is triggered, as long as the annotation was detected by the reconciliation process. When the rolling update of all the pods is complete, the annotation is automatically removed from the resource.
29.2. Performing a rolling update using a pod annotation
This procedure describes how to manually trigger a rolling update of existing Kafka, Kafka Connect, or MirrorMaker 2 clusters using an OpenShift Pod annotation. When multiple pods are annotated, consecutive rolling updates are performed within the same reconciliation run.
Prerequisites
To perform a manual rolling update, you need a running Cluster Operator. The cluster for the component you are updating, whether it’s Kafka, Kafka Connect, or MirrorMaker 2, must also be running.
You can perform a rolling update on a Kafka cluster regardless of the topic replication factor used. But for Kafka to stay operational during the update, you’ll need the following:
- A highly available Kafka cluster deployment running with nodes that you wish to update.
Topics replicated for high availability.
Topic configuration specifies a replication factor of at least 3 and a minimum number of in-sync replicas to 1 less than the replication factor.
Kafka topic replicated for high availability
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Procedure
Find the name of the
Podyou want to manually update.Pod naming conventions are as follows:
-
<cluster_name>-kafka-<index_number>for a Kafka cluster -
<cluster_name>-connect-<index_number>for a Kafka Connect cluster -
<cluster_name>-mirrormaker2-<index_number>for a MirrorMaker 2 cluster
The
<index_number>assigned to a pod starts at zero and ends at the total number of replicas minus one.-
Use
oc annotateto annotate thePodresource in OpenShift:oc annotate pod <cluster_name>-kafka-<index_number> strimzi.io/manual-rolling-update="true" oc annotate pod <cluster_name>-connect-<index_number> strimzi.io/manual-rolling-update="true" oc annotate pod <cluster_name>-mirrormaker2-<index_number> strimzi.io/manual-rolling-update="true"
oc annotate pod <cluster_name>-kafka-<index_number> strimzi.io/manual-rolling-update="true" oc annotate pod <cluster_name>-connect-<index_number> strimzi.io/manual-rolling-update="true" oc annotate pod <cluster_name>-mirrormaker2-<index_number> strimzi.io/manual-rolling-update="true"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Wait for the next reconciliation to occur (every two minutes by default). A rolling update of the annotated
Podis triggered, as long as the annotation was detected by the reconciliation process. When the rolling update of a pod is complete, the annotation is automatically removed from thePod.
The reconciliation continues even if the manual rolling update of the cluster fails. This allows the Cluster Operator to recover from certain rectifiable situations that can be addressed later in the reconciliation. For example, it can recreate a missing Persistent Volume Claim (PVC) or Persistent Volume (PV) that caused the update to fail.
Chapter 30. Finding information on Kafka restarts
After the Cluster Operator restarts a Kafka pod in an OpenShift cluster, it emits an OpenShift event into the pod’s namespace explaining why the pod restarted. For help in understanding cluster behavior, you can check restart events from the command line.
You can export and monitor restart events using metrics collection tools like Prometheus. Use the metrics tool with an event exporter that can export the output in a suitable format.
30.1. Reasons for a restart event
The Cluster Operator initiates a restart event for a specific reason. You can check the reason by fetching information on the restart event.
| Event | Description |
|---|---|
| CaCertHasOldGeneration | The pod is still using a server certificate signed with an old CA, so needs to be restarted as part of the certificate update. |
| CaCertRemoved | Expired CA certificates have been removed, and the pod is restarted to run with the current certificates. |
| CaCertRenewed | CA certificates have been renewed, and the pod is restarted to run with the updated certificates. |
| ClusterCaCertKeyReplaced | The key used to sign the cluster’s CA certificates has been replaced, and the pod is being restarted as part of the CA renewal process. |
| ConfigChangeRequiresRestart | Some Kafka configuration properties are changed dynamically, but others require that the broker be restarted. |
| FileSystemResizeNeeded | The file system size has been increased, and a restart is needed to apply it. |
| KafkaCertificatesChanged | One or more TLS certificates used by the Kafka broker have been updated, and a restart is needed to use them. |
| ManualRollingUpdate |
A user annotated the pod, or the |
| PodForceRestartOnError | An error occurred that requires a pod restart to rectify. |
| PodHasOldRevision |
A disk was added or removed from the Kafka volumes, and a restart is needed to apply the change. When using |
| PodHasOldRevision |
The |
| PodStuck | The pod is still pending, and is not scheduled or cannot be scheduled, so the operator has restarted the pod in a final attempt to get it running. |
| PodUnresponsive | Streams for Apache Kafka was unable to connect to the pod, which can indicate a broker not starting correctly, so the operator restarted it in an attempt to resolve the issue. |
30.2. Restart event filters
When checking restart events from the command line, you can specify a field-selector to filter on OpenShift event fields.
The following fields are available when filtering events with field-selector.
regardingObject.kind-
The resource that owns the Pod being restarted, and for restart events, the kind is always
Kafka. regarding.namespace- The namespace that the resource belongs to.
regardingObject.name-
The resource’s name, for example,
strimzi-cluster. regardingObject.uid- The unique ID of the resource.
reason-
The reason the Pod was restarted, for example,
JbodVolumesChanged. reportingController-
The reporting component is always
strimzi.io/cluster-operatorfor Streams for Apache Kafka restart events. source-
sourceis an older version ofreportingController. The reporting component is alwaysstrimzi.io/cluster-operatorfor Streams for Apache Kafka restart events. type-
The event type, which is either
WarningorNormal. For Streams for Apache Kafka restart events, the type isNormal.
In older versions of OpenShift, the fields using the regarding prefix might use an involvedObject prefix instead. reportingController was previously called reportingComponent.
30.3. Checking Kafka restarts
Use a oc command to list restart events initiated by the Cluster Operator. Filter restart events emitted by the Cluster Operator by setting the Cluster Operator as the reporting component using the reportingController or source event fields.
Prerequisites
- The Cluster Operator is running in the OpenShift cluster.
Procedure
Get all restart events emitted by the Cluster Operator:
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example showing events returned
LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal CaCertRenewed kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-0 due to CA certificate renewed 58m Normal PodForceRestartOnError kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-1 due to Pod needs to be forcibly restarted due to an error 5m47s Normal ManualRollingUpdate kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-2 due to Pod was manually annotated to be rolled
LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal CaCertRenewed kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-0 due to CA certificate renewed 58m Normal PodForceRestartOnError kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-1 due to Pod needs to be forcibly restarted due to an error 5m47s Normal ManualRollingUpdate kafka/strimzi-cluster Rolling Pod strimzi-cluster-kafka-2 due to Pod was manually annotated to be rolledCopy to Clipboard Copied! Toggle word wrap Toggle overflow You can also specify a
reasonor otherfield-selectoroptions to constrain the events returned.Here, a specific reason is added:
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnError
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnErrorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Use an output format, such as YAML, to return more detailed information about one or more events.
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnError -o yaml
oc -n kafka get events --field-selector reportingController=strimzi.io/cluster-operator,reason=PodForceRestartOnError -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example showing detailed events output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
The following fields are deprecated, so they are not populated for these events:
-
firstTimestamp -
lastTimestamp -
source
Chapter 31. Upgrading Streams for Apache Kafka
Download the latest Streams for Apache Kafka deployment files and upgrade your Streams for Apache Kafka installation to version 3.1 to benefit from new features, performance improvements, and enhanced security options. During the upgrade, Kafka is also be updated to the latest supported version, introducing additional features and bug fixes to your Streams for Apache Kafka deployment.
Use the same method to upgrade the Cluster Operator as the initial method of deployment. For example, if you used the Streams for Apache Kafka installation files, modify those files to perform the upgrade. After you have upgraded your Cluster Operator to 3.1, the next step is to upgrade all Kafka nodes to the latest supported version of Kafka. Kafka upgrades are performed by the Cluster Operator through rolling updates of the Kafka nodes.
If you encounter any issues with the new version, Streams for Apache Kafka can be downgraded to the previous version.
Released Streams for Apache Kafka versions can be found at Streams for Apache Kafka software downloads page.
Upgrade without downtime
For topics configured with high availability (replication factor of at least 3 and evenly distributed partitions), the upgrade process should not cause any downtime for consumers and producers.
The upgrade triggers rolling updates, where brokers are restarted one by one at different stages of the process. During this time, overall cluster availability is temporarily reduced, which may increase the risk of message loss in the event of a broker failure.
31.1. Required upgrade sequence
To upgrade brokers and clients without downtime, you must complete the Streams for Apache Kafka upgrade procedures in the following order:
Make sure your OpenShift cluster version is supported.
Streams for Apache Kafka 3.1 requires OpenShift 4.16 to 4.20.
Make sure you use Streams for Apache Kafka 2.7 or newer and all your Apache Kafka clusters are Kraft-based.
ZooKeeper-based Apache Kafka clusters are not supported anymore and need to be migrated to KRaft before upgrading the Cluster Operator or Apache Kafka.
- Upgrade the Cluster Operator.
-
Update the Kafka
versionandmetadataVersion.
From Streams for Apache Kafka 2.7, upgrades and downgrades between KRaft-based clusters are supported.
31.2. Streams for Apache Kafka upgrade paths
Two upgrade paths are available for Streams for Apache Kafka.
- Incremental upgrade
- An incremental upgrade moves between consecutive minor versions (such as 3.0 to 3.1), following a supported upgrade path.
- Multi-version upgrade
- A multi-version upgrade skips one or more minor versions and is supported only between consecutive Long-Term Support (LTS) versions. From the OperatorHub, you can perform a multi-version upgrade by selecting the version-specific channel for the target LTS version, and manually updating the Kafka version. While temporary errors due to Kafka version changes may occur, they are expected and can be resolved by updating the Kafka version during the upgrade.
Before upgrading in production, test your specific scenario in a controlled environment to identify potential issues.
Multi version upgrade is not supported in the LTS channel (amq-streams-lts). No further LTS versions are being added to the LTS channel, which is deprecated and will be removed in a later release. For upgrade between LTS versions, use the version-specific channels.
31.2.1. Support for Kafka versions when upgrading
When upgrading Streams for Apache Kafka, it is important to ensure compatibility with the Kafka version being used.
Multi-version upgrades are possible even if the supported Kafka versions differ between the old and new versions. However, if you attempt to upgrade to a new Streams for Apache Kafka version that does not support the current Kafka version, an error indicating that the Kafka version is not supported is generated. In this case, you must upgrade the Kafka version as part of the Streams for Apache Kafka upgrade by changing the spec.kafka.version in the Kafka custom resource to the supported version for the new Streams for Apache Kafka version.
Streams for Apache Kafka 3.1 supports upgrades only for KRaft-based Apache Kafka clusters managed by Streams for Apache Kafka 2.7 and newer. When upgrading from older Streams for Apache Kafka versions, please make sure to first upgrade to a Streams for Apache Kafka version from 2.7 to 2.9 and migrate to KRaft before upgrading to Streams for Apache Kafka 3.1.
31.2.3. Kafka version and image mappings
When upgrading Kafka, consider your settings for the STRIMZI_KAFKA_IMAGES environment variable and the Kafka.spec.kafka.version property.
-
Each
Kafkaresource can be configured with aKafka.spec.kafka.version, which defaults to the latest supported Kafka version (4.1.0) if not specified. The Cluster Operator’s
STRIMZI_KAFKA_IMAGESenvironment variable provides a mapping (<kafka_version>=<image>) between a Kafka version and the image to be used when a specific Kafka version is requested in a givenKafkaresource. For example,4.1.0=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0.-
If
Kafka.spec.kafka.imageis not configured, the default image for the given version is used. -
If
Kafka.spec.kafka.imageis configured, the default image is overridden.
-
If
The Cluster Operator cannot validate that an image actually contains a Kafka broker of the expected version. Take care to ensure that the given image corresponds to the given Kafka version.
31.3. Strategies for upgrading clients
Upgrading Kafka clients ensures that they benefit from the features, fixes, and improvements that are introduced in new versions of Kafka. Upgraded clients maintain compatibility with other upgraded Kafka components. The performance and stability of the clients might also be improved.
Consider the best approach for upgrading Kafka clients and brokers to ensure a smooth transition. The chosen upgrade strategy depends on whether you are upgrading brokers or clients first. Since Kafka 3.0, you can upgrade brokers and client independently and in any order. The decision to upgrade clients or brokers first depends on several factors, such as the number of applications that need to be upgraded and how much downtime is tolerable.
If you upgrade clients before brokers, some new features may not work as they are not yet supported by brokers. However, brokers can handle producers and consumers running with different versions and supporting different log message versions.
31.4. Upgrading OpenShift with minimal downtime
If you are upgrading OpenShift, refer to the OpenShift upgrade documentation to check the upgrade path and the steps to upgrade your nodes correctly. Before upgrading OpenShift, check the supported versions for your version of Streams for Apache Kafka.
When performing your upgrade, ensure the availability of your Kafka clusters by following these steps:
- Configure pod disruption budgets
Roll pods using one of these methods:
- Use the Streams for Apache Kafka Drain Cleaner (recommended)
- Apply an annotation to your pods to roll them manually
For Kafka to stay operational, topics must also be replicated for high availability. This requires topic configuration that specifies a replication factor of at least 3 and a minimum number of in-sync replicas to 1 less than the replication factor.
Kafka topic replicated for high availability
In a highly available environment, the Cluster Operator maintains a minimum number of in-sync replicas for topics during the upgrade process so that there is no downtime.
31.4.1. Rolling pods using Drain Cleaner
When using the Streams for Apache Kafka Drain Cleaner to evict nodes during OpenShift upgrade, it annotates pods with a manual rolling update annotation to inform the Cluster Operator to perform a rolling update of the pod that should be evicted and have it moved away from the OpenShift node that is being upgraded.
For more information, see Chapter 27, Evicting pods with the Streams for Apache Kafka Drain Cleaner.
As an alternative to using the Drain Cleaner to roll pods, you can trigger a manual rolling update of pods through the Cluster Operator. Using Pod resources, rolling updates restart the pods of resources with new pods. To replicate the operation of the Drain Cleaner by keeping topics available, you must also set the maxUnavailable value to zero for the pod disruption budget. Reducing the pod disruption budget to zero prevents OpenShift from evicting pods automatically.
Specifying a pod disruption budget
The PodDisruptionBudget resource created for Kafka clusters covers all associated node pool pods.
Watch the node pool pods that need to be drained and add a pod annotation to make the update. Here, the annotation updates a Kafka pod named my-cluster-pool-a-1.
Performing a manual rolling update on a Kafka pod
oc annotate pod my-cluster-pool-a-1 strimzi.io/manual-rolling-update="true"
oc annotate pod my-cluster-pool-a-1 strimzi.io/manual-rolling-update="true"31.5. Upgrading the Cluster Operator
Use the same method to upgrade the Cluster Operator as the initial method of deployment.
This procedure describes how to upgrade a Cluster Operator deployment to use Streams for Apache Kafka 3.1.
Follow this procedure if you deployed the Cluster Operator using the installation YAML files in the install/cluster-operator/ directory. The steps include the necessary configuration changes when the Cluster Operator watches multiple or all namespaces.
The availability of Kafka clusters managed by the Cluster Operator is not affected by the upgrade operation.
Refer to the documentation supporting a specific version of Streams for Apache Kafka for information on how to upgrade to that version.
Prerequisites
- An existing Cluster Operator deployment is available.
- You have downloaded the release artifacts for Streams for Apache Kafka 3.1.
-
You need an account with permission to create and manage
CustomResourceDefinitionand RBAC (ClusterRole, andRoleBinding) resources.
Currently, no feature gates require enabling or disabling before upgrading. If a feature gate introduces such a requirement, the details will be provided here.
Procedure
Take note of any configuration changes made during the previous Cluster Operator installation.
Any changes will be overwritten by the new version of the Cluster Operator.
- Update your custom resources to reflect the supported configuration options available for Streams for Apache Kafka version 3.1.
Modify the installation files for the new Cluster Operator version to reflect the namespace in which the Cluster Operator is running.
On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you modified environment variables in the
Deploymentconfiguration, edit the060-Deployment-strimzi-cluster-operator.yamlfile to use those environment variables.-
If the Cluster Operator is watching multiple namespaces, add the list of namespaces to the
STRIMZI_NAMESPACEenvironment variable. -
If the Cluster Operator is watching all namespaces, specify
value: "*"for theSTRIMZI_NAMESPACEenvironment variable.
-
If the Cluster Operator is watching multiple namespaces, add the list of namespaces to the
If the Cluster Operator is watching more than one namespace, update the role bindings.
If watching multiple namespaces, replace the
namespacein theRoleBindinginstallation files with the actual namespace name and create the role bindings for each namespace:Creating role bindings for a namespace
oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>
oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace> oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example, if the Cluster Operator is watching three namespaces, create three sets of role bindings by substituting
<watched_namespace>with the name of each namespace.If watching all namespaces, recreate the cluster role bindings that grant cluster-wide access (if needed):
Granting cluster-wide access using role bindings
oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow
When you have an updated configuration, deploy it along with the rest of the installation resources:
oc replace -f install/cluster-operator
oc replace -f install/cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the rolling updates to complete.
If the new operator version no longer supports the Kafka version you are upgrading from, an error message is returned.
To resolve this, upgrade to a supported Kafka version:
-
Edit the
Kafkacustom resource. -
Change the
spec.kafka.versionproperty to a supported Kafka version.
If no error message is returned, you can proceed to the next step and upgrade the Kafka version later.
-
Edit the
Get the image for the Kafka pod to ensure the upgrade was successful:
oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow The image tag shows the new Streams for Apache Kafka version followed by the Kafka version:
registry.redhat.io/amq-streams/strimzi-kafka-41-rhel9:3.1.0
registry.redhat.io/amq-streams/strimzi-kafka-41-rhel9:3.1.0Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can also check the upgrade has completed successfully from the status of the
Kafkaresource.
The Cluster Operator is upgraded to version 3.1, but the version of Kafka running in the cluster it manages is unchanged.
If you deployed Streams for Apache Kafka from OperatorHub, use the Operator Lifecycle Manager (OLM) to change the update channel for the Streams for Apache Kafka operators to a new Streams for Apache Kafka version.
Updating the channel starts one of the following types of upgrade, depending on your chosen upgrade strategy:
- An automatic upgrade is initiated
- A manual upgrade that requires approval before installation begins
If you subscribe to the stable channel, you can get automatic updates without changing channels. However, enabling automatic updates is not recommended because of the potential for missing any pre-installation upgrade steps. Use automatic upgrades only on version-specific channels.
For more information on using OperatorHub to upgrade Operators, see the Updating installed Operators (as described in the Openshift Operators guide).
If you upgrade the Cluster Operator to a version that does not support the current version of Kafka you are using, you get an unsupported Kafka version error. This error applies to all installation methods and means that you must upgrade Kafka to a supported Kafka version. Change the spec.kafka.version in the Kafka resource to the supported version.
You can use oc to check for error messages like this in the status of the Kafka resource.
Checking the Kafka status for errors
oc get kafka <kafka_cluster_name> -n <namespace> -o jsonpath='{.status.conditions}'
oc get kafka <kafka_cluster_name> -n <namespace> -o jsonpath='{.status.conditions}'Replace <kafka_cluster_name> with the name of your Kafka cluster and <namespace> with the OpenShift namespace where the pod is running.
31.6. Upgrading Kafka clusters
Upgrade a Kafka cluster to a newer supported Kafka version and KRaft metadata version.
Refer to the Apache Kafka documentation for the latest on support for Kafka upgrades.
Prerequisites
- The Cluster Operator is up and running.
-
Before you upgrade the Kafka cluster, check that the properties of the
Kafkaresource do not contain configuration options that are not supported in the new Kafka version.
Procedure
Update the Kafka cluster configuration:
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow If configured, check that the current
spec.kafka.metadataVersionis set to a version supported by the version of Kafka you are upgrading to.For example, the current version is 4.0-IV0 if upgrading from Kafka version 4.0.0 to 4.1.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If
metadataVersionis not configured, Streams for Apache Kafka automatically updates it to the current default after the update to the Kafka version in the next step.NoteThe value of
metadataVersionmust be a string to prevent it from being interpreted as a floating point number.Change the
Kafka.spec.kafka.versionto specify the new Kafka version; leave themetadataVersionat the default for the current Kafka version.NoteChanging the
kafka.versionensures that all brokers in the cluster are upgraded to use the new broker binaries. During the rolling upgrade, some brokers will still run the old binaries while others have already switched to the new ones. KeepingmetadataVersionunchanged at its current value ensures that all brokers and controllers remain compatible and can continue to communicate throughout the upgrade.NoteFor upgrades that involve only a micro release of Kafka, (for example, from X.x.0 to X.x.1) do not modify the
Kafka.spec.kafka.version. Keep the existing version setting. The Cluster Operator automatically applies the updated Kafka binaries for the micro release, even though thekafka.versionfield continues to show the previous minor version.For example, if upgrading from Kafka 4.0.0 to 4.1.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the image for the Kafka cluster is defined in
Kafka.spec.kafka.imageof theKafkacustom resource, update theimageto point to a container image with the new Kafka version.Save and exit the editor, then wait for the rolling updates to upgrade the Kafka nodes to complete.
Check the progress of the rolling updates by watching the pod state transitions:
oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow The rolling updates ensure that each pod is using the broker binaries for the new version of Kafka.
If required, set the
versionproperty for Kafka Connect and MirrorMaker as the new version of Kafka:-
For Kafka Connect, update
KafkaConnect.spec.version. For MirrorMaker 2, update
KafkaMirrorMaker2.spec.version.NoteIf you are using custom images that are built manually, you must rebuild those images to ensure that they are up-to-date with the latest Streams for Apache Kafka base image. For example, if you created a container image from the base Kafka Connect image, update the Dockerfile to point to the latest base image and build configuration.
-
For Kafka Connect, update
If configured, update the Kafka resource to use the new
metadataVersionversion. Otherwise, go to step 9.For example, if upgrading to Kafka 4.1.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow WarningExercise caution when changing the
metadataVersion, as downgrading may not be possible. You cannot downgrade Kafka if themetadataVersionfor the new Kafka version is higher than the Kafka version you wish to downgrade to. However, understand the potential implications on support and compatibility when maintaining an older version.Wait for the Cluster Operator to update the cluster.
Check the upgrade has completed successfully from the status of the
Kafkaresource.
Upgrading client applications
Ensure all Kafka client applications are updated to use the new version of the client binaries as part of the upgrade process and verify their compatibility with the Kafka upgrade. If needed, coordinate with the team responsible for managing the client applications.
To check that a client is using the latest message format, use the kafka.server:type=BrokerTopicMetrics,name={Produce|Fetch}MessageConversionsPerSec metric. The metric shows 0 if the latest message format is being used.
31.7. Checking the status of an upgrade
When performing an upgrade (or downgrade), you can check it completed successfully in the status of the Kafka custom resource. The status provides information on the Streams for Apache Kafka and Kafka versions being used.
To ensure that you have the correct versions after completing an upgrade, verify the kafkaVersion and operatorLastSuccessfulVersion values in the Kafka status.
-
operatorLastSuccessfulVersionis the version of the Streams for Apache Kafka operator that last performed a successful reconciliation. -
kafkaVersionis the version of Kafka being used by the Kafka cluster. -
kafkaMetadataVersionis the metadata version used by Kafka clusters.
You can use these values to check an upgrade of Streams for Apache Kafka or Kafka has completed.
Checking an upgrade from the Kafka status
Chapter 32. Downgrading Streams for Apache Kafka
If you are encountering issues with the version of Streams for Apache Kafka you upgraded to, you can revert to a previous version.
If you installed Streams for Apache Kafka using the YAML installation files, you can use the files from the previous release to perform the downgrade.
Downgrading Streams for Apache Kafka involves two key steps:
- Downgrading the Kafka cluster to a version supported by the target Cluster Operator (if required).
- Downgrading the Cluster Operator to the previous release.
If downgrading through multiple versions, you may need to repeat these steps.
Always downgrade Kafka before the Cluster Operator to maintain compatibility. Ensure that the Kafka version you downgrade to is supported by the target Cluster Operator. The metadata version of the Kafka cluster must not be higher than the maximum version supported by the downgraded Kafka version.
The following downgrade instructions are only suitable if you installed Streams for Apache Kafka using the installation files. If you installed Streams for Apache Kafka using another method, such as OperatorHub, downgrade may not be supported by that method unless specified in its documentation.
32.1. Downgrading Kafka clusters and client applications
You can downgrade a Kafka cluster to an earlier version only when the metadata format is compatible. Kafka permits downgrades only if there are no metadata changes between your current Kafka version and the target version. In practice, this means the cluster’s metadata version must be set to a value that the target Kafka version supports, and there must be no intervening metadata changes between the two versions.
If the cluster is already using the latest metadata version for the current release, you cannot downgrade. The downgrade is only possible if there is a metadata format that is valid in both the current and target Kafka versions. Consult the Apache Kafka documentation for information on supported metadata version values and downgrade limitations.
Prerequisites
- The Cluster Operator is up and running.
Before you downgrade the Kafka cluster, check the following for the
Kafkaresource:-
The
Kafkacustom resource does not include options that are unsupported by the target Kafka version. -
The target Kafka version supports the metadata version (
spec.kafka.metadataVersion) you plan to use.
-
The
You can verify the current Kafka and metadata version by checking the status of the Kafka custom resource.
Procedure
Update the Kafka cluster configuration:
oc edit kafka <kafka_configuration_file>
oc edit kafka <kafka_configuration_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Change the
metadataVersionversion to a value that is supported by both the current Kafka version and the target version. The Kafka version must be able to operate with this metadata format.Leave the
Kafka.spec.kafka.versionunchanged at the current Kafka version.For example, if downgrading from Kafka 4.1.0 to 4.0.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe value of
metadataVersionmust be a string to prevent it from being interpreted as a floating point number.-
Save the change, and wait for Cluster Operator to update
.status.kafkaMetadataVersionfor theKafkaresource. Change the
Kafka.spec.kafka.versionto the previous version.For example, if downgrading from Kafka 4.1.0 to 4.0.0:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the image for the Kafka version is different from the image defined in
STRIMZI_KAFKA_IMAGESfor the Cluster Operator, updateKafka.spec.kafka.image.Wait for the Cluster Operator to update the cluster.
You can check the downgrade has completed successfully from the status of the
Kafkaresource.Downgrade all client applications (consumers) to use the previous version of the client binaries.
The Kafka cluster and clients are now using the previous Kafka version.
32.2. Streams for Apache Kafka downgrade paths
Two downgrade paths are available for Streams for Apache Kafka.
- Incremental downgrade
- An incremental downgrade moves between consecutive minor versions (such as 3.1 to 3.0), following a supported downgrade path.
- Multi-version downgrade
- A multi-version downgrade skips one or more minor versions. While possible, this approach is not tested or supported, and may result in errors due to configuration changes or incompatible Kafka versions. See Table 32.1, “Known limitations in downgrade paths” for details on specific paths.
Before downgrading in production, test your specific scenario in a controlled environment to identify potential issues.
32.2.1. Support for Kafka versions when downgrading
When downgrading Streams for Apache Kafka, it is important to ensure compatibility with the Kafka version being used.
Multi-version downgrades are possible even if the supported Kafka versions differ between the old and new versions. However, if you attempt to downgrade to an older Streams for Apache Kafka version that does not support the current Kafka version, an error indicating that the Kafka version is not supported is generated. In this case, you must, if possible, downgrade the Kafka version as part of the Streams for Apache Kafka downgrade by changing the spec.kafka.version in the Kafka custom resource to the supported version for the target Streams for Apache Kafka version. If it is not possible to downgrade the Kafka version (for example in Streams for Apache Kafka version 2.9 or earlier), then the Streams for Apache Kafka downgrade path you are attempting is not supported.
The following table details known limitations for specific downgrade paths.
| Target Streams for Apache Kafka Version | Limitation |
|---|---|
| <= 2.9 | The current Kafka version must be supported by the target Streams for Apache Kafka version. It is only possible to downgrade from unsupported Kafka versions in Streams for Apache Kafka version 3.0 and higher. |
32.3. Downgrading the Cluster Operator
Downgrading the Cluster Operator involves reverting the Streams for Apache Kafka Cluster Operator to a previous version. You may also need to downgrade to a supported Kafka version.
This procedure describes how to downgrade a Cluster Operator deployment to a previous version.
Kafka clusters managed by Streams for Apache Kafka always use node pools. Downgrading to a cluster without node pools is not possible.
Prerequisites
- An existing Cluster Operator deployment is available.
- You have downloaded the installation files for the previous version.
Currently, no feature gates require enabling or disabling before downgrading. If a feature gate introduces such a requirement, the details will be provided here.
Procedure
Take note of any configuration changes made during the previous Cluster Operator installation.
Any changes will be overwritten by the previous version of the Cluster Operator.
- Revert your custom resources to reflect the supported configuration options available for the version of Streams for Apache Kafka you are downgrading to.
Update the Cluster Operator.
Modify the installation files for the previous version according to the namespace the Cluster Operator is running in.
On Linux, use:
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow On MacOS, use:
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml
sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
If you modified one or more environment variables in your existing Cluster Operator
Deployment, edit theinstall/cluster-operator/060-Deployment-strimzi-cluster-operator.yamlfile to use those environment variables.
When you have an updated configuration, deploy it along with the rest of the installation resources:
oc replace -f install/cluster-operator
oc replace -f install/cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the rolling updates to complete.
If the target operator version does not support the current Kafka version, an error message is returned.
To resolve this, downgrade to a supported Kafka version if possible:
-
Edit the
Kafkacustom resource. -
Change the
spec.kafka.versionproperty to a supported Kafka version.
If no error message is returned, you can proceed to the next step and upgrade the Kafka version later.
-
Edit the
Get the image for the Kafka pod to ensure the downgrade was successful:
oc get pod my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'oc get pod my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow The image tag shows the new Streams for Apache Kafka version followed by the Kafka version. For example,
<strimzi_version>-kafka-<kafka_version>.You can also check the downgrade has completed successfully from the status of the
Kafkaresource.
If you downgrade the Cluster Operator to a version that does not support the current version of Kafka you are using, you get an unsupported Kafka version error. This error applies to all installation methods and means that you must downgrade Kafka to a supported Kafka version. Change the spec.kafka.version in the Kafka resource to the supported version.
You can use oc to check for error messages like this in the status of the Kafka resource.
Checking the Kafka status for errors
oc get kafka <kafka_cluster_name> -n <namespace> -o jsonpath='{.status.conditions}'
oc get kafka <kafka_cluster_name> -n <namespace> -o jsonpath='{.status.conditions}'Replace <kafka_cluster_name> with the name of your Kafka cluster and <namespace> with the OpenShift namespace where the pod is running.
Chapter 33. Uninstalling Streams for Apache Kafka
You can uninstall Streams for Apache Kafka on OpenShift 4.16–4.20 (tested); 4.12, 4.14 (supported) from the OperatorHub using the OpenShift Container Platform web console or CLI.
Use the same approach you used to install Streams for Apache Kafka.
When you uninstall Streams for Apache Kafka, you will need to identify resources created specifically for a deployment and referenced from the Streams for Apache Kafka resource.
Such resources include:
- Secrets (Custom CAs and certificates, Kafka Connect secrets, and other Kafka secrets)
-
Logging
ConfigMaps(of typeexternal)
These are resources referenced by Kafka, KafkaConnect, KafkaMirrorMaker2, or KafkaBridge configuration.
Deleting CRDs and related custom resources
When a CustomResourceDefinition is deleted, custom resources of that type are also deleted. This includes the Kafka, KafkaConnect, KafkaMirrorMaker2, and KafkaBridge resources managed by Streams for Apache Kafka, as well as the StrimziPodSet resource Streams for Apache Kafka uses to manage the pods of the Kafka components. In addition, any OpenShift resources created by these custom resources, such as Deployment, Pod, Service, and ConfigMap resources, are also removed. Always exercise caution when deleting these resources to avoid unintended data loss.
This procedure describes how to uninstall Streams for Apache Kafka from the OperatorHub and remove resources related to the deployment.
You can perform the steps from the console or use alternative CLI commands.
Prerequisites
-
Access to an OpenShift Container Platform web console using an account with
cluster-adminorstrimzi-adminpermissions. You have identified the resources to be deleted.
You can use the following
ocCLI command to find resources and also verify that they have been removed when you have uninstalled Streams for Apache Kafka.Command to find resources related to a Streams for Apache Kafka deployment
oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>
oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <resource_type> with the type of the resource you are checking, such as
secretorconfigmap.
Procedure
- Navigate in the OpenShift web console to Operators > Installed Operators.
For the installed Streams for Apache Kafka operator, select the options icon (three vertical dots) and click Uninstall Operator.
The operator is removed from Installed Operators.
- Navigate to Home > Projects and select the project where you installed Streams for Apache Kafka and the Kafka components.
Click the options under Inventory to delete related resources.
Resources include the following:
- Deployments
- StatefulSets
- Pods
- Services
- ConfigMaps
- Secrets
TipUse the search to find related resources that begin with the name of the Kafka cluster. You can also find the resources under Workloads.
Alternative CLI commands
You can use CLI commands to uninstall Streams for Apache Kafka from the OperatorHub.
Delete the Streams for Apache Kafka subscription.
oc delete subscription amq-streams -n openshift-operators
oc delete subscription amq-streams -n openshift-operatorsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the cluster service version (CSV).
oc delete csv amqstreams.<version> -n openshift-operators
oc delete csv amqstreams.<version> -n openshift-operatorsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove related CRDs.
oc get crd -l app=strimzi -o name | xargs oc delete
oc get crd -l app=strimzi -o name | xargs oc deleteCopy to Clipboard Copied! Toggle word wrap Toggle overflow
This procedure describes how to use the oc command-line tool to uninstall Streams for Apache Kafka and remove resources related to the deployment.
Prerequisites
-
Access to an OpenShift cluster using an account with
cluster-adminorstrimzi-adminpermissions. You have identified the resources to be deleted.
You can use the following
ocCLI command to find resources and also verify that they have been removed when you have uninstalled Streams for Apache Kafka.Command to find resources related to a Streams for Apache Kafka deployment
oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>
oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <resource_type> with the type of the resource you are checking, such as
secretorconfigmap.
Procedure
Delete the Cluster Operator
Deployment, relatedCustomResourceDefinitions, andRBACresources.Specify the installation files used to deploy the Cluster Operator.
oc delete -f install/cluster-operator
oc delete -f install/cluster-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the resources you identified in the prerequisites.
oc delete <resource_type> <resource_name> -n <namespace>
oc delete <resource_type> <resource_name> -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace <resource_type> with the type of resource you are deleting and <resource_name> with the name of the resource.
Example to delete a secret
oc delete secret my-cluster-clients-ca-cert -n my-project
oc delete secret my-cluster-clients-ca-cert -n my-projectCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 34. Cluster recovery from persistent volumes
You can recover a Kafka cluster from persistent volumes (PVs) if they are still present.
34.1. Cluster recovery scenarios
Recovering from PVs is possible in the following scenarios:
- Unintentional deletion of a namespace
- Loss of an entire OpenShift cluster while PVs remain in the infrastructure
The recovery procedure for both scenarios is to recreate the original PersistentVolumeClaim (PVC) resources.
34.1.1. Recovering from namespace deletion
When you delete a namespace, all resources within that namespace—including PVCs, pods, and services—are deleted. If the reclaimPolicy for the PV resource specification is set to Retain, the PV retains its data and is not deleted. This configuration allows you to recover from namespace deletion.
PV configuration to retain data
Alternatively, PVs can inherit the reclaim policy from an associated storage class. Storage classes are used for dynamic volume allocation.
By configuring the reclaimPolicy property for the storage class, PVs created with this class use the specified reclaim policy. The storage class is assigned to the PV using the storageClassName property.
Storage class configuration to retain data
Storage class specified for PV
When using Retain as the reclaim policy, you must manually delete PVs if you intend to delete the entire cluster.
34.1.2. Recovering from cluster loss
If you lose the entire OpenShift cluster, all resources—including PVs, PVCs, and namespaces—are lost. However, it’s possible to recover if the physical storage backing the PVs remains intact.
To recover, you need to set up a new OpenShift cluster and manually reconfigure the PVs to use the existing storage.
34.2. Recovering a deleted Kafka cluster
This procedure describes how to recover a deleted Kafka cluster from persistent volumes (PVs) by recreating the original PersistentVolumeClaim (PVC) resources.
If the Topic Operator and User Operator are deployed, you can recover KafkaTopic and KafkaUser resources by recreating them. It is important that you recreate the KafkaTopic resources with the same configurations, or the Topic Operator will try to update them in Kafka. This procedure shows how to recreate both resources.
If the User Operator is enabled and Kafka users are not recreated, users are deleted from the Kafka cluster immediately after recovery.
Before you begin
In this procedure, it is essential that PVs are mounted into the correct PVC to avoid data corruption. A volumeName is specified for the PVC and this must match the name of the PV.
For more information, see Section 10.4, “Configuring Kafka storage”.
Procedure
Check information on the PVs in the cluster:
oc get pv
oc get pvCopy to Clipboard Copied! Toggle word wrap Toggle overflow Information is presented for PVs with data.
Example PV output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
NAMEis the name of each PV. -
RECLAIMPOLICYshows that PVs are retained, meaning that the PV is not automatically deleted when the PVC is deleted. -
CLAIMshows the link to the original PVCs.
-
Recreate the original namespace:
oc create namespace myproject
oc create namespace myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Here, we recreate the
myprojectnamespace.Recreate the original PVC resource specifications, linking the PVCs to the appropriate PV:
Example PVC resource specification
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the PV specifications to delete the
claimRefproperties that bound the original PVC.Example PV specification
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the example, the following properties are deleted:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Deploy the Cluster Operator:
oc create -f install/cluster-operator -n myproject
oc create -f install/cluster-operator -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Recreate all
KafkaTopicresources by applying theKafkaTopicresource configuration:oc apply -f <topic_configuration_file> -n myproject
oc apply -f <topic_configuration_file> -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Recreate all
KafkaUserresources:If user passwords and certificates need to be retained, recreate the user secrets before recreating the
KafkaUserresources.If the secrets are not recreated, the User Operator will generate new credentials automatically. Ensure that the recreated secrets have exactly the same name, labels, and fields as the original secrets.
Apply the
KafkaUserresource configuration:oc apply -f <user_configuration_file> -n myproject
oc apply -f <user_configuration_file> -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Deploy the Kafka cluster using the original configuration for the
Kafkaresource. Add the annotationstrimzi.io/pause-reconciliation="true"to the original configuration for theKafkaresource, and then deploy the Kafka cluster using the updated configuration.oc apply -f <kafka_resource_configuration>.yaml -n myproject
oc apply -f <kafka_resource_configuration>.yaml -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Recover the original
clusterIdfrom logs or copies of theKafkacustom resource. Otherwise, you can retrieve it from one of the volumes by spinning up a temporary pod.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
Kafkaresource to set the.status.clusterIdwith the recovered value:oc edit kafka <cluster-name> --subresource status -n myproject
oc edit kafka <cluster-name> --subresource status -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Unpause the
Kafkaresource reconciliation:oc annotate kafka my-cluster strimzi.io/pause-reconciliation=false \ --overwrite -n myproject
oc annotate kafka my-cluster strimzi.io/pause-reconciliation=false \ --overwrite -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify the recovery of the
KafkaTopicresources:oc get kafkatopics -o wide -w -n myproject
oc get kafkatopics -o wide -w -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Kafka topic status
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-1 my-cluster 10 3 True my-topic-2 my-cluster 10 3 True my-topic-3 my-cluster 10 3 True
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY my-topic-1 my-cluster 10 3 True my-topic-2 my-cluster 10 3 True my-topic-3 my-cluster 10 3 TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow KafkaTopiccustom resource creation is successful when theREADYoutput showsTrue.Verify the recovery of the
KafkaUserresources:oc get kafkausers -o wide -w -n myproject
oc get kafkausers -o wide -w -n myprojectCopy to Clipboard Copied! Toggle word wrap Toggle overflow Kafka user status
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-1 my-cluster tls simple True my-user-2 my-cluster tls simple True my-user-3 my-cluster tls simple True
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY my-user-1 my-cluster tls simple True my-user-2 my-cluster tls simple True my-user-3 my-cluster tls simple TrueCopy to Clipboard Copied! Toggle word wrap Toggle overflow KafkaUsercustom resource creation is successful when theREADYoutput showsTrue.
Chapter 35. Using Metering on Streams for Apache Kafka
You can use the Metering tool that is available on OpenShift to generate metering reports from different data sources. As a cluster administrator, you can use metering to analyze what is happening in your cluster. You can either write your own, or use predefined SQL queries to define how you want to process data from the different data sources you have available. Using Prometheus as a default data source, you can generate reports on pods, namespaces, and most other OpenShift resources.
You can also use the OpenShift Metering operator to analyze your installed Streams for Apache Kafka components to determine whether you are in compliance with your Red Hat subscription.
To use metering with Streams for Apache Kafka, you must first install and configure the Metering operator on OpenShift Container Platform.
35.1. Metering resources
Metering has many resources which can be used to manage the deployment and installation of metering, as well as the reporting functionality metering provides. Metering is managed using the following CRDs:
| Name | Description |
|---|---|
|
| Configures the metering stack for deployment. Contains customizations and configuration options to control each component that makes up the metering stack. |
|
| Controls what query to use, when, and how often the query should be run, and where to store the results. |
|
|
Contains the SQL queries used to perform analysis on the data contained within |
|
| Controls the data available to ReportQueries and Reports. Allows configuring access to different databases for use within metering. |
35.2. Metering labels for Streams for Apache Kafka
The following table lists the metering labels for Streams for Apache Kafka infrastructure components and integrations.
| Label | Possible values |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Infrastructure
|
| Application
| |
|
|
|
Examples
Infrastructure example (where the infrastructure component is entity-operator)
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Application example (where the integration deployment name is kafka-bridge)
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Appendix A. Using your subscription
Streams for Apache Kafka is provided through a software subscription. To manage your subscriptions, access your account at the Red Hat Customer Portal.
A.1. Accessing Your Account
- Go to access.redhat.com.
- If you do not already have an account, create one.
- Log in to your account.
A.2. Activating a Subscription
- Go to access.redhat.com.
- Navigate to My Subscriptions.
- Navigate to Activate a subscription and enter your 16-digit activation number.
A.3. Downloading Zip and Tar Files
To access zip or tar files, use the customer portal to find the relevant files for download. If you are using RPM packages, this step is not required.
- Open a browser and log in to the Red Hat Customer Portal Product Downloads page at access.redhat.com/downloads.
- Locate the Streams for Apache Kafka entries in the INTEGRATION AND AUTOMATION category.
- Select the desired Streams for Apache Kafka product. The Software Downloads page opens.
- Click the Download link for your component.
A.4. Installing packages with DNF
To install a package and all the package dependencies, use:
dnf install <package_name>
dnf install <package_name>To install a previously-downloaded package from a local directory, use:
dnf install <path_to_download_package>
dnf install <path_to_download_package>Revised on 2026-02-10 16:11:07 UTC
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}