Chapter 1. Deployment overview
Streams for Apache Kafka simplifies the process of running Apache Kafka in an OpenShift cluster.
This guide provides instructions for deploying and managing Streams for Apache Kafka. Deployment options and steps are covered using the example installation files included with Streams for Apache Kafka. While the guide highlights important configuration considerations, it does not cover all available options. For a deeper understanding of the Kafka component configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.
In addition to deployment instructions, the guide offers pre- and post-deployment guidance. It covers setting up and securing client access to your Kafka cluster. Furthermore, it explores additional deployment options such as metrics integration, distributed tracing, and cluster management tools like Cruise Control and the Streams for Apache Kafka Drain Cleaner. You’ll also find recommendations on managing Streams for Apache Kafka and fine-tuning Kafka configuration for optimal performance.
Upgrade instructions are provided for both Streams for Apache Kafka and Kafka, to help keep your deployment up to date.
Streams for Apache Kafka is designed to be compatible with all types of OpenShift clusters, irrespective of their distribution. Whether your deployment involves public or private clouds, or if you are setting up a local development environment, the instructions in this guide are applicable in all cases.
1.1. Streams for Apache Kafka custom resources
The deployment of Kafka components onto an OpenShift cluster using Streams for Apache Kafka is highly configurable through the use of custom resources. These resources are created as instances of APIs introduced by Custom Resource Definitions (CRDs), which extend OpenShift resources.
CRDs act as configuration instructions to describe the custom resources in an OpenShift cluster, and are provided with Streams for Apache Kafka for each Kafka component used in a deployment, as well as users and topics. CRDs and custom resources are defined as YAML files. Example YAML files are provided with the Streams for Apache Kafka distribution.
CRDs also allow Streams for Apache Kafka resources to benefit from native OpenShift features like CLI accessibility and configuration validation.
1.1.1. Streams for Apache Kafka custom resource example
CRDs require a one-time installation in a cluster to define the schemas used to instantiate and manage Streams for Apache Kafka-specific resources.
After a new custom resource type is added to your cluster by installing a CRD, you can create instances of the resource based on its specification.
Depending on the cluster setup, installation typically requires cluster admin privileges.
Access to manage custom resources is limited to Streams for Apache Kafka administrators. For more information, see Section 4.6, “Designating Streams for Apache Kafka administrators”.
A CRD defines a new kind
of resource, such as kind:Kafka
, within an OpenShift cluster.
The Kubernetes API server allows custom resources to be created based on the kind
and understands from the CRD how to validate and store the custom resource when it is added to the OpenShift cluster.
Each Streams for Apache Kafka-specific custom resource conforms to the schema defined by the CRD for the resource’s kind
. The custom resources for Streams for Apache Kafka components have common configuration properties, which are defined under spec
.
To understand the relationship between a CRD and a custom resource, let’s look at a sample of the CRD for a Kafka topic.
Kafka topic CRD
apiVersion: kafka.strimzi.io/v1beta2 kind: CustomResourceDefinition metadata: 1 name: kafkatopics.kafka.strimzi.io labels: app: strimzi spec: 2 group: kafka.strimzi.io versions: v1beta2 scope: Namespaced names: # ... singular: kafkatopic plural: kafkatopics shortNames: - kt 3 additionalPrinterColumns: 4 # ... subresources: status: {} 5 validation: 6 openAPIV3Schema: properties: spec: type: object properties: partitions: type: integer minimum: 1 replicas: type: integer minimum: 1 maximum: 32767 # ...
- 1
- The metadata for the topic CRD, its name and a label to identify the CRD.
- 2
- The specification for this CRD, including the group (domain) name, the plural name and the supported schema version, which are used in the URL to access the API of the topic. The other names are used to identify instance resources in the CLI. For example,
oc get kafkatopic my-topic
oroc get kafkatopics
. - 3
- The shortname can be used in CLI commands. For example,
oc get kt
can be used as an abbreviation instead ofoc get kafkatopic
. - 4
- The information presented when using a
get
command on the custom resource. - 5
- The current status of the CRD as described in the schema reference for the resource.
- 6
- openAPIV3Schema validation provides validation for the creation of topic custom resources. For example, a topic requires at least one partition and one replica.
You can identify the CRD YAML files supplied with the Streams for Apache Kafka installation files, because the file names contain an index number followed by ‘Crd’.
Here is a corresponding example of a KafkaTopic
custom resource.
Kafka topic custom resource
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic 1 metadata: name: my-topic labels: strimzi.io/cluster: my-cluster 2 spec: 3 partitions: 1 replicas: 1 config: retention.ms: 7200000 segment.bytes: 1073741824 status: conditions: 4 lastTransitionTime: "2019-08-20T11:37:00.706Z" status: "True" type: Ready observedGeneration: 1 / ...
- 1
- The
kind
andapiVersion
identify the CRD of which the custom resource is an instance. - 2
- A label, applicable only to
KafkaTopic
andKafkaUser
resources, that defines the name of the Kafka cluster (which is same as the name of theKafka
resource) to which a topic or user belongs. - 3
- The spec shows the number of partitions and replicas for the topic as well as the configuration parameters for the topic itself. In this example, the retention period for a message to remain in the topic and the segment file size for the log are specified.
- 4
- Status conditions for the
KafkaTopic
resource. Thetype
condition changed toReady
at thelastTransitionTime
.
Custom resources can be applied to a cluster through the platform CLI. When the custom resource is created, it uses the same validation as the built-in resources of the Kubernetes API.
After a KafkaTopic
custom resource is created, the Topic Operator is notified and corresponding Kafka topics are created in Streams for Apache Kafka.
1.1.2. Performing oc
operations on custom resources
You can use oc
commands to retrieve information and perform other operations on Streams for Apache Kafka custom resources. Use oc
commands, such as get
, describe
, edit
, or delete
, to perform operations on resource types. For example, oc get kafkatopics
retrieves a list of all Kafka topics and oc get kafkas
retrieves all deployed Kafka clusters.
When referencing resource types, you can use both singular and plural names: oc get kafkas
gets the same results as oc get kafka
.
You can also use the short name of the resource. Learning short names can save you time when managing Streams for Apache Kafka. The short name for Kafka
is k
, so you can also run oc get k
to list all Kafka clusters.
Listing Kafka clusters
oc get k NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS my-cluster 3 3
Streams for Apache Kafka resource | Long name | Short name |
---|---|---|
Kafka | kafka | k |
Kafka Node Pool | kafkanodepool | knp |
Kafka Topic | kafkatopic | kt |
Kafka User | kafkauser | ku |
Kafka Connect | kafkaconnect | kc |
Kafka Connector | kafkaconnector | kctr |
Kafka Mirror Maker | kafkamirrormaker | kmm |
Kafka Mirror Maker 2 | kafkamirrormaker2 | kmm2 |
Kafka Bridge | kafkabridge | kb |
Kafka Rebalance | kafkarebalance | kr |
1.1.2.1. Resource categories
Categories of custom resources can also be used in oc
commands.
All Streams for Apache Kafka custom resources belong to the category strimzi
, so you can use strimzi
to get all the Streams for Apache Kafka resources with one command.
For example, running oc get strimzi
lists all Streams for Apache Kafka custom resources in a given namespace.
Listing all custom resources
oc get strimzi NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS kafka.kafka.strimzi.io/my-cluster 3 3 NAME PARTITIONS REPLICATION FACTOR kafkatopic.kafka.strimzi.io/kafka-apps 3 3 NAME AUTHENTICATION AUTHORIZATION kafkauser.kafka.strimzi.io/my-user tls simple
The oc get strimzi -o name
command returns all resource types and resource names. The -o name
option fetches the output in the type/name format
Listing all resource types and names
oc get strimzi -o name kafka.kafka.strimzi.io/my-cluster kafkatopic.kafka.strimzi.io/kafka-apps kafkauser.kafka.strimzi.io/my-user
You can combine this strimzi
command with other commands. For example, you can pass it into a oc delete
command to delete all resources in a single command.
Deleting all custom resources
oc delete $(oc get strimzi -o name) kafka.kafka.strimzi.io "my-cluster" deleted kafkatopic.kafka.strimzi.io "kafka-apps" deleted kafkauser.kafka.strimzi.io "my-user" deleted
Deleting all resources in a single operation might be useful, for example, when you are testing new Streams for Apache Kafka features.
1.1.2.2. Querying the status of sub-resources
There are other values you can pass to the -o
option. For example, by using -o yaml
you get the output in YAML format. Using -o json
will return it as JSON.
You can see all the options in oc get --help
.
One of the most useful options is the JSONPath support, which allows you to pass JSONPath expressions to query the Kubernetes API. A JSONPath expression can extract or navigate specific parts of any resource.
For example, you can use the JSONPath expression {.status.listeners[?(@.name=="tls")].bootstrapServers}
to get the bootstrap address from the status of the Kafka custom resource and use it in your Kafka clients.
Here, the command retrieves the bootstrapServers
value of the listener named tls
:
Retrieving the bootstrap address
oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="tls")].bootstrapServers}{"\n"}' my-cluster-kafka-bootstrap.myproject.svc:9093
By changing the name condition you can also get the address of the other Kafka listeners.
You can use jsonpath
to extract any other property or group of properties from any custom resource.
1.1.3. Streams for Apache Kafka custom resource status information
Status properties provide status information for certain custom resources.
The following table lists the custom resources that provide status information (when deployed) and the schemas that define the status properties.
For more information on the schemas, see the Streams for Apache Kafka Custom Resource API Reference.
Streams for Apache Kafka resource | Schema reference | Publishes status information on… |
---|---|---|
|
| The Kafka cluster, its listeners, and node pools |
|
| The nodes in the node pool, their roles, and the associated Kafka cluster |
|
| Kafka topics in the Kafka cluster |
|
| Kafka users in the Kafka cluster |
|
| The Kafka Connect cluster and connector plugins |
|
|
|
|
| The Kafka MirrorMaker 2 cluster and internal connectors |
|
| The Kafka MirrorMaker cluster |
|
| The Streams for Apache Kafka Bridge |
|
| The status and results of a rebalance |
|
| The number of pods: being managed, using the current version, and in a ready state |
The status
property of a resource provides information on the state of the resource. The status.conditions
and status.observedGeneration
properties are common to all resources.
status.conditions
-
Status conditions describe the current state of a resource. Status condition properties are useful for tracking progress related to the resource achieving its desired state, as defined by the configuration specified in its
spec
. Status condition properties provide the time and reason the state of the resource changed, and details of events preventing or delaying the operator from realizing the desired state. status.observedGeneration
-
Last observed generation denotes the latest reconciliation of the resource by the Cluster Operator. If the value of
observedGeneration
is different from the value ofmetadata.generation
(the current version of the deployment), the operator has not yet processed the latest update to the resource. If these values are the same, the status information reflects the most recent changes to the resource.
The status
properties also provide resource-specific information. For example, KafkaStatus
provides information on listener addresses, and the ID of the Kafka cluster.
KafkaStatus
also provides information on the Kafka and Streams for Apache Kafka versions being used. You can check the values of operatorLastSuccessfulVersion
and kafkaVersion
to determine whether an upgrade of Streams for Apache Kafka or Kafka has completed
Streams for Apache Kafka creates and maintains the status of custom resources, periodically evaluating the current state of the custom resource and updating its status accordingly. When performing an update on a custom resource using oc edit
, for example, its status
is not editable. Moreover, changing the status
would not affect the configuration of the Kafka cluster.
Here we see the status
properties for a Kafka
custom resource.
Kafka custom resource status
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: spec: # ... status: clusterId: XP9FP2P-RByvEy0W4cOEUA 1 conditions: 2 - lastTransitionTime: '2023-01-20T17:56:29.396588Z' status: 'True' type: Ready 3 kafkaMetadataState: KRaft 4 kafkaVersion: 3.7.0 5 kafkaNodePools: 6 - name: broker - name: controller listeners: 7 - addresses: - host: my-cluster-kafka-bootstrap.prm-project.svc port: 9092 bootstrapServers: 'my-cluster-kafka-bootstrap.prm-project.svc:9092' name: plain - addresses: - host: my-cluster-kafka-bootstrap.prm-project.svc port: 9093 bootstrapServers: 'my-cluster-kafka-bootstrap.prm-project.svc:9093' certificates: - | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- name: tls - addresses: - host: >- 2054284155.us-east-2.elb.amazonaws.com port: 9095 bootstrapServers: >- 2054284155.us-east-2.elb.amazonaws.com:9095 certificates: - | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- name: external3 - addresses: - host: ip-10-0-172-202.us-east-2.compute.internal port: 31644 bootstrapServers: 'ip-10-0-172-202.us-east-2.compute.internal:31644' certificates: - | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- name: external4 observedGeneration: 3 8 operatorLastSuccessfulVersion: 2.7 9
- 1
- The Kafka cluster ID.
- 2
- Status
conditions
describe the current state of the Kafka cluster. - 3
- The
Ready
condition indicates that the Cluster Operator considers the Kafka cluster able to handle traffic. - 4
- Kafka metadata state that shows the mechanism used (KRaft or ZooKeeper) to manage Kafka metadata and coordinate operations.
- 5
- The version of Kafka being used by the Kafka cluster.
- 6
- The node pools belonging to the Kafka cluster.
- 7
- The
listeners
describe Kafka bootstrap addresses by type. - 8
- The
observedGeneration
value indicates the last reconciliation of theKafka
custom resource by the Cluster Operator. - 9
- The version of the operator that successfully completed the last reconciliation.
The Kafka bootstrap addresses listed in the status do not signify that those endpoints or the Kafka cluster is in a Ready
state.
1.1.4. Finding the status of a custom resource
Use oc
with the status
subresource of a custom resource to retrieve information about the resource.
Prerequisites
- An OpenShift cluster.
- The Cluster Operator is running.
Procedure
Specify the custom resource and use the
-o jsonpath
option to apply a standard JSONPath expression to select thestatus
property:oc get kafka <kafka_resource_name> -o jsonpath='{.status}' | jq
This expression returns all the status information for the specified custom resource. You can use dot notation, such as
status.listeners
orstatus.observedGeneration
, to fine-tune the status information you wish to see.Using the
jq
command line JSON parser tool makes it easier to read the output.
Additional resources
- For more information about using JSONPath, see JSONPath support.
1.2. Streams for Apache Kafka operators
Streams for Apache Kafka operators are purpose-built with specialist operational knowledge to effectively manage Kafka on OpenShift. Each operator performs a distinct function.
- Cluster Operator
- The Cluster Operator handles the deployment and management of Apache Kafka clusters on OpenShift. It automates the setup of Kafka brokers, and other Kafka components and resources.
- Topic Operator
- The Topic Operator manages the creation, configuration, and deletion of topics within Kafka clusters.
- User Operator
- The User Operator manages Kafka users that require access to Kafka brokers.
When you deploy Streams for Apache Kafka, you first deploy the Cluster Operator. The Cluster Operator is then ready to handle the deployment of Kafka. You can also deploy the Topic Operator and User Operator using the Cluster Operator (recommended) or as standalone operators. You would use a standalone operator with a Kafka cluster that is not managed by the Cluster Operator.
The Topic Operator and User Operator are part of the Entity Operator. The Cluster Operator can deploy one or both operators based on the Entity Operator configuration.
To deploy the standalone operators, you need to set environment variables to connect to a Kafka cluster. These environment variables do not need to be set if you are deploying the operators using the Cluster Operator as they will be set by the Cluster Operator.
1.2.1. Watching Streams for Apache Kafka resources in OpenShift namespaces
Operators watch and manage Streams for Apache Kafka resources in OpenShift namespaces. The Cluster Operator can watch a single namespace, multiple namespaces, or all namespaces in an OpenShift cluster. The Topic Operator and User Operator can watch a single namespace.
-
The Cluster Operator watches for
Kafka
resources -
The Topic Operator watches for
KafkaTopic
resources -
The User Operator watches for
KafkaUser
resources
The Topic Operator and the User Operator can only watch a single Kafka cluster in a namespace. And they can only be connected to a single Kafka cluster.
If multiple Topic Operators watch the same namespace, name collisions and topic deletion can occur. This is because each Kafka cluster uses Kafka topics that have the same name (such as __consumer_offsets
). Make sure that only one Topic Operator watches a given namespace.
When using multiple User Operators with a single namespace, a user with a given username can exist in more than one Kafka cluster.
If you deploy the Topic Operator and User Operator using the Cluster Operator, they watch the Kafka cluster deployed by the Cluster Operator by default. You can also specify a namespace using watchedNamespace
in the operator configuration.
For a standalone deployment of each operator, you specify a namespace and connection to the Kafka cluster to watch in the configuration.
1.2.2. Managing RBAC resources
The Cluster Operator creates and manages role-based access control (RBAC) resources for Streams for Apache Kafka components that need access to OpenShift resources.
For the Cluster Operator to function, it needs permission within the OpenShift cluster to interact with Kafka resources, such as Kafka
and KafkaConnect
, as well as managed resources like ConfigMap
, Pod
, Deployment
, and Service
.
Permission is specified through the following OpenShift RBAC resources:
-
ServiceAccount
-
Role
andClusterRole
-
RoleBinding
andClusterRoleBinding
1.2.2.1. Delegating privileges to Streams for Apache Kafka components
The Cluster Operator runs under a service account called strimzi-cluster-operator
. It is assigned cluster roles that give it permission to create the RBAC resources for Streams for Apache Kafka components. Role bindings associate the cluster roles with the service account.
OpenShift prevents components operating under one ServiceAccount
from granting another ServiceAccount
privileges that the granting ServiceAccount
does not have. Because the Cluster Operator creates the RoleBinding
and ClusterRoleBinding
RBAC resources needed by the resources it manages, it requires a role that gives it the same privileges.
The following sections describe the RBAC resources required by the Cluster Operator.
1.2.2.2. ClusterRole
resources
The Cluster Operator uses ClusterRole
resources to provide the necessary access to resources. Depending on the OpenShift cluster setup, a cluster administrator might be needed to create the cluster roles.
Cluster administrator rights are only needed for the creation of ClusterRole
resources. The Cluster Operator will not run under a cluster admin account.
The RBAC resources follow the principle of least privilege and contain only those privileges needed by the Cluster Operator to operate the cluster of the Kafka component.
All cluster roles are required by the Cluster Operator in order to delegate privileges.
Name | Description |
---|---|
| Access rights for namespace-scoped resources used by the Cluster Operator to deploy and manage the operands. |
| Access rights for cluster-scoped resources used by the Cluster Operator to deploy and manage the operands. |
| Access rights used by the Cluster Operator for leader election. |
| Access rights used by the Cluster Operator to watch and manage the Streams for Apache Kafka custom resources. |
| Access rights to allow Kafka brokers to get the topology labels from OpenShift worker nodes when rack-awareness is used. |
| Access rights used by the Topic and User Operators to manage Kafka users and topics. |
| Access rights to allow Kafka Connect, MirrorMaker (1 and 2), and Kafka Bridge to get the topology labels from OpenShift worker nodes when rack-awareness is used. |
1.2.2.3. ClusterRoleBinding
resources
The Cluster Operator uses ClusterRoleBinding
and RoleBinding
resources to associate its ClusterRole
with its ServiceAccount
. Cluster role bindings are required by cluster roles containing cluster-scoped resources.
Name | Description |
---|---|
|
Grants the Cluster Operator the rights from the |
|
Grants the Cluster Operator the rights from the |
|
Grants the Cluster Operator the rights from the |
Name | Description |
---|---|
|
Grants the Cluster Operator the rights from the |
|
Grants the Cluster Operator the rights from the |
|
Grants the Cluster Operator the rights from the |
|
Grants the Cluster Operator the rights from the |
1.2.2.4. ServiceAccount
resources
The Cluster Operator runs using the strimzi-cluster-operator
ServiceAccount
. This service account grants it the privileges it requires to manage the operands. The Cluster Operator creates additional ClusterRoleBinding
and RoleBinding
resources to delegate some of these RBAC rights to the operands.
Each of the operands uses its own service account created by the Cluster Operator. This allows the Cluster Operator to follow the principle of least privilege and give the operands only the access rights that are really need.
Name | Used by |
---|---|
| ZooKeeper pods |
| Kafka broker pods |
| Entity Operator |
| Cruise Control pods |
| Kafka Exporter pods |
| Kafka Connect pods |
| MirrorMaker pods |
| MirrorMaker 2 pods |
| Kafka Bridge pods |
1.2.3. Managing pod resources
The StrimziPodSet
custom resource is used by Streams for Apache Kafka to create and manage Kafka, Kafka Connect, and MirrorMaker 2 pods. If you are using ZooKeeper, ZooKeeper pods are also created and managed using StrimziPodSet
resources.
You must not create, update, or delete StrimziPodSet
resources. The StrimziPodSet
custom resource is used internally and resources are managed solely by the Cluster Operator. As a consequence, the Cluster Operator must be running properly to avoid the possibility of pods not starting and Kafka clusters not being available.
OpenShift Deployment
resources are used for creating and managing the pods of other components: Kafka Bridge, Kafka Exporter, Cruise Control, (deprecated) MirrorMaker 1, User Operator and Topic Operator.
1.3. Using the Kafka Bridge to connect with a Kafka cluster
You can use the Streams for Apache Kafka Bridge API to create and manage consumers and send and receive records over HTTP rather than the native Kafka protocol.
When you set up the Kafka Bridge you configure HTTP access to the Kafka cluster. You can then use the Kafka Bridge to produce and consume messages from the cluster, as well as performing other operations through its REST interface.
Additional resources
- For information on installing and using the Kafka Bridge, see Using the Streams for Apache Kafka Bridge.
1.4. Seamless FIPS support
Federal Information Processing Standards (FIPS) are standards for computer security and interoperability. When running Streams for Apache Kafka on a FIPS-enabled OpenShift cluster, the OpenJDK used in Streams for Apache Kafka container images automatically switches to FIPS mode. From version 2.3, Streams for Apache Kafka can run on FIPS-enabled OpenShift clusters without any changes or special configuration. It uses only the FIPS-compliant security libraries from the OpenJDK.
If you are using FIPS-enabled OpenShift clusters, you may experience higher memory consumption compared to regular OpenShift clusters. To avoid any issues, we suggest increasing the memory request to at least 512Mi.
For more information about the NIST validation program and validated modules, see Cryptographic Module Validation Program on the NIST website.
Compatibility with the technology previews of Streams for Apache Kafka Proxy and Streams for Apache Kafka Console has not been tested regarding FIPS support. While they are expected to function properly, we cannot guarantee full support at this time.
1.4.1. Minimum password length
When running in the FIPS mode, SCRAM-SHA-512 passwords need to be at least 32 characters long. From Streams for Apache Kafka 2.3, the default password length in Streams for Apache Kafka User Operator is set to 32 characters as well. If you have a Kafka cluster with custom configuration that uses a password length that is less than 32 characters, you need to update your configuration. If you have any users with passwords shorter than 32 characters, you need to regenerate a password with the required length. You can do that, for example, by deleting the user secret and waiting for the User Operator to create a new password with the appropriate length.
1.5. Document Conventions
User-replaced values
User-replaced values, also known as replaceables, are shown in with angle brackets (< >). Underscores ( _ ) are used for multi-word values. If the value refers to code or commands, monospace
is also used.
For example, the following code shows that <my_namespace>
must be replaced by the correct namespace name:
sed -i 's/namespace: .*/namespace: <my_namespace>/' install/cluster-operator/*RoleBinding*.yaml