ホーム
製品
Streams for Apache Kafka
2.8
Deploying and Managing Streams for Apache Kafka on OpenShift
7.3. Deploying Kafka

7.3. Deploying Kafka

To be able to manage a Kafka cluster with the Cluster Operator, you must deploy it as a Kafka resource. Streams for Apache Kafka provides example deployment files to do this. You can use these files to deploy the Topic Operator and User Operator at the same time.

After you have deployed the Cluster Operator, use a Kafka resource to deploy the following components:

A Kafka cluster that uses KRaft or ZooKeeper:
- KRaft-based Kafka cluster
- ZooKeeper-based Kafka cluster
Topic Operator
User Operator

Node pools are used in the deployment of a Kafka cluster in KRaft (Kafka Raft metadata) mode, and may be used for the deployment of a Kafka cluster with ZooKeeper. Node pools represent a distinct group of Kafka nodes within the Kafka cluster that share the same configuration. For each Kafka node in the node pool, any configuration not defined in node pool is inherited from the cluster configuration in the Kafka resource.

If you haven’t deployed a Kafka cluster as a Kafka resource, you can’t use the Cluster Operator to manage it. This applies, for example, to a Kafka cluster running outside of OpenShift. However, you can use the Topic Operator and User Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, by deploying them as standalone components. You can also deploy and use other Kafka components with a Kafka cluster not managed by Streams for Apache Kafka.

7.3.1. Deploying a Kafka cluster in KRaft mode
リンクのコピー

This procedure shows how to deploy a Kafka cluster in KRaft mode and associated node pools using the Cluster Operator.

The deployment uses a YAML file to provide the specification to create a Kafka resource and KafkaNodePool resources.

Streams for Apache Kafka provides the following example deployment files that you can use to create a Kafka cluster that uses node pools:

kafka/kraft/kafka-with-dual-role-nodes.yaml: Deploys a Kafka cluster with one pool of nodes that share the broker and controller roles.
kafka/kraft/kafka.yaml: Deploys a persistent Kafka cluster with one pool of controller nodes and one pool of broker nodes.
kafka/kraft/kafka-ephemeral.yaml: Deploys an ephemeral Kafka cluster with one pool of controller nodes and one pool of broker nodes.
kafka/kraft/kafka-single-node.yaml: Deploys a Kafka cluster with a single node.
kafka/kraft/kafka-jbod.yaml: Deploys a Kafka cluster with multiple volumes in each broker node.

In this procedure, we use the example deployment file that deploys a Kafka cluster with one pool of nodes that share the broker and controller roles.

The Kafka resource configuration for each example includes the strimzi.io/node-pools: enabled annotation, which is required when using node pools. Kafka resources using KRaft mode must also have the annotation strimzi.io/kraft: enabled.

The example YAML files specify the latest supported Kafka version and KRaft metadata version used by the Kafka cluster.

注記

You can perform the steps outlined here to deploy a new Kafka cluster with KafkaNodePool resources or migrate your existing Kafka cluster.

Prerequisites

The Cluster Operator must be deployed.

Before you begin

By default, the example deployment files specify my-cluster as the Kafka cluster name. The name cannot be changed after the cluster has been deployed. To change the cluster name before you deploy the cluster, edit the Kafka.metadata.name property of the Kafka resource in the relevant YAML file.

Procedure

Deploy a KRaft-based Kafka cluster.
To deploy a Kafka cluster with a single node pool that uses dual-role nodes:
```
oc apply -f examples/kafka/kraft/kafka-with-dual-role-nodes.yaml
```
Check the status of the deployment:
```
oc get pods -n <my_cluster_operator_namespace>
```
Output shows the node pool names and readiness
```
NAME                        READY  STATUS   RESTARTS
my-cluster-entity-operator  3/3    Running  0
my-cluster-pool-a-0         1/1    Running  0
my-cluster-pool-a-1         1/1    Running  0
my-cluster-pool-a-4         1/1    Running  0
```
- my-cluster is the name of the Kafka cluster.
- pool-a is the name of the node pool.
  A sequential index number starting with 0 identifies each Kafka pod created. If you are using ZooKeeper, you’ll also see the ZooKeeper pods.
  READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.
  Information on the deployment is also shown in the status of the KafkaNodePool resource, including a list of IDs for nodes in the pool.
  注記
  Node IDs are assigned sequentially starting at 0 (zero) across all node pools within a cluster. This means that node IDs might not run sequentially within a specific node pool. If there are gaps in the sequence of node IDs across the cluster, the next node to be added is assigned an ID that fills the gap. When scaling down, the node with the highest node ID within a pool is removed.

7.3.2. Deploying a ZooKeeper-based Kafka cluster
リンクのコピー

This procedure shows how to deploy a ZooKeeper-based Kafka cluster to your OpenShift cluster using the Cluster Operator.

The deployment uses a YAML file to provide the specification to create a Kafka resource.

Streams for Apache Kafka provides the following example deployment files to create a Kafka cluster that uses ZooKeeper for cluster management:

kafka-persistent.yaml: Deploys a persistent cluster with three ZooKeeper and three Kafka nodes.
kafka-jbod.yaml: Deploys a persistent cluster with three ZooKeeper and three Kafka nodes (each using multiple persistent volumes).
kafka-persistent-single.yaml: Deploys a persistent cluster with a single ZooKeeper node and a single Kafka node.
kafka-ephemeral.yaml: Deploys an ephemeral cluster with three ZooKeeper and three Kafka nodes.
kafka-ephemeral-single.yaml: Deploys an ephemeral cluster with three ZooKeeper nodes and a single Kafka node.

To deploy a Kafka cluster that uses node pools, the following example YAML file provides the specification to create a Kafka resource and KafkaNodePool resources:

kafka/kafka-with-node-pools.yaml: Deploys ZooKeeper with 3 nodes, and 2 different pools of Kafka brokers. Each of the pools has 3 brokers. The pools in the example use different storage configuration.

In this procedure, we use the examples for an ephemeral and persistent Kafka cluster deployment.

The example YAML files specify the latest supported Kafka version and inter-broker protocol version.

注記

From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set.

Prerequisites

The Cluster Operator must be deployed.

Before you begin

Procedure

Deploy a ZooKeeper-based Kafka cluster.
- To deploy an ephemeral cluster:
  oc apply -f examples/kafka/kafka-ephemeral.yaml
- To deploy a persistent cluster:
  oc apply -f examples/kafka/kafka-persistent.yaml

Check the status of the deployment:

oc get pods -n <my_cluster_operator_namespace>

Output shows the pod names and readiness

NAME                        READY   STATUS    RESTARTS
my-cluster-entity-operator  3/3     Running   0
my-cluster-kafka-0          1/1     Running   0
my-cluster-kafka-1          1/1     Running   0
my-cluster-kafka-2          1/1     Running   0
my-cluster-zookeeper-0      1/1     Running   0
my-cluster-zookeeper-1      1/1     Running   0
my-cluster-zookeeper-2      1/1     Running   0

my-cluster is the name of the Kafka cluster.

A sequential index number starting with 0 identifies each Kafka and ZooKeeper pod created.

With the default deployment, you create an Entity Operator cluster, 3 Kafka pods, and 3 ZooKeeper pods.

READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

7.3.3. Deploying the Topic Operator using the Cluster Operator
リンクのコピー

This procedure describes how to deploy the Topic Operator using the Cluster Operator.

You configure the entityOperator property of the Kafka resource to include the topicOperator. By default, the Topic Operator watches for KafkaTopic resources in the namespace of the Kafka cluster deployed by the Cluster Operator. You can also specify a namespace using watchedNamespace in the Topic Operator spec. A single Topic Operator can watch a single namespace. One namespace should be watched by only one Topic Operator.

If you use Streams for Apache Kafka to deploy multiple Kafka clusters into the same namespace, enable the Topic Operator for only one Kafka cluster or use the watchedNamespace property to configure the Topic Operators to watch other namespaces.

If you want to use the Topic Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, you must deploy the Topic Operator as a standalone component.

For more information about configuring the entityOperator and topicOperator properties, see Configuring the Entity Operator.

Prerequisites

The Cluster Operator must be deployed.

Procedure

Edit the entityOperator properties of the Kafka resource to include topicOperator:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  #...
  entityOperator:
    topicOperator: {}
    userOperator: {}

Configure the Topic Operator spec using the properties described in the EntityTopicOperatorSpec schema reference.
Use an empty object ({}) if you want all properties to use their default values.
Create or update the resource:
```
oc apply -f <kafka_configuration_file>
```
Check the status of the deployment:
```
oc get pods -n <my_cluster_operator_namespace>
```
Output shows the pod name and readiness
```
NAME                        READY   STATUS    RESTARTS
my-cluster-entity-operator  3/3     Running   0
# ...
```
my-cluster is the name of the Kafka cluster.
READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

7.3.4. Deploying the User Operator using the Cluster Operator
リンクのコピー

This procedure describes how to deploy the User Operator using the Cluster Operator.

You configure the entityOperator property of the Kafka resource to include the userOperator. By default, the User Operator watches for KafkaUser resources in the namespace of the Kafka cluster deployment. You can also specify a namespace using watchedNamespace in the User Operator spec. A single User Operator can watch a single namespace. One namespace should be watched by only one User Operator.

If you want to use the User Operator with a Kafka cluster that is not managed by Streams for Apache Kafka, you must deploy the User Operator as a standalone component.

For more information about configuring the entityOperator and userOperator properties, see Configuring the Entity Operator.

Prerequisites

The Cluster Operator must be deployed.

Procedure

Edit the entityOperator properties of the Kafka resource to include userOperator:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  #...
  entityOperator:
    topicOperator: {}
    userOperator: {}

Configure the User Operator spec using the properties described in EntityUserOperatorSpec schema reference.
Use an empty object ({}) if you want all properties to use their default values.
Create or update the resource:
```
oc apply -f <kafka_configuration_file>
```
Check the status of the deployment:
```
oc get pods -n <my_cluster_operator_namespace>
```
Output shows the pod name and readiness
```
NAME                        READY   STATUS    RESTARTS
my-cluster-entity-operator  3/3     Running   0
# ...
```
my-cluster is the name of the Kafka cluster.
READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

7.3.5. Connecting to ZooKeeper from a terminal
リンクのコピー

ZooKeeper services are secured with encryption and authentication and are not intended to be used by external applications that are not part of Streams for Apache Kafka.

However, if you want to use CLI tools that require a connection to ZooKeeper, you can use a terminal inside a ZooKeeper pod and connect to localhost:12181 as the ZooKeeper address.

Prerequisites

An OpenShift cluster is available.
A Kafka cluster is running.
The Cluster Operator is running.

Procedure

Open the terminal using the OpenShift console or run the exec command from your CLI.
For example:
```
oc exec -ti my-cluster-zookeeper-0 -- bin/zookeeper-shell.sh localhost:12181 ls /
```
Be sure to use localhost:12181.

7.3.6. List of Kafka cluster resources
リンクのコピー

The following resources are created by the Cluster Operator in the OpenShift cluster.

Shared resources

<kafka_cluster_name>-cluster-ca: Secret with the Cluster CA private key used to encrypt the cluster communication.
<kafka_cluster_name>-cluster-ca-cert: Secret with the Cluster CA public key. This key can be used to verify the identity of the Kafka brokers.
<kafka_cluster_name>-clients-ca: Secret with the Clients CA private key used to sign user certificates
<kafka_cluster_name>-clients-ca-cert: Secret with the Clients CA public key. This key can be used to verify the identity of the Kafka users.
<kafka_cluster_name>-cluster-operator-certs: Secret with Cluster operators keys for communication with Kafka and ZooKeeper.

ZooKeeper nodes

<kafka_cluster_name>-zookeeper

Name given to the following ZooKeeper resources:

StrimziPodSet for managing the ZooKeeper node pods.
Service account used by the ZooKeeper nodes.
PodDisruptionBudget configured for the ZooKeeper nodes.

<kafka_cluster_name>-zookeeper-<pod_id>

Pods created by the StrimziPodSet.

<kafka_cluster_name>-zookeeper-nodes

Headless Service needed to have DNS resolve the ZooKeeper pods IP addresses directly.

<kafka_cluster_name>-zookeeper-client

Service used by Kafka brokers to connect to ZooKeeper nodes as clients.

<kafka_cluster_name>-zookeeper-config

ConfigMap that contains the ZooKeeper ancillary configuration, and is mounted as a volume by the ZooKeeper node pods.

<kafka_cluster_name>-zookeeper-nodes

Secret with ZooKeeper node keys.

<kafka_cluster_name>-network-policy-zookeeper

Network policy managing access to the ZooKeeper services.

data-<kafka_cluster_name>-zookeeper-<pod_id>

Persistent Volume Claim for the volume used for storing data for a specific ZooKeeper node. This resource will be created only if persistent storage is selected for provisioning persistent volumes to store data.

Kafka brokers

<kafka_cluster_name>-kafka

Name given to the following Kafka resources:

StrimziPodSet for managing the Kafka broker pods.
Service account used by the Kafka pods.
PodDisruptionBudget configured for the Kafka brokers.

<kafka_cluster_name>-kafka-<pod_id>

Name given to the following Kafka resources:

Pods created by the StrimziPodSet.
ConfigMaps with Kafka broker configuration.

<kafka_cluster_name>-kafka-brokers

Service needed to have DNS resolve the Kafka broker pods IP addresses directly.

<kafka_cluster_name>-kafka-bootstrap

Service can be used as bootstrap servers for Kafka clients connecting from within the OpenShift cluster.

<kafka_cluster_name>-kafka-external-bootstrap

Bootstrap service for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled. The old service name will be used for backwards compatibility when the listener name is external and port is 9094.

<kafka_cluster_name>-kafka-<pod_id>

Service used to route traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled. The old service name will be used for backwards compatibility when the listener name is external and port is 9094.

<kafka_cluster_name>-kafka-external-bootstrap

Bootstrap route for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled and set to type route. The old route name will be used for backwards compatibility when the listener name is external and port is 9094.

<kafka_cluster_name>-kafka-<pod_id>

Route for traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled and set to type route. The old route name will be used for backwards compatibility when the listener name is external and port is 9094.

<kafka_cluster_name>-kafka-<listener_name>-bootstrap

Bootstrap service for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled. The new service name will be used for all other external listeners.

<kafka_cluster_name>-kafka-<listener_name>-<pod_id>

Service used to route traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled. The new service name will be used for all other external listeners.

<kafka_cluster_name>-kafka-<listener_name>-bootstrap

Bootstrap route for clients connecting from outside the OpenShift cluster. This resource is created only when an external listener is enabled and set to type route. The new route name will be used for all other external listeners.

<kafka_cluster_name>-kafka-<listener_name>-<pod_id>

Route for traffic from outside the OpenShift cluster to individual pods. This resource is created only when an external listener is enabled and set to type route. The new route name will be used for all other external listeners.

<kafka_cluster_name>-kafka-config

ConfigMap containing the Kafka ancillary configuration, which is mounted as a volume by the broker pods when the UseStrimziPodSets feature gate is disabled.

<kafka_cluster_name>-kafka-brokers

Secret with Kafka broker keys.

<kafka_cluster_name>-network-policy-kafka

Network policy managing access to the Kafka services.

strimzi-namespace-name-<kafka_cluster_name>-kafka-init

Cluster role binding used by the Kafka brokers.

<kafka_cluster_name>-jmx

Secret with JMX username and password used to secure the Kafka broker port. This resource is created only when JMX is enabled in Kafka.

data-<kafka_cluster_name>-kafka-<pod_id>

Persistent Volume Claim for the volume used for storing data for a specific Kafka broker. This resource is created only if persistent storage is selected for provisioning persistent volumes to store data.

data-<id>-<kafka_cluster_name>-kafka-<pod_id>

Persistent Volume Claim for the volume id used for storing data for a specific Kafka broker. This resource is created only if persistent storage is selected for JBOD volumes when provisioning persistent volumes to store data.

Kafka node pools

If you are using Kafka node pools, the resources created apply to the nodes managed in the node pools whether they are operating as brokers, controllers, or both. The naming convention includes the name of the Kafka cluster and the node pool: <kafka_cluster_name>-<pool_name>.

<kafka_cluster_name>-<pool_name>

Name given to the StrimziPodSet for managing the Kafka node pool.

<kafka_cluster_name>-<pool_name>-<pod_id>

Name given to the following Kafka node pool resources:

Pods created by the StrimziPodSet.
ConfigMaps with Kafka node configuration.

data-<kafka_cluster_name>-<pool_name>-<pod_id>

Persistent Volume Claim for the volume used for storing data for a specific node. This resource is created only if persistent storage is selected for provisioning persistent volumes to store data.

data-<id>-<kafka_cluster_name>-<pool_name>-<pod_id>

Persistent Volume Claim for the volume id used for storing data for a specific node. This resource is created only if persistent storage is selected for JBOD volumes when provisioning persistent volumes to store data.

Entity Operator

These resources are only created if the Entity Operator is deployed using the Cluster Operator.

<kafka_cluster_name>-entity-operator

Name given to the following Entity Operator resources:

Deployment with Topic and User Operators.
Service account used by the Entity Operator.
Network policy managing access to the Entity Operator metrics.

<kafka_cluster_name>-entity-operator-<random_string>

Pod created by the Entity Operator deployment.

<kafka_cluster_name>-entity-topic-operator-config

ConfigMap with ancillary configuration for Topic Operators.

<kafka_cluster_name>-entity-user-operator-config

ConfigMap with ancillary configuration for User Operators.

<kafka_cluster_name>-entity-topic-operator-certs

Secret with Topic Operator keys for communication with Kafka and ZooKeeper.

<kafka_cluster_name>-entity-user-operator-certs

Secret with User Operator keys for communication with Kafka and ZooKeeper.

strimzi-<kafka_cluster_name>-entity-topic-operator

Role binding used by the Entity Topic Operator.

strimzi-<kafka_cluster_name>-entity-user-operator

Role binding used by the Entity User Operator.

Kafka Exporter

These resources are only created if the Kafka Exporter is deployed using the Cluster Operator.

<kafka_cluster_name>-kafka-exporter

Name given to the following Kafka Exporter resources:

Deployment with Kafka Exporter.
Service used to collect consumer lag metrics.
Service account used by the Kafka Exporter.
Network policy managing access to the Kafka Exporter metrics.

<kafka_cluster_name>-kafka-exporter-<random_string>

Pod created by the Kafka Exporter deployment.

Cruise Control

These resources are only created if Cruise Control was deployed using the Cluster Operator.

<kafka_cluster_name>-cruise-control

Name given to the following Cruise Control resources:

Deployment with Cruise Control.
Service used to communicate with Cruise Control.
Service account used by the Cruise Control.

<kafka_cluster_name>-cruise-control-<random_string>

Pod created by the Cruise Control deployment.

<kafka_cluster_name>-cruise-control-config

ConfigMap that contains the Cruise Control ancillary configuration, and is mounted as a volume by the Cruise Control pods.

<kafka_cluster_name>-cruise-control-certs

Secret with Cruise Control keys for communication with Kafka and ZooKeeper.

<kafka_cluster_name>-network-policy-cruise-control

Network policy managing access to the Cruise Control service.

7.3. Deploying Kafka

7.3.1. Deploying a Kafka cluster in KRaft mode
リンクのコピー

7.3.2. Deploying a ZooKeeper-based Kafka cluster
リンクのコピー

7.3.3. Deploying the Topic Operator using the Cluster Operator
リンクのコピー

7.3.4. Deploying the User Operator using the Cluster Operator
リンクのコピー

7.3.5. Connecting to ZooKeeper from a terminal
リンクのコピー

7.3.6. List of Kafka cluster resources
リンクのコピー

詳細情報

試用、購入および販売

コミュニティー

会社概要

多様性を受け入れるオープンソースの強化

Red Hat ドキュメントについて

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

7.3. Deploying Kafka

7.3.1. Deploying a Kafka cluster in KRaft modeリンクのコピーリンクがクリップボードにコピーされました!

7.3.2. Deploying a ZooKeeper-based Kafka clusterリンクのコピーリンクがクリップボードにコピーされました!

7.3.3. Deploying the Topic Operator using the Cluster Operatorリンクのコピーリンクがクリップボードにコピーされました!

7.3.4. Deploying the User Operator using the Cluster Operatorリンクのコピーリンクがクリップボードにコピーされました!

7.3.5. Connecting to ZooKeeper from a terminalリンクのコピーリンクがクリップボードにコピーされました!

7.3.6. List of Kafka cluster resourcesリンクのコピーリンクがクリップボードにコピーされました!

詳細情報

試用、購入および販売

コミュニティー

会社概要

多様性を受け入れるオープンソースの強化

Red Hat ドキュメントについて

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

7.3.1. Deploying a Kafka cluster in KRaft mode
リンクのコピー

7.3.2. Deploying a ZooKeeper-based Kafka cluster
リンクのコピー

7.3.3. Deploying the Topic Operator using the Cluster Operator
リンクのコピー

7.3.4. Deploying the User Operator using the Cluster Operator
リンクのコピー

7.3.5. Connecting to ZooKeeper from a terminal
リンクのコピー

7.3.6. List of Kafka cluster resources
リンクのコピー