ホーム
製品
Streams for Apache Kafka
2.7
Deploying and Managing Streams for Apache Kafka on OpenShift
9.3. Configuring node pools

9.3. Configuring node pools

Update the spec properties of the KafkaNodePool custom resource to configure a node pool deployment.

A node pool refers to a distinct group of Kafka nodes within a Kafka cluster. Each pool has its own unique configuration, which includes mandatory settings for the number of replicas, roles, and storage allocation.

Optionally, you can also specify values for the following properties:

resources to specify memory and cpu requests and limits
template to specify custom configuration for pods and other OpenShift resources
jvmOptions to specify custom JVM configuration for heap size, runtime and other options

The Kafka resource represents the configuration for all nodes in the Kafka cluster. The KafkaNodePool resource represents the configuration for nodes only in the node pool. If a configuration property is not specified in KafkaNodePool, it is inherited from the Kafka resource. Configuration specified in the KafkaNodePool resource takes precedence if set in both resources. For example, if both the node pool and Kafka configuration includes jvmOptions, the values specified in the node pool configuration are used. When -Xmx: 1024m is set in KafkaNodePool.spec.jvmOptions and -Xms: 512m is set in Kafka.spec.kafka.jvmOptions, the node uses the value from its node pool configuration.

Properties from Kafka and KafkaNodePool schemas are not combined. To clarify, if KafkaNodePool.spec.template includes only podSet.metadata.labels, and Kafka.spec.kafka.template includes podSet.metadata.annotations and pod.metadata.labels, the template values from the Kafka configuration are ignored since there is a template value in the node pool configuration.

For a deeper understanding of the node pool configuration options, refer to the Streams for Apache Kafka Custom Resource API Reference.

Example configuration for a node pool in a cluster using KRaft mode

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kraft-dual-role


  labels:
    strimzi.io/cluster: my-cluster


spec:
  replicas: 3


  roles:


    - controller
    - broker
  storage:


    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  resources:


      requests:
        memory: 64Gi
        cpu: "8"
      limits:
        memory: 64Gi
        cpu: "12"

1: Unique name for the node pool.
2: The Kafka cluster the node pool belongs to. A node pool can only belong to a single cluster.
3: Number of replicas for the nodes.
4: Roles for the nodes in the node pool. In this example, the nodes have dual roles as controllers and brokers.
5: Storage specification for the nodes.
6: Requests for reservation of supported resources, currently cpu and memory, and limits to specify the maximum resources that can be consumed.

注記

The configuration for the Kafka resource must be suitable for KRaft mode. Currently, KRaft mode has a number of limitations.

Example configuration for a node pool in a cluster using ZooKeeper

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - broker


  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  resources:
      requests:
        memory: 64Gi
        cpu: "8"
      limits:
        memory: 64Gi
        cpu: "12"

1: Roles for the nodes in the node pool, which can only be broker when using Kafka with ZooKeeper.

9.3.1. Assigning IDs to node pools for scaling operations
リンクのコピー

This procedure describes how to use annotations for advanced node ID handling by the Cluster Operator when performing scaling operations on node pools. You specify the node IDs to use, rather than the Cluster Operator using the next ID in sequence. Management of node IDs in this way gives greater control.

To add a range of IDs, you assign the following annotations to the KafkaNodePool resource:

strimzi.io/next-node-ids to add a range of IDs that are used for new brokers
strimzi.io/remove-node-ids to add a range of IDs for removing existing brokers

You can specify an array of individual node IDs, ID ranges, or a combination of both. For example, you can specify the following range of IDs: [0, 1, 2, 10-20, 30] for scaling up the Kafka node pool. This format allows you to specify a combination of individual node IDs (0, 1, 2, 30) as well as a range of IDs (10-20).

In a typical scenario, you might specify a range of IDs for scaling up and a single node ID to remove a specific node when scaling down.

In this procedure, we add the scaling annotations to node pools as follows:

pool-a is assigned a range of IDs for scaling up
pool-b is assigned a range of IDs for scaling down

During the scaling operation, IDs are used as follows:

Scale up picks up the lowest available ID in the range for the new node.
Scale down removes the node with the highest available ID in the range.

If there are gaps in the sequence of node IDs assigned in the node pool, the next node to be added is assigned an ID that fills the gap.

The annotations don’t need to be updated after every scaling operation. Any unused IDs are still valid for the next scaling event.

The Cluster Operator allows you to specify a range of IDs in either ascending or descending order, so you can define them in the order the nodes are scaled. For example, when scaling up, you can specify a range such as [1000-1999], and the new nodes are assigned the next lowest IDs: 1000, 1001, 1002, 1003, and so on. Conversely, when scaling down, you can specify a range like [1999-1000], ensuring that nodes with the next highest IDs are removed: 1003, 1002, 1001, 1000, and so on.

If you don’t specify an ID range using the annotations, the Cluster Operator follows its default behavior for handling IDs during scaling operations. Node IDs start at 0 (zero) and run sequentially across the Kafka cluster. The next lowest ID is assigned to a new node. Gaps to node IDs are filled across the cluster. This means that they might not run sequentially within a node pool. The default behavior for scaling up is to add the next lowest available node ID across the cluster; and for scaling down, it is to remove the node in the node pool with the highest available node ID. The default approach is also applied if the assigned range of IDs is misformatted, the scaling up range runs out of IDs, or the scaling down range does not apply to any in-use nodes.

Prerequisites

The Cluster Operator must be deployed.
(Optional) Use the reserved.broker-max.id configuration property to extend the allowable range for node IDs within your node pools.

By default, Apache Kafka restricts node IDs to numbers ranging from 0 to 999. To use node ID values greater than 999, add the reserved.broker-max.id configuration property to the Kafka custom resource and specify the required maximum node ID value.

In this example, the maximum node ID is set at 10000. Node IDs can then be assigned up to that value.

Example configuration for the maximum node ID number

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    config:
      reserved.broker.max.id: 10000
  # ...

Procedure

Annotate the node pool with the IDs to use when scaling up or scaling down, as shown in the following examples.
IDs for scaling up are assigned to node pool pool-a:
Assigning IDs for scaling up
```
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids="[0,1,2,10-20,30]"
```
The lowest available ID from this range is used when adding a node to pool-a.
IDs for scaling down are assigned to node pool pool-b:
Assigning IDs for scaling down
```
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[60-50,9,8,7]"
```
The highest available ID from this range is removed when scaling down pool-b.
注記
If you want to remove a specific node, you can assign a single node ID to the scaling down annotation: oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[3]".
You can now scale the node pool.
For more information, see the following:
On reconciliation, a warning is given if the annotations are misformatted.
After you have performed the scaling operation, you can remove the annotation if it’s no longer needed.
Removing the annotation for scaling up
```
oc annotate kafkanodepool pool-a strimzi.io/next-node-ids-
```
Removing the annotation for scaling down
```
oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids-
```

9.3.2. Impact on racks when moving nodes from node pools
リンクのコピー

If rack awareness is enabled on a Kafka cluster, replicas can be spread across different racks, data centers, or availability zones. When moving nodes from node pools, consider the implications on the cluster topology, particularly regarding rack awareness. Removing specific pods from node pools, especially out of order, may break the cluster topology or cause an imbalance in distribution across racks. An imbalance can impact both the distribution of nodes themselves and the partition replicas within the cluster. An uneven distribution of nodes and partitions across racks can affect the performance and resilience of the Kafka cluster.

Plan the removal of nodes strategically to maintain the required balance and resilience across racks. Use the strimzi.io/remove-node-ids annotation to move nodes with specific IDs with caution. Ensure that configuration to spread partition replicas across racks and for clients to consume from the closest replicas is not broken.

ヒント

Use Cruise Control and the KafkaRebalance resource with the RackAwareGoal to make sure that replicas remain distributed across different racks.

9.3.3. Adding nodes to a node pool
リンクのコピー

This procedure describes how to scale up a node pool to add new nodes. Currently, scale up is only possible for broker-only node pools containing nodes that run as dedicated brokers.

In this procedure, we start with three nodes for node pool pool-a:

Kafka nodes in the node pool

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0

Node IDs are appended to the name of the node on creation. We add node my-cluster-pool-a-3, which has a node ID of 3.

注記

During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

Prerequisites

The Cluster Operator must be deployed.
Cruise Control is deployed with Kafka.
(Optional) For scale up operations, you can specify the node IDs to use in the operation.
If you have assigned a range of node IDs for the operation, the ID of the node being added is determined by the sequence of nodes given. If you have assigned a single node ID, a node is added with the specified ID. Otherwise, the lowest available node ID across the cluster is used.

Procedure

Create a new node in the node pool.
For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:
```
oc scale kafkanodepool pool-a --replicas=4
```

Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows four Kafka nodes in the node pool

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0
my-cluster-pool-a-3  1/1    Running  0

Reassign the partitions after increasing the number of nodes in the node pool.
After scaling up a node pool, use the Cruise Control add-brokers mode to move partition replicas from existing brokers to the newly added brokers.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: add-brokers
  brokers: [3]
```
We are reassigning partitions to node my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.

9.3.4. Removing nodes from a node pool
リンクのコピー

This procedure describes how to scale down a node pool to remove nodes. Currently, scale down is only possible for broker-only node pools containing nodes that run as dedicated brokers.

In this procedure, we start with four nodes for node pool pool-a:

Kafka nodes in the node pool

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0
my-cluster-pool-a-3  1/1    Running  0

Node IDs are appended to the name of the node on creation. We remove node my-cluster-pool-a-3, which has a node ID of 3.

注記

During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

Prerequisites

The Cluster Operator must be deployed.
Cruise Control is deployed with Kafka.
(Optional) For scale down operations, you can specify the node IDs to use in the operation.
If you have assigned a range of node IDs for the operation, the ID of the node being removed is determined by the sequence of nodes given. If you have assigned a single node ID, the node with the specified ID is removed. Otherwise, the node with the highest available ID in the node pool is removed.

Procedure

Reassign the partitions before decreasing the number of nodes in the node pool.
Before scaling down a node pool, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [3]
```
We are reassigning partitions from node my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.
After the reassignment process is complete, and the node being removed has no live partitions, reduce the number of Kafka nodes in the node pool.
For example, node pool pool-a has four replicas. We remove a node by decreasing the number of replicas:
```
oc scale kafkanodepool pool-a --replicas=3
```
Output shows three Kafka nodes in the node pool
```
NAME                       READY  STATUS   RESTARTS
my-cluster-pool-b-kafka-0  1/1    Running  0
my-cluster-pool-b-kafka-1  1/1    Running  0
my-cluster-pool-b-kafka-2  1/1    Running  0
```

9.3.5. Moving nodes between node pools
リンクのコピー

This procedure describes how to move nodes between source and target Kafka node pools without downtime. You create a new node on the target node pool and reassign partitions to move data from the old node on the source node pool. When the replicas on the new node are in-sync, you can delete the old node.

In this procedure, we start with two node pools:

pool-a with three replicas is the target node pool
pool-b with four replicas is the source node pool

We scale up pool-a, and reassign partitions and scale down pool-b, which results in the following:

pool-a with four replicas
pool-b with three replicas

注記

During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

Prerequisites

The Cluster Operator must be deployed.
Cruise Control is deployed with Kafka.
(Optional) For scale up and scale down operations, you can specify the range of node IDs to use.
If you have assigned node IDs for the operation, the ID of the node being added or removed is determined by the sequence of nodes given. Otherwise, the lowest available node ID across the cluster is used when adding nodes; and the node with the highest available ID in the node pool is removed.

Procedure

Create a new node in the target node pool.
For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:
```
oc scale kafkanodepool pool-a --replicas=4
```

Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows four Kafka nodes in the source and target node pools

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-4  1/1    Running  0
my-cluster-pool-a-7  1/1    Running  0
my-cluster-pool-b-2  1/1    Running  0
my-cluster-pool-b-3  1/1    Running  0
my-cluster-pool-b-5  1/1    Running  0
my-cluster-pool-b-6  1/1    Running  0

Node IDs are appended to the name of the node on creation. We add node my-cluster-pool-a-7, which has a node ID of 7.

Reassign the partitions from the old node to the new node.
Before scaling down the source node pool, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [6]
```
We are reassigning partitions from node my-cluster-pool-b-6. The reassignment can take some time depending on the number of topics and partitions in the cluster.
After the reassignment process is complete, reduce the number of Kafka nodes in the source node pool.
For example, node pool pool-b has four replicas. We remove a node by decreasing the number of replicas:
```
oc scale kafkanodepool pool-b --replicas=3
```
The node with the highest ID (6) within the pool is removed.
Output shows three Kafka nodes in the source node pool
```
NAME                       READY  STATUS   RESTARTS
my-cluster-pool-b-kafka-2  1/1    Running  0
my-cluster-pool-b-kafka-3  1/1    Running  0
my-cluster-pool-b-kafka-5  1/1    Running  0
```

9.3.6. Changing node pool roles
リンクのコピー

Node pools can be used with Kafka clusters that operate in KRaft mode (using Kafka Raft metadata) or use ZooKeeper for metadata management. If you are using KRaft mode, you can specify roles for all nodes in the node pool to operate as brokers, controllers, or both. If you are using ZooKeeper, nodes must be set as brokers only.

In certain circumstances you might want to change the roles assigned to a node pool. For example, you may have a node pool that contains nodes that perform dual broker and controller roles, and then decide to split the roles between two node pools. In this case, you create a new node pool with nodes that act only as brokers, and then reassign partitions from the dual-role nodes to the new brokers. You can then switch the old node pool to a controller-only role.

You can also perform the reverse operation by moving from node pools with controller-only and broker-only roles to a node pool that contains nodes that perform dual broker and controller roles. In this case, you add the broker role to the existing controller-only node pool, reassign partitions from the broker-only nodes to the dual-role nodes, and then delete the broker-only node pool.

When removing broker roles in the node pool configuration, keep in mind that Kafka does not automatically reassign partitions. Before removing the broker role, ensure that nodes changing to controller-only roles do not have any assigned partitions. If partitions are assigned, the change is prevented. No replicas must be left on the node before removing the broker role. The best way to reassign partitions before changing roles is to apply a Cruise Control optimization proposal in remove-brokers mode. For more information, see 「Generating optimization proposals」.

9.3.7. Transitioning to separate broker and controller roles
リンクのコピー

This procedure describes how to transition to using node pools with separate roles. If your Kafka cluster is using a node pool with combined controller and broker roles, you can transition to using two node pools with separate roles. To do this, rebalance the cluster to move partition replicas to a node pool with a broker-only role, and then switch the old node pool to a controller-only role.

In this procedure, we start with node pool pool-a, which has controller and broker roles:

Dual-role node pool

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
  # ...

The node pool has three nodes:

Kafka nodes in the node pool

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0

Each node performs a combined role of broker and controller. We create a second node pool called pool-b, with three nodes that act as brokers only.

注記

During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

Prerequisites

Procedure

Create a node pool with a broker role.

Example node pool configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-b
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  # ...

The new node pool also has three nodes. If you already have a broker-only node pool, you can skip this step.

Apply the new KafkaNodePool resource to create the brokers.

Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows pods running in two node pools

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0
my-cluster-pool-b-3  1/1    Running  0
my-cluster-pool-b-4  1/1    Running  0
my-cluster-pool-b-5  1/1    Running  0

Node IDs are appended to the name of the node on creation.

Use the Cruise Control remove-brokers mode to reassign partition replicas from the dual-role nodes to the newly added brokers.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [0, 1, 2]
```
The reassignment can take some time depending on the number of topics and partitions in the cluster.
注記
If nodes changing to controller-only roles have any assigned partitions, the change is prevented. The status.conditions of the Kafka resource provide details of events preventing the change.

Remove the broker role from the node pool that originally had a combined role.

Dual-role nodes switched to controllers

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
  # ...

Apply the configuration change so that the node pool switches to a controller-only role.

9.3.8. Transitioning to dual-role nodes
リンクのコピー

This procedure describes how to transition from separate node pools with broker-only and controller-only roles to using a dual-role node pool. If your Kafka cluster is using node pools with dedicated controller and broker nodes, you can transition to using a single node pool with both roles. To do this, add the broker role to the controller-only node pool, rebalance the cluster to move partition replicas to the dual-role node pool, and then delete the old broker-only node pool.

In this procedure, we start with two node pools pool-a, which has only the controller role and pool-b which has only the broker role:

Single role node pools

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  # ...
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-b
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  # ...

The Kafka cluster has six nodes:

Kafka nodes in the node pools

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0
my-cluster-pool-b-3  1/1    Running  0
my-cluster-pool-b-4  1/1    Running  0
my-cluster-pool-b-5  1/1    Running  0

The pool-a nodes perform the role of controller. The pool-b nodes perform the role of broker.

注記

During this process, the ID of the node that holds the partition replicas changes. Consider any dependencies that reference the node ID.

Prerequisites

Procedure

Edit the node pool pool-a and add the broker role to it.

Example node pool configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  # ...

Check the status and wait for the pods in the node pool to be restarted and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows pods running in two node pools

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0
my-cluster-pool-b-3  1/1    Running  0
my-cluster-pool-b-4  1/1    Running  0
my-cluster-pool-b-5  1/1    Running  0

Node IDs are appended to the name of the node on creation.

Use the Cruise Control remove-brokers mode to reassign partition replicas from the broker-only nodes to the dual-role nodes.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [3, 4, 5]
```
The reassignment can take some time depending on the number of topics and partitions in the cluster.

Remove the pool-b node pool that has the old broker-only nodes.

oc delete kafkanodepool pool-b -n <my_cluster_operator_namespace>

9.3.9. Managing storage using node pools
リンクのコピー

Storage management in Streams for Apache Kafka is usually straightforward, and requires little change when set up, but there might be situations where you need to modify your storage configurations. Node pools simplify this process, because you can set up separate node pools that specify your new storage requirements.

In this procedure we create and manage storage for a node pool called pool-a containing three nodes. We show how to change the storage class (volumes.class) that defines the type of persistent storage it uses. You can use the same steps to change the storage size (volumes.size).

注記

We strongly recommend using block storage. Streams for Apache Kafka is only tested for use with block storage.

Prerequisites

The Cluster Operator must be deployed.
Cruise Control is deployed with Kafka.
For storage that uses persistent volume claims for dynamic volume allocation, storage classes are defined and available in the OpenShift cluster that correspond to the storage solutions you need.

Procedure

Create the node pool with its own storage settings.

For example, node pool pool-a uses JBOD storage with persistent volumes:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-a
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: gp2-ebs
  # ...

Nodes in pool-a are configured to use Amazon EBS (Elastic Block Store) GP2 volumes.

Apply the node pool configuration for pool-a.

Check the status of the deployment and wait for the pods in pool-a to be created and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows three Kafka nodes in the node pool

NAME                 READY  STATUS   RESTARTS
my-cluster-pool-a-0  1/1    Running  0
my-cluster-pool-a-1  1/1    Running  0
my-cluster-pool-a-2  1/1    Running  0

To migrate to a new storage class, create a new node pool with the required storage configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-b
  labels:
    strimzi.io/cluster: my-cluster
spec:
  roles:
    - broker
  replicas: 3
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 1Ti
        class: gp3-ebs
  # ...

Nodes in pool-b are configured to use Amazon EBS (Elastic Block Store) GP3 volumes.

Apply the node pool configuration for pool-b.
Check the status of the deployment and wait for the pods in pool-b to be created and ready.
Reassign the partitions from pool-a to pool-b.
When migrating to a new storage configuration, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.
Using Cruise Control to reassign partition replicas
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  # ...
spec:
  mode: remove-brokers
  brokers: [0, 1, 2]
```
We are reassigning partitions from pool-a. The reassignment can take some time depending on the number of topics and partitions in the cluster.
After the reassignment process is complete, delete the old node pool:
```
oc delete kafkanodepool pool-a
```

9.3.10. Managing storage affinity using node pools
リンクのコピー

In situations where storage resources, such as local persistent volumes, are constrained to specific worker nodes, or availability zones, configuring storage affinity helps to schedule pods to use the right nodes.

Node pools allow you to configure affinity independently. In this procedure, we create and manage storage affinity for two availability zones: zone-1 and zone-2.

You can configure node pools for separate availability zones, but use the same storage class. We define an all-zones persistent storage class representing the storage resources available in each zone.

We also use the .spec.template.pod properties to configure the node affinity and schedule Kafka pods on zone-1 and zone-2 worker nodes.

The storage class and affinity is specified in node pools representing the nodes in each availability zone:

pool-zone-1
pool-zone-2.

Prerequisites

The Cluster Operator must be deployed.
If you are not familiar with the concepts of affinity, see the Kubernetes node and pod affinity documentation.

Procedure

Define the storage class for use with each availability zone:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: all-zones
provisioner: kubernetes.io/my-storage
parameters:
  type: ssd
volumeBindingMode: WaitForFirstConsumer

Create node pools representing the two availability zones, specifying the all-zones storage class and the affinity for each zone:

Node pool configuration for zone-1

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-zone-1
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: all-zones
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                  - zone-1
  # ...

Node pool configuration for zone-2

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: pool-zone-2
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 4
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: all-zones
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                  - zone-2
  # ...

Apply the node pool configuration.

Check the status of the deployment and wait for the pods in the node pools to be created and ready (1/1).

oc get pods -n <my_cluster_operator_namespace>

Output shows 3 Kafka nodes in pool-zone-1 and 4 Kafka nodes in pool-zone-2

NAME                            READY  STATUS   RESTARTS
my-cluster-pool-zone-1-kafka-0  1/1    Running  0
my-cluster-pool-zone-1-kafka-1  1/1    Running  0
my-cluster-pool-zone-1-kafka-2  1/1    Running  0
my-cluster-pool-zone-2-kafka-3  1/1    Running  0
my-cluster-pool-zone-2-kafka-4  1/1    Running  0
my-cluster-pool-zone-2-kafka-5  1/1    Running  0
my-cluster-pool-zone-2-kafka-6  1/1    Running  0

9.3.11. Migrating existing Kafka clusters to use Kafka node pools
リンクのコピー

This procedure describes how to migrate existing Kafka clusters to use Kafka node pools. After you have updated the Kafka cluster, you can use the node pools to manage the configuration of nodes within each pool.

注記

Currently, replica and storage configuration in the KafkaNodePool resource must also be present in the Kafka resource. The configuration is ignored when node pools are being used.

Prerequisites

The Cluster Operator must be deployed.

Procedure

Create a new KafkaNodePool resource.
1. Name the resource kafka.
2. Point a strimzi.io/cluster label to your existing Kafka resource.
3. Set the replica count and storage configuration to match your current Kafka cluster.
4. Set the roles to broker.
Example configuration for a node pool used in migrating a Kafka cluster
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
```
警告
To preserve cluster data and the names of its nodes and resources, the node pool name must be kafka, and the strimzi.io/cluster label must match the Kafka resource name. Otherwise, nodes and resources are created with new names, including the persistent volume storage used by the nodes. Consequently, your previous data may not be available.
Apply the KafkaNodePool resource:
```
oc apply -f <node_pool_configuration_file>
```
By applying this resource, you switch Kafka to using node pools.
There is no change or rolling update and resources are identical to how they were before.

Enable support for node pools in the Kafka resource using the strimzi.io/node-pools: enabled annotation.

Example configuration for a node pool in a cluster using ZooKeeper

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  annotations:
    strimzi.io/node-pools: enabled
spec:
  kafka:
    # ...
  zookeeper:
    # ...

Apply the Kafka resource:
```
oc apply -f <kafka_configuration_file>
```
There is no change or rolling update. The resources remain identical to how they were before.
Remove the replicated properties from the Kafka custom resource. When the KafkaNodePool resource is in use, you can remove the properties that you copied to the KafkaNodePool resource, such as the .spec.kafka.replicas and .spec.kafka.storage properties.

Reversing the migration

To revert to managing Kafka nodes using only Kafka custom resources:

If you have multiple node pools, consolidate them into a single KafkaNodePool named kafka with node IDs from 0 to N (where N is the number of replicas).
Ensure that the .spec.kafka configuration in the Kafka resource matches the KafkaNodePool configuration, including storage, resources, and replicas.
Disable support for node pools in the Kafka resource using the strimzi.io/node-pools: disabled annotation.
Delete the Kafka node pool named kafka.

9.3. Configuring node pools

9.3.1. Assigning IDs to node pools for scaling operations
リンクのコピー

9.3.2. Impact on racks when moving nodes from node pools
リンクのコピー

9.3.3. Adding nodes to a node pool
リンクのコピー

9.3.4. Removing nodes from a node pool
リンクのコピー

9.3.5. Moving nodes between node pools
リンクのコピー

9.3.6. Changing node pool roles
リンクのコピー

9.3.7. Transitioning to separate broker and controller roles
リンクのコピー

9.3.8. Transitioning to dual-role nodes
リンクのコピー

9.3.9. Managing storage using node pools
リンクのコピー

9.3.10. Managing storage affinity using node pools
リンクのコピー

9.3.11. Migrating existing Kafka clusters to use Kafka node pools
リンクのコピー

詳細情報

試用、購入および販売

コミュニティー

会社概要

多様性を受け入れるオープンソースの強化

Red Hat ドキュメントについて

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

9.3. Configuring node pools

9.3.1. Assigning IDs to node pools for scaling operationsリンクのコピーリンクがクリップボードにコピーされました!

9.3.2. Impact on racks when moving nodes from node poolsリンクのコピーリンクがクリップボードにコピーされました!

9.3.3. Adding nodes to a node poolリンクのコピーリンクがクリップボードにコピーされました!

9.3.4. Removing nodes from a node poolリンクのコピーリンクがクリップボードにコピーされました!

9.3.5. Moving nodes between node poolsリンクのコピーリンクがクリップボードにコピーされました!

9.3.6. Changing node pool rolesリンクのコピーリンクがクリップボードにコピーされました!

9.3.7. Transitioning to separate broker and controller rolesリンクのコピーリンクがクリップボードにコピーされました!

9.3.8. Transitioning to dual-role nodesリンクのコピーリンクがクリップボードにコピーされました!

9.3.9. Managing storage using node poolsリンクのコピーリンクがクリップボードにコピーされました!

9.3.10. Managing storage affinity using node poolsリンクのコピーリンクがクリップボードにコピーされました!

9.3.11. Migrating existing Kafka clusters to use Kafka node poolsリンクのコピーリンクがクリップボードにコピーされました!

詳細情報

試用、購入および販売

コミュニティー

会社概要

多様性を受け入れるオープンソースの強化

Red Hat ドキュメントについて

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

9.3.1. Assigning IDs to node pools for scaling operations
リンクのコピー

9.3.2. Impact on racks when moving nodes from node pools
リンクのコピー

9.3.3. Adding nodes to a node pool
リンクのコピー

9.3.4. Removing nodes from a node pool
リンクのコピー

9.3.5. Moving nodes between node pools
リンクのコピー

9.3.6. Changing node pool roles
リンクのコピー

9.3.7. Transitioning to separate broker and controller roles
リンクのコピー

9.3.8. Transitioning to dual-role nodes
リンクのコピー

9.3.9. Managing storage using node pools
リンクのコピー

9.3.10. Managing storage affinity using node pools
リンクのコピー

9.3.11. Migrating existing Kafka clusters to use Kafka node pools
リンクのコピー