Chapter 8. Configuring a deployment

Procedure

1. Add the plugin properties to the config of the Kafka resource.

   The plugin properties are shown in this example configuration.

   Example Kafka Static Quota plugin configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     kafka:
       # ...
       config:
         client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback 

   1

   
         client.quota.callback.static.produce: 1000000 

   2

   
         client.quota.callback.static.fetch: 1000000 

   3

   
         client.quota.callback.static.storage.soft: 400000000000 

   4

   
         client.quota.callback.static.storage.hard: 500000000000 

   5

   
         client.quota.callback.static.storage.check-interval: 5 

   6

   Copy to Clipboard Toggle word wrap

   1

   Loads the Kafka Static Quota plugin.

   2

   Sets the producer byte-rate threshold. 1 MBps in this example.

   3

   Sets the consumer byte-rate threshold. 1 MBps in this example.

   4

   Sets the lower soft limit for storage. 400 GB in this example.

   5

   Sets the higher hard limit for storage. 500 GB in this example.

   6

   Sets the interval in seconds between checks on storage. 5 seconds in this example. You can set this to 0 to disable the check.

2. Update the resource.

   oc apply -f <kafka_configuration_file>

   Copy to Clipboard Toggle word wrap

Procedure

1. Annotate the node pool with the IDs to use when scaling up or scaling down, as shown in the following examples.

   IDs for scaling up are assigned to node pool pool-a:

   Assigning IDs for scaling up

   oc annotate kafkanodepool pool-a strimzi.io/next-node-ids="[0,1,2,10-20,30]"

   Copy to Clipboard Toggle word wrap

   The lowest available ID from this range is used when adding a node to pool-a.

   IDs for scaling down are assigned to node pool pool-b:

   Assigning IDs for scaling down

   oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[60-50,9,8,7]"

   Copy to Clipboard Toggle word wrap

   The highest available ID from this range is removed when scaling down pool-b.

2. You can now scale the node pool.

   For more information, see the following:

   • Section 8.3.2, “(Preview) Adding nodes to a node pool”
   • Section 8.3.3, “(Preview) Removing nodes from a node pool”
   • Section 8.3.4, “(Preview) Moving nodes between node pools”

   On reconciliation, a warning is given if the annotations are misformatted.

Procedure

1. Create a new node in the node pool.

   For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=4

   Copy to Clipboard Toggle word wrap

2. Check the status of the deployment and wait for the pods in the node pool to be created and have a status of READY.

   oc get pods -n <my_cluster_operator_namespace>

   Copy to Clipboard Toggle word wrap

   Output shows four Kafka nodes in the node pool

   NAME                       READY  STATUS   RESTARTS
   my-cluster-pool-a-kafka-0  1/1    Running  0
   my-cluster-pool-a-kafka-1  1/1    Running  0
   my-cluster-pool-a-kafka-2  1/1    Running  0
   my-cluster-pool-a-kafka-3  1/1    Running  0

   Copy to Clipboard Toggle word wrap

3. Reassign the partitions after increasing the number of nodes in the node pool.

   After scaling up a node pool, you can use the Cruise Control add-brokers mode to move partition replicas from existing brokers to the newly added brokers.

Procedure

1. Reassign the partitions before decreasing the number of nodes in the node pool.

   Before scaling down a node pool, you can use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.

2. After the reassignment process is complete, and the node being removed has no live partitions, reduce the number of Kafka nodes in the node pool.

   For example, node pool pool-a has four replicas. We remove a node by decreasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=3

   Copy to Clipboard Toggle word wrap

   Output shows three Kafka nodes in the node pool

   NAME                        READY   STATUS    RESTARTS
   my-cluster-pool-b-kafka-0   1/1     Running   0
   my-cluster-pool-b-kafka-1   1/1     Running   0
   my-cluster-pool-b-kafka-2   1/1     Running   0

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new node in the target node pool.

   For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=4

   Copy to Clipboard Toggle word wrap

2. Check the status of the deployment and wait for the pods in the node pool to be created and have a status of READY.

   oc get pods -n <my_cluster_operator_namespace>

   Copy to Clipboard Toggle word wrap

   Output shows four Kafka nodes in the target node pool

   NAME                       READY  STATUS   RESTARTS
   my-cluster-pool-a-kafka-0  1/1    Running  0
   my-cluster-pool-a-kafka-1  1/1    Running  0
   my-cluster-pool-a-kafka-4  1/1    Running  0
   my-cluster-pool-a-kafka-5  1/1    Running  0

   Copy to Clipboard Toggle word wrap

   Node IDs are appended to the name of the node on creation. We add node my-cluster-pool-a-kafka-5, which has a node ID of 5.

3. Reassign the partitions from the old node to the new node.

   Before scaling down the source node pool, you can use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.

4. After the reassignment process is complete, reduce the number of Kafka nodes in the source node pool.

   For example, node pool pool-b has four replicas. We remove a node by decreasing the number of replicas:

   oc scale kafkanodepool pool-b --replicas=3

   Copy to Clipboard Toggle word wrap

   The node with the highest ID within a pool is removed.

   Output shows three Kafka nodes in the source node pool

   NAME                       READY  STATUS   RESTARTS
   my-cluster-pool-b-kafka-2  1/1    Running  0
   my-cluster-pool-b-kafka-3  1/1    Running  0
   my-cluster-pool-b-kafka-6  1/1    Running  0

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new KafkaNodePool resource.

   1. Name the resource kafka.
   2. Point a strimzi.io/cluster label to your existing Kafka resource.
   3. Set the replica count and storage configuration to match your current Kafka cluster.
   4. Set the roles to broker.

   Example configuration for a node pool used in migrating a Kafka cluster

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: kafka
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - broker
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false

   Copy to Clipboard Toggle word wrap

2. Apply the KafkaNodePool resource:

   oc apply -f <node_pool_configuration_file>

   Copy to Clipboard Toggle word wrap

   By applying this resource, you switch Kafka to using node pools.

   There is no change or rolling update and resources are identical to how they were before.

3. Update the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration to include +KafkaNodePools.

   env:
     - name: STRIMZI_FEATURE_GATES
       value: +KafkaNodePools

   Copy to Clipboard Toggle word wrap

4. Enable the KafkaNodePools feature gate in the Kafka resource using the strimzi.io/node-pools: enabled annotation.

   Example configuration for a node pool in a cluster using ZooKeeper

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     annotations:
       strimzi.io/node-pools: enabled
   spec:
     kafka:
       version: 3.5.0
       replicas: 3
     # ...
     storage:
         type: jbod
         volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false

   Copy to Clipboard Toggle word wrap

5. Apply the Kafka resource:

   oc apply -f <kafka_configuration_file>

   Copy to Clipboard Toggle word wrap

Procedure

1. To add proxy environment variables to the Cluster Operator, update its Deployment configuration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml).

   Example proxy configuration for the Cluster Operator

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
           # ...
           env:
           # ...
           - name: "HTTP_PROXY"
             value: "http://proxy.com" 

   1

   
           - name: "HTTPS_PROXY"
             value: "https://proxy.com" 

   2

   
           - name: "NO_PROXY"
             value: "internal.com, other.domain.com" 

   3

   
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Address of the proxy server.

   2

   Secure address of the proxy server.

   3

   Addresses for servers that are accessed directly as exceptions to the proxy server. The URLs are comma-separated.

   Alternatively, edit the Deployment directly:

   oc edit deployment strimzi-cluster-operator

   Copy to Clipboard Toggle word wrap

2. If you updated the YAML file instead of editing the Deployment directly, apply the changes:

   oc create -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. To disable the FIPS mode in the Cluster Operator, update its Deployment configuration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml) and add the FIPS_MODE environment variable.

   Example FIPS configuration for the Cluster Operator

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
           # ...
           env:
           # ...
           - name: "FIPS_MODE"
             value: "disabled" 

   1

   
     # ...

   Copy to Clipboard Toggle word wrap

   1

   Disables the FIPS mode.

   Alternatively, edit the Deployment directly:

   oc edit deployment strimzi-cluster-operator

   Copy to Clipboard Toggle word wrap

2. If you updated the YAML file instead of editing the Deployment directly, apply the changes:

   oc apply -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the authorization property in the KafkaUser resource to provide access rights to the user.

   In the following example, access rights are configured for the Kafka Connect topics and consumer group using literal name values:

   Expand

   Property	Name
   offset.storage.topic	connect-cluster-offsets
   status.storage.topic	connect-cluster-status
   config.storage.topic	connect-cluster-configs
   group	connect-cluster

   Show more

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaUser
   metadata:
     name: my-user
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     # ...
     authorization:
       type: simple
       acls:
         # access to offset.storage.topic
         - resource:
             type: topic
             name: connect-cluster-offsets
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # access to status.storage.topic
         - resource:
             type: topic
             name: connect-cluster-status
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # access to config.storage.topic
         - resource:
             type: topic
             name: connect-cluster-configs
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # consumer group
         - resource:
             type: group
             name: connect-cluster
             patternType: literal
           operations:
             - Read
           host: "*"

   Copy to Clipboard Toggle word wrap

2. Create or update the resource.

   oc apply -f KAFKA-USER-CONFIG-FILE

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure two Kafka resources, one to secure the source Kafka cluster and one to secure the target Kafka cluster.

   You can add listener configuration for authentication and enable authorization.

   In this example, an internal listener is configured for a Kafka cluster with TLS encryption and mTLS authentication. Kafka simple authorization is enabled.

   Example source Kafka cluster configuration with TLS encryption and mTLS authentication

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-source-cluster
   spec:
     kafka:
       version: 3.5.0
       replicas: 1
       listeners:
         - name: tls
           port: 9093
           type: internal
           tls: true
           authentication:
             type: tls
       authorization:
         type: simple
       config:
         offsets.topic.replication.factor: 1
         transaction.state.log.replication.factor: 1
         transaction.state.log.min.isr: 1
         default.replication.factor: 1
         min.insync.replicas: 1
         inter.broker.protocol.version: "3.5"
       storage:
         type: jbod
         volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false
     zookeeper:
       replicas: 1
       storage:
         type: persistent-claim
         size: 100Gi
         deleteClaim: false
     entityOperator:
       topicOperator: {}
       userOperator: {}

   Copy to Clipboard Toggle word wrap

   Example target Kafka cluster configuration with TLS encryption and mTLS authentication

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-target-cluster
   spec:
     kafka:
       version: 3.5.0
       replicas: 1
       listeners:
         - name: tls
           port: 9093
           type: internal
           tls: true
           authentication:
             type: tls
       authorization:
         type: simple
       config:
         offsets.topic.replication.factor: 1
         transaction.state.log.replication.factor: 1
         transaction.state.log.min.isr: 1
         default.replication.factor: 1
         min.insync.replicas: 1
         inter.broker.protocol.version: "3.5"
       storage:
         type: jbod
         volumes:
           - id: 0
             type: persistent-claim
             size: 100Gi
             deleteClaim: false
     zookeeper:
       replicas: 1
       storage:
         type: persistent-claim
         size: 100Gi
         deleteClaim: false
     entityOperator:
       topicOperator: {}
       userOperator: {}

   Copy to Clipboard Toggle word wrap

2. Create or update the Kafka resources in separate namespaces.

   oc apply -f <kafka_configuration_file> -n <namespace>

   Copy to Clipboard Toggle word wrap

   The Cluster Operator creates the listeners and sets up the cluster and client certificate authority (CA) certificates to enable authentication within the Kafka cluster.

   The certificates are created in the secret <cluster_name>-cluster-ca-cert.

3. Configure two KafkaUser resources, one for a user of the source Kafka cluster and one for a user of the target Kafka cluster.

   1. Configure the same authentication and authorization types as the corresponding source and target Kafka cluster. For example, if you used tls authentication and the simple authorization type in the Kafka configuration for the source Kafka cluster, use the same in the KafkaUser configuration.

   2. Configure the ACLs needed by MirrorMaker 2 to allow operations on the source and target Kafka clusters.

      The ACLs are used by the internal MirrorMaker connectors, and by the underlying Kafka Connect framework.

   Example source user configuration for mTLS authentication

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaUser
   metadata:
     name: my-source-user
     labels:
       strimzi.io/cluster: my-source-cluster
   spec:
     authentication:
       type: tls
     authorization:
       type: simple
       acls:
         # MirrorSourceConnector
         - resource: # Not needed if offset-syncs.topic.location=target
             type: topic
             name: mm2-offset-syncs.my-target-cluster.internal
           operations:
             - Create
             - DescribeConfigs
             - Read
             - Write
         - resource: # Needed for every topic which is mirrored
             type: topic
             name: "*"
           operations:
             - DescribeConfigs
             - Read
         # MirrorCheckpointConnector
         - resource:
             type: cluster
           operations:
             - Describe
         - resource: # Needed for every group for which offsets are synced
             type: group
             name: "*"
           operations:
             - Describe
         - resource: # Not needed if offset-syncs.topic.location=target
             type: topic
             name: mm2-offset-syncs.my-target-cluster.internal
           operations:
             - Read

   Copy to Clipboard Toggle word wrap

   Example target user configuration for mTLS authentication

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaUser
   metadata:
     name: my-target-user
     labels:
       strimzi.io/cluster: my-target-cluster
   spec:
     authentication:
       type: tls
     authorization:
       type: simple
       acls:
         # Underlying Kafka Connect internal topics to store configuration, offsets, or status
         - resource:
             type: group
             name: mirrormaker2-cluster
           operations:
             - Read
         - resource:
             type: topic
             name: mirrormaker2-cluster-configs
           operations:
             - Create
             - Describe
             - DescribeConfigs
             - Read
             - Write
         - resource:
             type: topic
             name: mirrormaker2-cluster-status
           operations:
             - Create
             - Describe
             - DescribeConfigs
             - Read
             - Write
         - resource:
             type: topic
             name: mirrormaker2-cluster-offsets
           operations:
             - Create
             - Describe
             - DescribeConfigs
             - Read
             - Write
         # MirrorSourceConnector
         - resource: # Needed for every topic which is mirrored
             type: topic
             name: "*"
           operations:
             - Create
             - Alter
             - AlterConfigs
             - Write
         # MirrorCheckpointConnector
         - resource:
             type: cluster
           operations:
             - Describe
         - resource:
             type: topic
             name: my-source-cluster.checkpoints.internal
           operations:
             - Create
             - Describe
             - Read
             - Write
         - resource: # Needed for every group for which the offset is synced
             type: group
             name: "*"
           operations:
             - Read
             - Describe
         # MirrorHeartbeatConnector
         - resource:
             type: topic
             name: heartbeats
           operations:
             - Create
             - Describe
             - Write

   Copy to Clipboard Toggle word wrap

   Note

   You can use a certificate issued outside the User Operator by setting type to tls-external. For more information, see the KafkaUserSpec schema reference.

4. Create or update a KafkaUser resource in each of the namespaces you created for the source and target Kafka clusters.

   oc apply -f <kafka_user_configuration_file> -n <namespace>

   Copy to Clipboard Toggle word wrap

   The User Operator creates the users representing the client (MirrorMaker), and the security credentials used for client authentication, based on the chosen authentication type.

   The User Operator creates a new secret with the same name as the KafkaUser resource. The secret contains a private and public key for mTLS authentication. The public key is contained in a user certificate, which is signed by the clients CA.

5. Configure a KafkaMirrorMaker2 resource with the authentication details to connect to the source and target Kafka clusters.

   Example MirrorMaker 2 configuration with TLS encryption and mTLS authentication

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaMirrorMaker2
   metadata:
     name: my-mirror-maker-2
   spec:
     version: 3.5.0
     replicas: 1
     connectCluster: "my-target-cluster"
     clusters:
       - alias: "my-source-cluster"
         bootstrapServers: my-source-cluster-kafka-bootstrap:9093
         tls: 

   1

   
           trustedCertificates:
             - secretName: my-source-cluster-cluster-ca-cert
               certificate: ca.crt
         authentication: 

   2

   
           type: tls
           certificateAndKey:
             secretName: my-source-user
             certificate: user.crt
             key: user.key
       - alias: "my-target-cluster"
         bootstrapServers: my-target-cluster-kafka-bootstrap:9093
         tls: 

   3

   
           trustedCertificates:
             - secretName: my-target-cluster-cluster-ca-cert
               certificate: ca.crt
         authentication: 

   4

   
           type: tls
           certificateAndKey:
             secretName: my-target-user
             certificate: user.crt
             key: user.key
         config:
           # -1 means it will use the default replication factor configured in the broker
           config.storage.replication.factor: -1
           offset.storage.replication.factor: -1
           status.storage.replication.factor: -1
     mirrors:
       - sourceCluster: "my-source-cluster"
         targetCluster: "my-target-cluster"
         sourceConnector:
           config:
             replication.factor: 1
             offset-syncs.topic.replication.factor: 1
             sync.topic.acls.enabled: "false"
         heartbeatConnector:
           config:
             heartbeats.topic.replication.factor: 1
         checkpointConnector:
           config:
             checkpoints.topic.replication.factor: 1
             sync.group.offsets.enabled: "true"
         topicsPattern: "topic1|topic2|topic3"
         groupsPattern: "group1|group2|group3"

   Copy to Clipboard Toggle word wrap

   1

   The TLS certificates for the source Kafka cluster. If they are in a separate namespace, copy the cluster secrets from the namespace of the Kafka cluster.

   2

   The user authentication for accessing the source Kafka cluster using the TLS mechanism.

   3

   The TLS certificates for the target Kafka cluster.

   4

   The user authentication for accessing the target Kafka cluster.

6. Create or update the KafkaMirrorMaker2 resource in the same namespace as the target Kafka cluster.

   oc apply -f <mirrormaker2_configuration_file> -n <namespace_of_target_cluster>

   Copy to Clipboard Toggle word wrap