Deploying and Managing Streams for Apache Kafka on OpenShift

Procedure

1. Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

3. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to add a list of all the namespaces the Cluster Operator will watch to the STRIMZI_NAMESPACE environment variable.

   For example, in this procedure the Cluster Operator will watch the namespaces watched-namespace-1, watched-namespace-2, watched-namespace-3.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq-streams/strimzi-rhel9-operator:2.7.0
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: watched-namespace-1,watched-namespace-2,watched-namespace-3

3. For each namespace listed, install the RoleBindings.

   In this example, we replace watched-namespace in these commands with the namespaces listed in the previous step, repeating them for watched-namespace-1, watched-namespace-2, watched-namespace-3:

   oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace>
   oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace>
   oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>

4. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the Streams for Apache Kafka installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to set the value of the STRIMZI_NAMESPACE environment variable to *.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         # ...
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq-streams/strimzi-rhel9-operator:2.7.0
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: "*"
           # ...

3. Create ClusterRoleBindings that grant cluster-wide access for all namespaces to the Cluster Operator.

   oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
   oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
   oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator

4. Deploy the Cluster Operator to your OpenShift cluster.

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Deploy a KRaft-based Kafka cluster.

   • To deploy a Kafka cluster in KRaft mode with a single node pool that uses dual-role nodes:

     oc apply -f examples/kafka/kraft/kafka-with-dual-role-nodes.yaml

   • To deploy a persistent Kafka cluster in KRaft mode with separate node pools for broker and controller nodes:

     oc apply -f examples/kafka/kraft/kafka.yaml

   • To deploy an ephemeral Kafka cluster in KRaft mode with separate node pools for broker and controller nodes:

     oc apply -f examples/kafka/kraft/kafka-ephemeral.yaml

   • To deploy a Kafka cluster and ZooKeeper cluster with two node pools of three brokers:

     oc apply -f examples/kafka/kafka-with-node-pools.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the node pool names and readiness

   NAME                        READY  STATUS   RESTARTS
   my-cluster-entity-operator  3/3    Running  0
   my-cluster-pool-a-0         1/1    Running  0
   my-cluster-pool-a-1         1/1    Running  0
   my-cluster-pool-a-4         1/1    Running  0

   • my-cluster is the name of the Kafka cluster.

   • pool-a is the name of the node pool.

     A sequential index number starting with 0 identifies each Kafka pod created. If you are using ZooKeeper, you’ll also see the ZooKeeper pods.

     READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

     Information on the deployment is also shown in the status of the KafkaNodePool resource, including a list of IDs for nodes in the pool.

     Note

     Node IDs are assigned sequentially starting at 0 (zero) across all node pools within a cluster. This means that node IDs might not run sequentially within a specific node pool. If there are gaps in the sequence of node IDs across the cluster, the next node to be added is assigned an ID that fills the gap. When scaling down, the node with the highest node ID within a pool is removed.

Procedure

1. Deploy a ZooKeeper-based Kafka cluster.

   • To deploy an ephemeral cluster:

     oc apply -f examples/kafka/kafka-ephemeral.yaml

   • To deploy a persistent cluster:

     oc apply -f examples/kafka/kafka-persistent.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod names and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   my-cluster-kafka-0          1/1     Running   0
   my-cluster-kafka-1          1/1     Running   0
   my-cluster-kafka-2          1/1     Running   0
   my-cluster-zookeeper-0      1/1     Running   0
   my-cluster-zookeeper-1      1/1     Running   0
   my-cluster-zookeeper-2      1/1     Running   0

   my-cluster is the name of the Kafka cluster.

   A sequential index number starting with 0 identifies each Kafka and ZooKeeper pod created.

   With the default deployment, you create an Entity Operator cluster, 3 Kafka pods, and 3 ZooKeeper pods.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. Edit the entityOperator properties of the Kafka resource to include topicOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the Topic Operator spec using the properties described in the EntityTopicOperatorSpec schema reference.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   oc apply -f <kafka_configuration_file>

4. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod name and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   # ...

   my-cluster is the name of the Kafka cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. Edit the entityOperator properties of the Kafka resource to include userOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the User Operator spec using the properties described in EntityUserOperatorSpec schema reference.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   oc apply -f <kafka_configuration_file>

4. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod name and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   # ...

   my-cluster is the name of the Kafka cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. Open the terminal using the OpenShift console or run the exec command from your CLI.

   For example:

   oc exec -ti my-cluster-zookeeper-0 -- bin/zookeeper-shell.sh localhost:12181 ls /

   Be sure to use localhost:12181.

Procedure

1. Deploy Kafka Connect to your OpenShift cluster. Use the examples/connect/kafka-connect.yaml file to deploy Kafka Connect.

   oc apply -f examples/connect/kafka-connect.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                                 READY  STATUS   RESTARTS
   my-connect-cluster-connect-<pod_id>  1/1    Running  0

   my-connect-cluster is the name of the Kafka Connect cluster.

   A pod ID identifies each pod created.

   With the default deployment, you create a single Kafka Connect pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. Configure the KafkaConnect custom resource by specifying the container registry in .spec.build.output, and additional connectors in .spec.build.plugins:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnect
   metadata:
     name: my-connect-cluster
   spec: 1
     #...
     build:
       output: 2
         type: docker
         image: my-registry.io/my-org/my-connect-cluster:latest
         pushSecret: my-registry-credentials
       plugins: 3
         - name: connector-1
           artifacts:
             - type: tgz
               url: <url_to_download_connector_1_artifact>
               sha512sum: <SHA-512_checksum_of_connector_1_artifact>
         - name: connector-2
           artifacts:
             - type: jar
               url: <url_to_download_connector_2_artifact>
               sha512sum: <SHA-512_checksum_of_connector_2_artifact>
     #...

   1

   The specification for the Kafka Connect cluster.

   2

   (Required) Configuration of the container registry where new images are pushed.

   3

   (Required) List of connector plugins and their artifacts to add to the new container image. Each plugin must be configured with at least one artifact.

2. Create or update the resource:

   $ oc apply -f <kafka_connect_configuration_file>

3. Wait for the new container image to build, and for the Kafka Connect cluster to be deployed.
4. Use the Kafka Connect REST API or KafkaConnector custom resources to use the connector plugins you added.

Procedure

1. Create a new Dockerfile using registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0 as the base image:

   FROM registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0
   USER root:root
   COPY ./my-plugins/ /opt/kafka/plugins/
   USER 1001

   Example plugins file

   $ tree ./my-plugins/
   ./my-plugins/
   ├── debezium-connector-mongodb
   │   ├── bson-<version>.jar
   │   ├── CHANGELOG.md
   │   ├── CONTRIBUTE.md
   │   ├── COPYRIGHT.txt
   │   ├── debezium-connector-mongodb-<version>.jar
   │   ├── debezium-core-<version>.jar
   │   ├── LICENSE.txt
   │   ├── mongodb-driver-core-<version>.jar
   │   ├── README.md
   │   └── # ...
   ├── debezium-connector-mysql
   │   ├── CHANGELOG.md
   │   ├── CONTRIBUTE.md
   │   ├── COPYRIGHT.txt
   │   ├── debezium-connector-mysql-<version>.jar
   │   ├── debezium-core-<version>.jar
   │   ├── LICENSE.txt
   │   ├── mysql-binlog-connector-java-<version>.jar
   │   ├── mysql-connector-java-<version>.jar
   │   ├── README.md
   │   └── # ...
   └── debezium-connector-postgres
       ├── CHANGELOG.md
       ├── CONTRIBUTE.md
       ├── COPYRIGHT.txt
       ├── debezium-connector-postgres-<version>.jar
       ├── debezium-core-<version>.jar
       ├── LICENSE.txt
       ├── postgresql-<version>.jar
       ├── protobuf-java-<version>.jar
       ├── README.md
       └── # ...

   The COPY command points to the plugin files to copy to the container image.

   This example adds plugins for Debezium connectors (MongoDB, MySQL, and PostgreSQL), though not all files are listed for brevity. Debezium running in Kafka Connect looks the same as any other Kafka Connect task.

2. Build the container image.
3. Push your custom image to your container registry.

4. Point to the new container image.

   You can point to the image in one of the following ways:

   • Edit the KafkaConnect.spec.image property of the KafkaConnect custom resource.

     If set, this property overrides the STRIMZI_KAFKA_CONNECT_IMAGES environment variable in the Cluster Operator.

     apiVersion: kafka.strimzi.io/v1beta2
     kind: KafkaConnect
     metadata:
       name: my-connect-cluster
     spec: 1
       #...
       image: my-new-container-image 2
       config: 3
         #...

     1

     The specification for the Kafka Connect cluster.

     2

     The docker image for Kafka Connect pods.

     3

     Configuration of the Kafka Connect workers (not connectors).

   • Edit the STRIMZI_KAFKA_CONNECT_IMAGES environment variable in the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to point to the new container image, and then reinstall the Cluster Operator.

Procedure

1. Add the FileStreamSourceConnector and FileStreamSinkConnector plugins to Kafka Connect in one of the following ways:

   • Configure Kafka Connect to build a new container image with plugins automatically
   • Create a Docker image from the base Kafka Connect image (manually or using continuous integration)

2. Set the strimzi.io/use-connector-resources annotation to true in the Kafka Connect configuration.

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnect
   metadata:
     name: my-connect-cluster
     annotations:
       strimzi.io/use-connector-resources: "true"
   spec:
       # ...

   With the KafkaConnector resources enabled, the Cluster Operator watches for them.

3. Edit the examples/connect/source-connector.yaml file:

   Example KafkaConnector source connector configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-source-connector  1
     labels:
       strimzi.io/cluster: my-connect-cluster 2
   spec:
     class: org.apache.kafka.connect.file.FileStreamSourceConnector 3
     tasksMax: 2 4
     autoRestart: 5
       enabled: true
     config: 6
       file: "/opt/kafka/LICENSE" 7
       topic: my-topic 8
       # ...

   1

   Name of the KafkaConnector resource, which is used as the name of the connector. Use any name that is valid for an OpenShift resource.

   2

   Name of the Kafka Connect cluster to create the connector instance in. Connectors must be deployed to the same namespace as the Kafka Connect cluster they link to.

   3

   Full name of the connector class. This should be present in the image being used by the Kafka Connect cluster.

   4

   Maximum number of Kafka Connect tasks that the connector can create.

   5

   Enables automatic restarts of failed connectors and tasks. By default, the number of restarts is indefinite, but you can set a maximum on the number of automatic restarts using the maxRestarts property.

   6

   Connector configuration as key-value pairs.

   7

   Location of the external data file. In this example, we’re configuring the FileStreamSourceConnector to read from the /opt/kafka/LICENSE file.

   8

   Kafka topic to publish the source data to.

4. Create the source KafkaConnector in your OpenShift cluster:

   oc apply -f examples/connect/source-connector.yaml

5. Create an examples/connect/sink-connector.yaml file:

   touch examples/connect/sink-connector.yaml

6. Paste the following YAML into the sink-connector.yaml file:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-sink-connector
     labels:
       strimzi.io/cluster: my-connect
   spec:
     class: org.apache.kafka.connect.file.FileStreamSinkConnector 1
     tasksMax: 2
     config: 2
       file: "/tmp/my-file" 3
       topics: my-topic 4

   1

   Full name or alias of the connector class. This should be present in the image being used by the Kafka Connect cluster.

   2

   Connector configuration as key-value pairs.

   3

   Temporary file to publish the source data to.

   4

   Kafka topic to read the source data from.

7. Create the sink KafkaConnector in your OpenShift cluster:

   oc apply -f examples/connect/sink-connector.yaml

8. Check that the connector resources were created:

   oc get kctr --selector strimzi.io/cluster=<my_connect_cluster> -o name
   
   my-source-connector
   my-sink-connector

   Replace <my_connect_cluster> with the name of your Kafka Connect cluster.

9. In the container, execute kafka-console-consumer.sh to read the messages that were written to the topic by the source connector:

   oc exec <my_kafka_cluster>-kafka-0 -i -t -- bin/kafka-console-consumer.sh --bootstrap-server <my_kafka_cluster>-kafka-bootstrap.NAMESPACE.svc:9092 --topic my-topic --from-beginning

   Replace <my_kafka_cluster> with the name of your Kafka cluster.

1. Deploy KafkaConnector resources with the configuration to create your connector instances.
2. Enable KafkaConnector resources in your Kafka Connect configuration by setting the strimzi.io/use-connector-resources annotation to true.

Procedure

1. Deploy Kafka MirrorMaker to your OpenShift cluster:

   For MirrorMaker:

   oc apply -f examples/mirror-maker/kafka-mirror-maker.yaml

   For MirrorMaker 2:

   oc apply -f examples/mirror-maker/kafka-mirror-maker-2.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                                    READY  STATUS   RESTARTS
   my-mirror-maker-mirror-maker-<pod_id>   1/1    Running  1
   my-mm2-cluster-mirrormaker2-<pod_id>    1/1    Running  1

   my-mirror-maker is the name of the Kafka MirrorMaker cluster. my-mm2-cluster is the name of the Kafka MirrorMaker 2 cluster.

   A pod ID identifies each pod created.

   With the default deployment, you install a single MirrorMaker or MirrorMaker 2 pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. Deploy Kafka Bridge to your OpenShift cluster:

   oc apply -f examples/bridge/kafka-bridge.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                       READY  STATUS   RESTARTS
   my-bridge-bridge-<pod_id>  1/1    Running  0

   my-bridge is the name of the Kafka Bridge cluster.

   A pod ID identifies each pod created.

   With the default deployment, you install a single Kafka Bridge pod.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

Procedure

1. List the names of the pods in your OpenShift cluster:

   oc get pods -o name
   
   pod/kafka-consumer
   # ...
   pod/my-bridge-bridge-<pod_id>

2. Connect to the Kafka Bridge pod on port 8080:

   oc port-forward pod/my-bridge-bridge-<pod_id> 8080:8080 &

   Note

   If port 8080 on your local machine is already in use, use an alternative HTTP port, such as 8008.

Procedure

1. Edit the env properties in the install/topic-operator/05-Deployment-strimzi-topic-operator.yaml standalone deployment file.

   Example standalone Topic Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-topic-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-topic-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 1
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 2
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_RESOURCE_LABELS 3
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 4
                 value: "120000"
               - name: STRIMZI_LOG_LEVEL 5
                 value: INFO
               - name: STRIMZI_TLS_ENABLED 6
                 value: "false"
               - name: STRIMZI_JAVA_OPTS 7
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 8
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_PUBLIC_CA 9
                 value: "false"
               - name: STRIMZI_TLS_AUTH_ENABLED 10
                 value: "false"
               - name: STRIMZI_SASL_ENABLED 11
                 value: "false"
               - name: STRIMZI_SASL_USERNAME 12
                 value: "admin"
               - name: STRIMZI_SASL_PASSWORD 13
                 value: "password"
               - name: STRIMZI_SASL_MECHANISM 14
                 value: "scram-sha-512"
               - name: STRIMZI_SECURITY_PROTOCOL 15
                 value: "SSL"
               - name: STRIMZI_USE_FINALIZERS
                 value: "false" 16

   1

   The OpenShift namespace for the Topic Operator to watch for KafkaTopic resources. Specify the namespace of the Kafka cluster.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The label to identify the KafkaTopic resources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaTopic resource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.

   4

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes).

   5

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   6

   Enables TLS support for encrypted communication with the Kafka brokers.

   7

   (Optional) The Java options used by the JVM running the Topic Operator.

   8

   (Optional) The debugging (-D) options set for the Topic Operator.

   9

   (Optional) Skips the generation of trust store certificates if TLS is enabled through STRIMZI_TLS_ENABLED. If this environment variable is enabled, the brokers must use a public trusted certificate authority for their TLS certificates. The default is false.

   10

   (Optional) Generates key store certificates for mTLS authentication. Setting this to false disables client authentication with mTLS to the Kafka brokers. The default is true.

   11

   (Optional) Enables SASL support for client authentication when connecting to Kafka brokers. The default is false.

   12

   (Optional) The SASL username for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   13

   (Optional) The SASL password for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   14

   (Optional) The SASL mechanism for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED. You can set the value to plain, scram-sha-256, or scram-sha-512.

   15

   (Optional) The security protocol used for communication with Kafka brokers. The default value is "PLAINTEXT". You can set the value to PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL.

   16

   Set STRIMZI_USE_FINALIZERS to false if you do not want to use finalizers to control topic deletion.

2. If you want to connect to Kafka brokers that are using certificates from a public certificate authority, set STRIMZI_PUBLIC_CA to true. Set this property to true, for example, if you are using Amazon AWS MSK service.

3. If you enabled mTLS with the STRIMZI_TLS_ENABLED environment variable, specify the keystore and truststore used to authenticate connection to the Kafka cluster.

   Example mTLS configuration

   # ....
   env:
     - name: STRIMZI_TRUSTSTORE_LOCATION 1
       value: "/path/to/truststore.p12"
     - name: STRIMZI_TRUSTSTORE_PASSWORD 2
       value: "TRUSTSTORE-PASSWORD"
     - name: STRIMZI_KEYSTORE_LOCATION 3
       value: "/path/to/keystore.p12"
     - name: STRIMZI_KEYSTORE_PASSWORD 4
       value: "KEYSTORE-PASSWORD"
   # ...

   1

   The truststore contains the public keys of the Certificate Authorities used to sign the Kafka and ZooKeeper server certificates.

   2

   The password for accessing the truststore.

   3

   The keystore contains the private key for mTLS authentication.

   4

   The password for accessing the keystore.

4. Apply the changes to the Deployment configuration to deploy the Topic Operator.

5. Check the status of the deployment:

   oc get deployments

   Output shows the deployment name and readiness

   NAME                    READY  UP-TO-DATE  AVAILABLE
   strimzi-topic-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the following env properties in the install/user-operator/05-Deployment-strimzi-user-operator.yaml standalone deployment file.

   Example standalone User Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-user-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-user-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 1
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 2
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_CA_CERT_NAME 3
                 value: my-cluster-clients-ca-cert
               - name: STRIMZI_CA_KEY_NAME 4
                 value: my-cluster-clients-ca
               - name: STRIMZI_LABELS 5
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 6
                 value: "120000"
               - name: STRIMZI_WORK_QUEUE_SIZE 7
                 value: 10000
               - name: STRIMZI_CONTROLLER_THREAD_POOL_SIZE 8
                 value: 10
               - name: STRIMZI_USER_OPERATIONS_THREAD_POOL_SIZE 9
                 value: 4
               - name: STRIMZI_LOG_LEVEL 10
                 value: INFO
               - name: STRIMZI_GC_LOG_ENABLED 11
                 value: "true"
               - name: STRIMZI_CA_VALIDITY 12
                 value: "365"
               - name: STRIMZI_CA_RENEWAL 13
                 value: "30"
               - name: STRIMZI_JAVA_OPTS 14
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 15
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_SECRET_PREFIX 16
                 value: "kafka-"
               - name: STRIMZI_ACLS_ADMIN_API_SUPPORTED 17
                 value: "true"
               - name: STRIMZI_MAINTENANCE_TIME_WINDOWS 18
                 value: '* * 8-10 * * ?;* * 14-15 * * ?'
               - name: STRIMZI_KAFKA_ADMIN_CLIENT_CONFIGURATION 19
                 value: |
                   default.api.timeout.ms=120000
                   request.timeout.ms=60000

   1

   The OpenShift namespace for the User Operator to watch for KafkaUser resources. Only one namespace can be specified.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The OpenShift Secret that contains the public key (ca.crt) value of the CA (certificate authority) that signs new user certificates for mTLS authentication.

   4

   The OpenShift Secret that contains the private key (ca.key) value of the CA that signs new user certificates for mTLS authentication.

   5

   The label to identify the KafkaUser resources managed by the User Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaUser resource. If you deploy more than one User Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.

   6

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes).

   7

   The size of the controller event queue. The size of the queue should be at least as big as the maximal amount of users you expect the User Operator to operate. The default is 1024.

   8

   The size of the worker pool for reconciling the users. Bigger pool might require more resources, but it will also handle more KafkaUser resources The default is 50.

   9

   The size of the worker pool for Kafka Admin API and OpenShift operations. Bigger pool might require more resources, but it will also handle more KafkaUser resources The default is 4.

   10

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   11

   Enables garbage collection (GC) logging. The default is true.

   12

   The validity period for the CA. The default is 365 days.

   13

   The renewal period for the CA. The renewal period is measured backwards from the expiry date of the current certificate. The default is 30 days to initiate certificate renewal before the old certificates expire.

   14

   (Optional) The Java options used by the JVM running the User Operator

   15

   (Optional) The debugging (-D) options set for the User Operator

   16

   (Optional) Prefix for the names of OpenShift secrets created by the User Operator.

   17

   (Optional) Indicates whether the Kafka cluster supports management of authorization ACL rules using the Kafka Admin API. When set to false, the User Operator will reject all resources with simple authorization ACL rules. This helps to avoid unnecessary exceptions in the Kafka cluster logs. The default is true.

   18

   (Optional) Semi-colon separated list of Cron Expressions defining the maintenance time windows during which the expiring user certificates will be renewed.

   19

   (Optional) Configuration options for configuring the Kafka Admin client used by the User Operator in the properties format.

2. If you are using mTLS to connect to the Kafka cluster, specify the secrets used to authenticate connection. Otherwise, go to the next step.

   Example mTLS configuration

   # ....
   env:
     - name: STRIMZI_CLUSTER_CA_CERT_SECRET_NAME 1
       value: my-cluster-cluster-ca-cert
     - name: STRIMZI_EO_KEY_SECRET_NAME 2
       value: my-cluster-entity-operator-certs
   # ..."

   1

   The OpenShift Secret that contains the public key (ca.crt) value of the CA that signs Kafka broker certificates.

   2

   The OpenShift Secret that contains the certificate public key (entity-operator.crt) and private key (entity-operator.key) that is used for mTLS authentication against the Kafka cluster.

3. Deploy the User Operator.

   oc create -f install/user-operator

4. Check the status of the deployment:

   oc get deployments

   Output shows the deployment name and readiness

   NAME                   READY  UP-TO-DATE  AVAILABLE
   strimzi-user-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Add the plugin properties to the config of the Kafka resource.

   The plugin properties are shown in this example configuration.

   Example Kafka Static Quota plugin configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     kafka:
       # ...
       config:
         client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback 1
         client.quota.callback.static.produce: 1000000 2
         client.quota.callback.static.fetch: 1000000 3
         client.quota.callback.static.storage.soft: 400000000000 4
         client.quota.callback.static.storage.hard: 500000000000 5
         client.quota.callback.static.storage.check-interval: 5 6

   1

   Loads the Kafka Static Quota plugin.

   2

   Sets the producer byte-rate threshold. 1 MBps in this example.

   3

   Sets the consumer byte-rate threshold. 1 MBps in this example.

   4

   Sets the lower soft limit for storage. 400 GB in this example.

   5

   Sets the higher hard limit for storage. 500 GB in this example.

   6

   Sets the interval in seconds between checks on storage. 5 seconds in this example. You can set this to 0 to disable the check.

2. Update the resource.

   oc apply -f <kafka_configuration_file>

Procedure

1. Find the name of the Pod that you want to delete.

   Kafka broker pods are named <cluster_name>-kafka-<index_number>, where <index_number> starts at zero and ends at the total number of replicas minus one. For example, my-cluster-kafka-0.

2. Use oc annotate to annotate the Pod resource in OpenShift:

   oc annotate pod <cluster_name>-kafka-<index_number> strimzi.io/delete-pod-and-pvc="true"

3. Wait for the next reconciliation, when the annotated pod with the underlying persistent volume claim will be deleted and then recreated.

Procedure

1. Find the name of the Pod that you want to delete.

   ZooKeeper pods are named <cluster_name>-zookeeper-<index_number>, where <index_number> starts at zero and ends at the total number of replicas minus one. For example, my-cluster-zookeeper-0.

2. Use oc annotate to annotate the Pod resource in OpenShift:

   oc annotate pod <cluster_name>-zookeeper-<index_number> strimzi.io/delete-pod-and-pvc="true"

3. Wait for the next reconciliation, when the annotated pod with the underlying persistent volume claim will be deleted and then recreated.

Procedure

1. Annotate the node pool with the IDs to use when scaling up or scaling down, as shown in the following examples.

   IDs for scaling up are assigned to node pool pool-a:

   Assigning IDs for scaling up

   oc annotate kafkanodepool pool-a strimzi.io/next-node-ids="[0,1,2,10-20,30]"

   The lowest available ID from this range is used when adding a node to pool-a.

   IDs for scaling down are assigned to node pool pool-b:

   Assigning IDs for scaling down

   oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[60-50,9,8,7]"

   The highest available ID from this range is removed when scaling down pool-b.

   Note

   If you want to remove a specific node, you can assign a single node ID to the scaling down annotation: oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids="[3]".

2. You can now scale the node pool.

   For more information, see the following:

   • Section 9.3.3, “Adding nodes to a node pool”
   • Section 9.3.4, “Removing nodes from a node pool”
   • Section 9.3.5, “Moving nodes between node pools”

   On reconciliation, a warning is given if the annotations are misformatted.

3. After you have performed the scaling operation, you can remove the annotation if it’s no longer needed.

   Removing the annotation for scaling up

   oc annotate kafkanodepool pool-a strimzi.io/next-node-ids-

   Removing the annotation for scaling down

   oc annotate kafkanodepool pool-b strimzi.io/remove-node-ids-

Procedure

1. Create a new node in the node pool.

   For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=4

2. Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows four Kafka nodes in the node pool

   NAME                 READY  STATUS   RESTARTS
   my-cluster-pool-a-0  1/1    Running  0
   my-cluster-pool-a-1  1/1    Running  0
   my-cluster-pool-a-2  1/1    Running  0
   my-cluster-pool-a-3  1/1    Running  0

3. Reassign the partitions after increasing the number of nodes in the node pool.

   After scaling up a node pool, use the Cruise Control add-brokers mode to move partition replicas from existing brokers to the newly added brokers.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: add-brokers
     brokers: [3]

   We are reassigning partitions to node my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.

Procedure

1. Reassign the partitions before decreasing the number of nodes in the node pool.

   Before scaling down a node pool, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: remove-brokers
     brokers: [3]

   We are reassigning partitions from node my-cluster-pool-a-3. The reassignment can take some time depending on the number of topics and partitions in the cluster.

2. After the reassignment process is complete, and the node being removed has no live partitions, reduce the number of Kafka nodes in the node pool.

   For example, node pool pool-a has four replicas. We remove a node by decreasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=3

   Output shows three Kafka nodes in the node pool

   NAME                       READY  STATUS   RESTARTS
   my-cluster-pool-b-kafka-0  1/1    Running  0
   my-cluster-pool-b-kafka-1  1/1    Running  0
   my-cluster-pool-b-kafka-2  1/1    Running  0

Procedure

1. Create a new node in the target node pool.

   For example, node pool pool-a has three replicas. We add a node by increasing the number of replicas:

   oc scale kafkanodepool pool-a --replicas=4

2. Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows four Kafka nodes in the source and target node pools

   NAME                 READY  STATUS   RESTARTS
   my-cluster-pool-a-0  1/1    Running  0
   my-cluster-pool-a-1  1/1    Running  0
   my-cluster-pool-a-4  1/1    Running  0
   my-cluster-pool-a-7  1/1    Running  0
   my-cluster-pool-b-2  1/1    Running  0
   my-cluster-pool-b-3  1/1    Running  0
   my-cluster-pool-b-5  1/1    Running  0
   my-cluster-pool-b-6  1/1    Running  0

   Node IDs are appended to the name of the node on creation. We add node my-cluster-pool-a-7, which has a node ID of 7.

3. Reassign the partitions from the old node to the new node.

   Before scaling down the source node pool, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: remove-brokers
     brokers: [6]

   We are reassigning partitions from node my-cluster-pool-b-6. The reassignment can take some time depending on the number of topics and partitions in the cluster.

4. After the reassignment process is complete, reduce the number of Kafka nodes in the source node pool.

   For example, node pool pool-b has four replicas. We remove a node by decreasing the number of replicas:

   oc scale kafkanodepool pool-b --replicas=3

   The node with the highest ID (6) within the pool is removed.

   Output shows three Kafka nodes in the source node pool

   NAME                       READY  STATUS   RESTARTS
   my-cluster-pool-b-kafka-2  1/1    Running  0
   my-cluster-pool-b-kafka-3  1/1    Running  0
   my-cluster-pool-b-kafka-5  1/1    Running  0

Procedure

1. Create a node pool with a broker role.

   Example node pool configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-b
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - broker
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false
     # ...

   The new node pool also has three nodes. If you already have a broker-only node pool, you can skip this step.

2. Apply the new KafkaNodePool resource to create the brokers.

3. Check the status of the deployment and wait for the pods in the node pool to be created and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows pods running in two node pools

   NAME                 READY  STATUS   RESTARTS
   my-cluster-pool-a-0  1/1    Running  0
   my-cluster-pool-a-1  1/1    Running  0
   my-cluster-pool-a-2  1/1    Running  0
   my-cluster-pool-b-3  1/1    Running  0
   my-cluster-pool-b-4  1/1    Running  0
   my-cluster-pool-b-5  1/1    Running  0

   Node IDs are appended to the name of the node on creation.

4. Use the Cruise Control remove-brokers mode to reassign partition replicas from the dual-role nodes to the newly added brokers.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: remove-brokers
     brokers: [0, 1, 2]

   The reassignment can take some time depending on the number of topics and partitions in the cluster.

   Note

   If nodes changing to controller-only roles have any assigned partitions, the change is prevented. The status.conditions of the Kafka resource provide details of events preventing the change.

5. Remove the broker role from the node pool that originally had a combined role.

   Dual-role nodes switched to controllers

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-a
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - controller
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 20Gi
           deleteClaim: false
     # ...

6. Apply the configuration change so that the node pool switches to a controller-only role.

Procedure

1. Edit the node pool pool-a and add the broker role to it.

   Example node pool configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-a
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - controller
       - broker
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false
     # ...

2. Check the status and wait for the pods in the node pool to be restarted and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows pods running in two node pools

   NAME                 READY  STATUS   RESTARTS
   my-cluster-pool-a-0  1/1    Running  0
   my-cluster-pool-a-1  1/1    Running  0
   my-cluster-pool-a-2  1/1    Running  0
   my-cluster-pool-b-3  1/1    Running  0
   my-cluster-pool-b-4  1/1    Running  0
   my-cluster-pool-b-5  1/1    Running  0

   Node IDs are appended to the name of the node on creation.

3. Use the Cruise Control remove-brokers mode to reassign partition replicas from the broker-only nodes to the dual-role nodes.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: remove-brokers
     brokers: [3, 4, 5]

   The reassignment can take some time depending on the number of topics and partitions in the cluster.

4. Remove the pool-b node pool that has the old broker-only nodes.

   oc delete kafkanodepool pool-b -n <my_cluster_operator_namespace>

Procedure

1. Create the node pool with its own storage settings.

   For example, node pool pool-a uses JBOD storage with persistent volumes:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-a
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 500Gi
           class: gp2-ebs
     # ...

   Nodes in pool-a are configured to use Amazon EBS (Elastic Block Store) GP2 volumes.

2. Apply the node pool configuration for pool-a.

3. Check the status of the deployment and wait for the pods in pool-a to be created and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows three Kafka nodes in the node pool

   NAME                 READY  STATUS   RESTARTS
   my-cluster-pool-a-0  1/1    Running  0
   my-cluster-pool-a-1  1/1    Running  0
   my-cluster-pool-a-2  1/1    Running  0

4. To migrate to a new storage class, create a new node pool with the required storage configuration:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-b
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     roles:
       - broker
     replicas: 3
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 1Ti
           class: gp3-ebs
     # ...

   Nodes in pool-b are configured to use Amazon EBS (Elastic Block Store) GP3 volumes.

5. Apply the node pool configuration for pool-b.
6. Check the status of the deployment and wait for the pods in pool-b to be created and ready.

7. Reassign the partitions from pool-a to pool-b.

   When migrating to a new storage configuration, use the Cruise Control remove-brokers mode to move partition replicas off the brokers that are going to be removed.

   Using Cruise Control to reassign partition replicas

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaRebalance
   metadata:
     # ...
   spec:
     mode: remove-brokers
     brokers: [0, 1, 2]

   We are reassigning partitions from pool-a. The reassignment can take some time depending on the number of topics and partitions in the cluster.

8. After the reassignment process is complete, delete the old node pool:

   oc delete kafkanodepool pool-a

Procedure

1. Define the storage class for use with each availability zone:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: all-zones
   provisioner: kubernetes.io/my-storage
   parameters:
     type: ssd
   volumeBindingMode: WaitForFirstConsumer

2. Create node pools representing the two availability zones, specifying the all-zones storage class and the affinity for each zone:

   Node pool configuration for zone-1

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-zone-1
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 500Gi
           class: all-zones
     template:
       pod:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
                 - matchExpressions:
                   - key: topology.kubernetes.io/zone
                     operator: In
                     values:
                     - zone-1
     # ...

   Node pool configuration for zone-2

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: pool-zone-2
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 4
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 500Gi
           class: all-zones
     template:
       pod:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
                 - matchExpressions:
                   - key: topology.kubernetes.io/zone
                     operator: In
                     values:
                     - zone-2
     # ...

3. Apply the node pool configuration.

4. Check the status of the deployment and wait for the pods in the node pools to be created and ready (1/1).

   oc get pods -n <my_cluster_operator_namespace>

   Output shows 3 Kafka nodes in pool-zone-1 and 4 Kafka nodes in pool-zone-2

   NAME                            READY  STATUS   RESTARTS
   my-cluster-pool-zone-1-kafka-0  1/1    Running  0
   my-cluster-pool-zone-1-kafka-1  1/1    Running  0
   my-cluster-pool-zone-1-kafka-2  1/1    Running  0
   my-cluster-pool-zone-2-kafka-3  1/1    Running  0
   my-cluster-pool-zone-2-kafka-4  1/1    Running  0
   my-cluster-pool-zone-2-kafka-5  1/1    Running  0
   my-cluster-pool-zone-2-kafka-6  1/1    Running  0

Procedure

1. Create a new KafkaNodePool resource.

   1. Name the resource kafka.
   2. Point a strimzi.io/cluster label to your existing Kafka resource.
   3. Set the replica count and storage configuration to match your current Kafka cluster.
   4. Set the roles to broker.

   Example configuration for a node pool used in migrating a Kafka cluster

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: kafka
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - broker
     storage:
       type: jbod
       volumes:
         - id: 0
           type: persistent-claim
           size: 100Gi
           deleteClaim: false

   Warning

   To preserve cluster data and the names of its nodes and resources, the node pool name must be kafka, and the strimzi.io/cluster label must match the Kafka resource name. Otherwise, nodes and resources are created with new names, including the persistent volume storage used by the nodes. Consequently, your previous data may not be available.

2. Apply the KafkaNodePool resource:

   oc apply -f <node_pool_configuration_file>

   By applying this resource, you switch Kafka to using node pools.

   There is no change or rolling update and resources are identical to how they were before.

3. Enable support for node pools in the Kafka resource using the strimzi.io/node-pools: enabled annotation.

   Example configuration for a node pool in a cluster using ZooKeeper

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     annotations:
       strimzi.io/node-pools: enabled
   spec:
     kafka:
       # ...
     zookeeper:
       # ...

4. Apply the Kafka resource:

   oc apply -f <kafka_configuration_file>

   There is no change or rolling update. The resources remain identical to how they were before.

5. Remove the replicated properties from the Kafka custom resource. When the KafkaNodePool resource is in use, you can remove the properties that you copied to the KafkaNodePool resource, such as the .spec.kafka.replicas and .spec.kafka.storage properties.

1. If you have multiple node pools, consolidate them into a single KafkaNodePool named kafka with node IDs from 0 to N (where N is the number of replicas).
2. Ensure that the .spec.kafka configuration in the Kafka resource matches the KafkaNodePool configuration, including storage, resources, and replicas.
3. Disable support for node pools in the Kafka resource using the strimzi.io/node-pools: disabled annotation.
4. Delete the Kafka node pool named kafka.

Procedure

1. Annotate the custom resource in OpenShift, setting pause-reconciliation to true:

   oc annotate <kind_of_custom_resource> <name_of_custom_resource> strimzi.io/pause-reconciliation="true"

   For example, for the KafkaConnect custom resource:

   oc annotate KafkaConnect my-connect strimzi.io/pause-reconciliation="true"

2. Check that the status conditions of the custom resource show a change to ReconciliationPaused:

   oc describe <kind_of_custom_resource> <name_of_custom_resource>

   The type condition changes to ReconciliationPaused at the lastTransitionTime.

   Example custom resource with a paused reconciliation condition type

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnect
   metadata:
     annotations:
       strimzi.io/pause-reconciliation: "true"
       strimzi.io/use-connector-resources: "true"
     creationTimestamp: 2021-03-12T10:47:11Z
     #...
   spec:
     # ...
   status:
     conditions:
     - lastTransitionTime: 2021-03-12T10:47:41.689249Z
       status: "True"
       type: ReconciliationPaused

Procedure

1. To add proxy environment variables to the Cluster Operator, update its Deployment configuration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml).

   Example proxy configuration for the Cluster Operator

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
           # ...
           env:
           # ...
           - name: "HTTP_PROXY"
             value: "http://proxy.com" 1
           - name: "HTTPS_PROXY"
             value: "https://proxy.com" 2
           - name: "NO_PROXY"
             value: "internal.com, other.domain.com" 3
     # ...

   1

   Address of the proxy server.

   2

   Secure address of the proxy server.

   3

   Addresses for servers that are accessed directly as exceptions to the proxy server. The URLs are comma-separated.

   Alternatively, edit the Deployment directly:

   oc edit deployment strimzi-cluster-operator

2. If you updated the YAML file instead of editing the Deployment directly, apply the changes:

   oc create -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

Procedure

1. To disable the FIPS mode in the Cluster Operator, update its Deployment configuration (install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml) and add the FIPS_MODE environment variable.

   Example FIPS configuration for the Cluster Operator

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
           # ...
           env:
           # ...
           - name: "FIPS_MODE"
             value: "disabled" 1
     # ...

   1

   Disables the FIPS mode.

   Alternatively, edit the Deployment directly:

   oc edit deployment strimzi-cluster-operator

2. If you updated the YAML file instead of editing the Deployment directly, apply the changes:

   oc apply -f install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml

Procedure

1. Edit the authorization property in the KafkaUser resource to provide access rights to the user.

   Access rights are configured for the Kafka Connect topics and cluster group using literal name values. The following table shows the default names configured for the topics and cluster group ID.

   Table 9.2. Names for the access rights configuration

   Property	Name
   offset.storage.topic	connect-cluster-offsets
   status.storage.topic	connect-cluster-status
   config.storage.topic	connect-cluster-configs
   group	connect-cluster

   In this example configuration, the default names are used to specify access rights. If you are using different names for a Kafka Connect instance, use those names in the ACLs configuration.

   Example configuration for simple authorization

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaUser
   metadata:
     name: my-user
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     # ...
     authorization:
       type: simple
       acls:
         # access to offset.storage.topic
         - resource:
             type: topic
             name: connect-cluster-offsets
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # access to status.storage.topic
         - resource:
             type: topic
             name: connect-cluster-status
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # access to config.storage.topic
         - resource:
             type: topic
             name: connect-cluster-configs
             patternType: literal
           operations:
             - Create
             - Describe
             - Read
             - Write
           host: "*"
         # cluster group
         - resource:
             type: group
             name: connect-cluster
             patternType: literal
           operations:
             - Read
           host: "*"

2. Create or update the resource.

   oc apply -f KAFKA-USER-CONFIG-FILE

Procedure

1. Find the name of the KafkaConnector custom resource that controls the connector you want to pause or stop:

   oc get KafkaConnector

2. Edit the KafkaConnector resource to stop or pause the connector.

   Example configuration for stopping a Kafka Connect connector

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-source-connector
     labels:
       strimzi.io/cluster: my-connect-cluster
   spec:
     class: org.apache.kafka.connect.file.FileStreamSourceConnector
     tasksMax: 2
     config:
       file: "/opt/kafka/LICENSE"
       topic: my-topic
     state: stopped
     # ...

   Change the state configuration to stopped or paused. The default state for the connector when this property is not set is running.

3. Apply the changes to the KafkaConnector configuration.

   You can resume the connector by changing state to running or removing the configuration.

Procedure

1. Find the name of the KafkaConnector custom resource that controls the Kafka connector you want to restart:

   oc get KafkaConnector

2. Restart the connector by annotating the KafkaConnector resource in OpenShift:

   oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart="true"

   The restart annotation is set to true.

3. Wait for the next reconciliation to occur (every two minutes by default).

   The Kafka connector is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the KafkaConnector custom resource.

Procedure

1. Find the name of the KafkaConnector custom resource that controls the Kafka connector task you want to restart:

   oc get KafkaConnector

2. Find the ID of the task to be restarted from the KafkaConnector custom resource:

   oc describe KafkaConnector <kafka_connector_name>

   Task IDs are non-negative integers, starting from 0.

3. Use the ID to restart the connector task by annotating the KafkaConnector resource in OpenShift:

   oc annotate KafkaConnector <kafka_connector_name> strimzi.io/restart-task="0"

   In this example, task 0 is restarted.

4. Wait for the next reconciliation to occur (every two minutes by default).

   The Kafka connector task is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the KafkaConnector custom resource.