Chapter 6. Deploying AMQ Streams using installation artifacts

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

3. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to add a list of all the namespaces the Cluster Operator will watch to the STRIMZI_NAMESPACE environment variable.

   For example, in this procedure the Cluster Operator will watch the namespaces watched-namespace-1, watched-namespace-2, watched-namespace-3.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq7/amq-streams-rhel8-operator:2.3.0
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: watched-namespace-1,watched-namespace-2,watched-namespace-3

3. For each namespace listed, install the RoleBindings.

   In this example, we replace watched-namespace in these commands with the namespaces listed in the previous step, repeating them for watched-namespace-1, watched-namespace-2, watched-namespace-3:

   oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace>
   oc create -f install/cluster-operator/023-RoleBinding-strimzi-cluster-operator.yaml -n <watched_namespace>
   oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n <watched_namespace>

4. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to set the value of the STRIMZI_NAMESPACE environment variable to *.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         # ...
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq7/amq-streams-rhel8-operator:2.3.0
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: "*"
           # ...

3. Create ClusterRoleBindings that grant cluster-wide access for all namespaces to the Cluster Operator.

   oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
   oc create clusterrolebinding strimzi-cluster-operator-watched --clusterrole=strimzi-cluster-operator-watched --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
   oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator

4. Deploy the Cluster Operator to your OpenShift cluster.

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Check the status of the deployment:

   oc get deployments -n my-cluster-operator-namespace

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   strimzi-cluster-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Create and deploy an ephemeral or persistent cluster.

   • To create and deploy an ephemeral cluster:

     oc apply -f examples/kafka/kafka-ephemeral.yaml

   • To create and deploy a persistent cluster:

     oc apply -f examples/kafka/kafka-persistent.yaml

2. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod names and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   my-cluster-kafka-0          1/1     Running   0
   my-cluster-kafka-1          1/1     Running   0
   my-cluster-kafka-2          1/1     Running   0
   my-cluster-zookeeper-0      1/1     Running   0
   my-cluster-zookeeper-1      1/1     Running   0
   my-cluster-zookeeper-2      1/1     Running   0

   my-cluster is the name of the Kafka cluster.

   With the default deployment, you install an Entity Operator cluster, 3 Kafka pods, and 3 ZooKeeper pods.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS shows as Running.

Procedure

1. Edit the entityOperator properties of the Kafka resource to include topicOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the Topic Operator spec using the properties described in EntityTopicOperatorSpec schema reference.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   oc apply -f <kafka_configuration_file>

4. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod name and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   # ...

   my-cluster is the name of the Kafka cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS shows as Running.

Procedure

1. Edit the entityOperator properties of the Kafka resource to include userOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the User Operator spec using the properties described in EntityUserOperatorSpec schema reference.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   oc apply -f <kafka_configuration_file>

4. Check the status of the deployment:

   oc get pods -n <my_cluster_operator_namespace>

   Output shows the pod name and readiness

   NAME                        READY   STATUS    RESTARTS
   my-cluster-entity-operator  3/3     Running   0
   # ...

   my-cluster is the name of the Kafka cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS shows as Running.

Procedure

1. Deploy Kafka Connect to your OpenShift cluster. Use the examples/connect/kafka-connect.yaml file to deploy Kafka Connect.

   oc apply -f examples/connect/kafka-connect.yaml

2. Check the status of the deployment:

   oc get deployments -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                        READY  UP-TO-DATE  AVAILABLE
   my-connect-cluster-connect  1/1    1           1

   my-connect-cluster is the name of the Kafka Connect cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Deploy Kafka MirrorMaker to your OpenShift cluster:

   For MirrorMaker:

   oc apply -f examples/mirror-maker/kafka-mirror-maker.yaml

   For MirrorMaker 2.0:

   oc apply -f examples/mirror-maker/kafka-mirror-maker-2.yaml

2. Check the status of the deployment:

   oc get deployments -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                          READY  UP-TO-DATE  AVAILABLE
   my-mirror-maker-mirror-maker  1/1    1           1
   my-mm2-cluster-mirrormaker2   1/1    1           1

   my-mirror-maker is the name of the Kafka MirrorMaker cluster. my-mm2-cluster is the name of the Kafka MirrorMaker 2.0 cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Deploy Kafka Bridge to your OpenShift cluster:

   oc apply -f examples/bridge/kafka-bridge.yaml

2. Check the status of the deployment:

   oc get deployments -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME              READY  UP-TO-DATE  AVAILABLE
   my-bridge-bridge  1/1    1           1

   my-bridge is the name of the Kafka Bridge cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. List the names of the pods in your OpenShift cluster:

   oc get pods -o name
   
   pod/kafka-consumer
   # ...
   pod/my-bridge-bridge-7cbd55496b-nclrt

2. Connect to the Kafka Bridge pod on port 8080:

   oc port-forward pod/my-bridge-bridge-7cbd55496b-nclrt 8080:8080 &

   Note

   If port 8080 on your local machine is already in use, use an alternative HTTP port, such as 8008.

Procedure

1. Edit the env properties in the install/topic-operator/05-Deployment-strimzi-topic-operator.yaml standalone deployment file.

   Example standalone Topic Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-topic-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-topic-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 1
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 2
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_RESOURCE_LABELS 3
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_ZOOKEEPER_CONNECT 4
                 value: my-cluster-zookeeper-client:2181
               - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS 5
                 value: "18000"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 6
                 value: "120000"
               - name: STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS 7
                 value: "6"
               - name: STRIMZI_LOG_LEVEL 8
                 value: INFO
               - name: STRIMZI_TLS_ENABLED 9
                 value: "false"
               - name: STRIMZI_JAVA_OPTS 10
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 11
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_PUBLIC_CA 12
                 value: "false"
               - name: STRIMZI_TLS_AUTH_ENABLED 13
                 value: "false"
               - name: STRIMZI_SASL_ENABLED 14
                 value: "false"
               - name: STRIMZI_SASL_USERNAME 15
                 value: "admin"
               - name: STRIMZI_SASL_PASSWORD 16
                 value: "password"
               - name: STRIMZI_SASL_MECHANISM 17
                 value: "scram-sha-512"
               - name: STRIMZI_SECURITY_PROTOCOL 18
                 value: "SSL"

   1

   The OpenShift namespace for the Topic Operator to watch for KafkaTopic resources. Specify the namespace of the Kafka cluster.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The label to identify the KafkaTopic resources managed by the Topic Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaTopic resource. If you deploy more than one Topic Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.

   4

   The host and port pair of the address to connect to the ZooKeeper cluster. This must be the same ZooKeeper cluster that your Kafka cluster is using.

   5

   The ZooKeeper session timeout, in milliseconds. The default is 18000 (18 seconds).

   6

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes).

   7

   The number of attempts at getting topic metadata from Kafka. The time between each attempt is defined as an exponential backoff. Consider increasing this value when topic creation takes more time due to the number of partitions or replicas. The default is 6 attempts.

   8

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   9

   Enables TLS support for encrypted communication with the Kafka brokers.

   10

   (Optional) The Java options used by the JVM running the Topic Operator.

   11

   (Optional) The debugging (-D) options set for the Topic Operator.

   12

   (Optional) Skips the generation of trust store certificates if TLS is enabled through STRIMZI_TLS_ENABLED. If this environment variable is enabled, the brokers must use a public trusted certificate authority for their TLS certificates. The default is false.

   13

   (Optional) Generates key store certificates for mTLS authentication. Setting this to false disables client authentication with mTLS to the Kafka brokers. The default is true.

   14

   (Optional) Enables SASL support for client authentication when connecting to Kafka brokers. The default is false.

   15

   (Optional) The SASL username for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   16

   (Optional) The SASL password for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   17

   (Optional) The SASL mechanism for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED. You can set the value to plain, scram-sha-256, or scram-sha-512.

   18

   (Optional) The security protocol used for communication with Kafka brokers. The default value is "PLAINTEXT". You can set the value to PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL.

2. If you want to connect to Kafka brokers that are using certificates from a public certificate authority, set STRIMZI_PUBLIC_CA to true. Set this property to true, for example, if you are using Amazon AWS MSK service.

3. If you enabled mTLS with the STRIMZI_TLS_ENABLED environment variable, specify the keystore and truststore used to authenticate connection to the Kafka cluster.

   Example mTLS configuration

   # ....
   env:
     - name: STRIMZI_TRUSTSTORE_LOCATION 1
       value: "/path/to/truststore.p12"
     - name: STRIMZI_TRUSTSTORE_PASSWORD 2
       value: "TRUSTSTORE-PASSWORD"
     - name: STRIMZI_KEYSTORE_LOCATION 3
       value: "/path/to/keystore.p12"
     - name: STRIMZI_KEYSTORE_PASSWORD 4
       value: "KEYSTORE-PASSWORD"
   # ...

   1

   The truststore contains the public keys of the Certificate Authorities used to sign the Kafka and ZooKeeper server certificates.

   2

   The password for accessing the truststore.

   3

   The keystore contains the private key for mTLS authentication.

   4

   The password for accessing the keystore.

4. Deploy the Topic Operator.

   oc create -f install/topic-operator

5. Check the status of the deployment:

   oc get deployments

   Output shows the deployment name and readiness

   NAME                    READY  UP-TO-DATE  AVAILABLE
   strimzi-topic-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.

Procedure

1. Edit the following env properties in the install/user-operator/05-Deployment-strimzi-user-operator.yaml standalone deployment file.

   Example standalone User Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-user-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-user-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 1
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 2
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_CA_CERT_NAME 3
                 value: my-cluster-clients-ca-cert
               - name: STRIMZI_CA_KEY_NAME 4
                 value: my-cluster-clients-ca
               - name: STRIMZI_LABELS 5
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 6
                 value: "120000"
               - name: STRIMZI_WORK_QUEUE_SIZE 7
                 value: 10000
               - name: STRIMZI_CONTROLLER_THREAD_POOL_SIZE 8
                 value: 10
               - name: STRIMZI_LOG_LEVEL 9
                 value: INFO
               - name: STRIMZI_GC_LOG_ENABLED 10
                 value: "true"
               - name: STRIMZI_CA_VALIDITY 11
                 value: "365"
               - name: STRIMZI_CA_RENEWAL 12
                 value: "30"
               - name: STRIMZI_JAVA_OPTS 13
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 14
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_SECRET_PREFIX 15
                 value: "kafka-"
               - name: STRIMZI_ACLS_ADMIN_API_SUPPORTED 16
                 value: "true"
               - name: STRIMZI_MAINTENANCE_TIME_WINDOWS 17
                 value: '* * 8-10 * * ?;* * 14-15 * * ?'
               - name: STRIMZI_KAFKA_ADMIN_CLIENT_CONFIGURATION 18
                 value: |
                   default.api.timeout.ms=120000
                   request.timeout.ms=60000

   1

   The OpenShift namespace for the User Operator to watch for KafkaUser resources. Only one namespace can be specified.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The OpenShift Secret that contains the public key (ca.crt) value of the Certificate Authority that signs new user certificates for mTLS authentication.

   4

   The OpenShift Secret that contains the private key (ca.key) value of the Certificate Authority that signs new user certificates for mTLS authentication.

   5

   The label to identify the KafkaUser resources managed by the User Operator. This does not have to be the name of the Kafka cluster. It can be the label assigned to the KafkaUser resource. If you deploy more than one User Operator, the labels must be unique for each. That is, the operators cannot manage the same resources.

   6

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes).

   7

   The size of the controller event queue. The size of the queue should be at least as big as the maximal amount of users you expect the User Operator to operate. The default is 1024.

   8

   The size of the worker pool for reconciling the users. Bigger pool might require more resources, but it will also handle more KafkaUser resources The default is 50.

   9

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   10

   Enables garbage collection (GC) logging. The default is true.

   11

   The validity period for the Certificate Authority. The default is 365 days.

   12

   The renewal period for the Certificate Authority. The renewal period is measured backwards from the expiry date of the current certificate. The default is 30 days to initiate certificate renewal before the old certificates expire.

   13

   (Optional) The Java options used by the JVM running the User Operator

   14

   (Optional) The debugging (-D) options set for the User Operator

   15

   (Optional) Prefix for the names of OpenShift secrets created by the User Operator.

   16

   (Optional) Indicates whether the Kafka cluster supports management of authorization ACL rules using the Kafka Admin API. When set to false, the User Operator will reject all resources with simple authorization ACL rules. This helps to avoid unnecessary exceptions in the Kafka cluster logs. The default is true.

   17

   (Optional) Semi-colon separated list of Cron Expressions defining the maintenance time windows during which the expiring user certificates will be renewed.

   18

   (Optional) Configuration options for configuring the Kafka Admin client used by the User Operator in the properties format.

2. If you are using mTLS to connect to the Kafka cluster, specify the secrets used to authenticate connection. Otherwise, go to the next step.

   Example mTLS configuration

   # ....
   env:
     - name: STRIMZI_CLUSTER_CA_CERT_SECRET_NAME 1
       value: my-cluster-cluster-ca-cert
     - name: STRIMZI_EO_KEY_SECRET_NAME 2
       value: my-cluster-entity-operator-certs
   # ..."

   1

   The OpenShift Secret that contains the public key (ca.crt) value of the CA that signs Kafka broker certificates.

   2

   The OpenShift Secret that contains the keystore (entity-operator.p12) with the private key and certificate for mTLS authentication against the Kafka cluster. The Secret must also contain the password (entity-operator.password) for accessing the keystore.

3. Deploy the User Operator.

   oc create -f install/user-operator

4. Check the status of the deployment:

   oc get deployments

   Output shows the deployment name and readiness

   NAME                   READY  UP-TO-DATE  AVAILABLE
   strimzi-user-operator  1/1    1           1

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.