Chapter 5. Deploying AMQ Streams using installation artifacts

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

3. Verify that the Cluster Operator was successfully deployed:

   oc get deployments

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to add a list of all the namespaces the Cluster Operator will watch to the STRIMZI_NAMESPACE environment variable.

   For example, in this procedure the Cluster Operator will watch the namespaces watched-namespace-1, watched-namespace-2, watched-namespace-3.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq7/amq-streams-rhel8-operator:2.0.1
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: watched-namespace-1,watched-namespace-2,watched-namespace-3

3. For each namespace listed, install the RoleBindings.

   In this example, we replace watched-namespace in these commands with the namespaces listed in the previous step, repeating them for watched-namespace-1, watched-namespace-2, watched-namespace-3:

   oc create -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n watched-namespace
   oc create -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n watched-namespace

4. Deploy the Cluster Operator:

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Verify that the Cluster Operator was successfully deployed:

   oc get deployments

Procedure

1. Edit the AMQ Streams installation files to use the namespace the Cluster Operator is going to be installed into.

   For example, in this procedure the Cluster Operator is installed into the namespace my-cluster-operator-namespace.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

2. Edit the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file to set the value of the STRIMZI_NAMESPACE environment variable to *.

   apiVersion: apps/v1
   kind: Deployment
   spec:
     # ...
     template:
       spec:
         # ...
         serviceAccountName: strimzi-cluster-operator
         containers:
         - name: strimzi-cluster-operator
           image: registry.redhat.io/amq7/amq-streams-rhel8-operator:2.0.1
           imagePullPolicy: IfNotPresent
           env:
           - name: STRIMZI_NAMESPACE
             value: "*"
           # ...

3. Create ClusterRoleBindings that grant cluster-wide access for all namespaces to the Cluster Operator.

   oc create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator
   oc create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-namespace:strimzi-cluster-operator

   Replace my-cluster-operator-namespace with the namespace you want to install the Cluster Operator into.

4. Deploy the Cluster Operator to your OpenShift cluster.

   oc create -f install/cluster-operator -n my-cluster-operator-namespace

5. Verify that the Cluster Operator was successfully deployed:

   oc get deployments

Procedure

1. Create and deploy an ephemeral or persistent cluster.

   For development or testing, you might prefer to use an ephemeral cluster. You can use a persistent cluster in any situation.

   • To create and deploy an ephemeral cluster:

     oc apply -f examples/kafka/kafka-ephemeral.yaml

   • To create and deploy a persistent cluster:

     oc apply -f examples/kafka/kafka-persistent.yaml

2. Verify that the Kafka cluster was successfully deployed:

   oc get deployments

Procedure

1. Edit the entityOperator properties of the Kafka resource to include topicOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the Topic Operator spec using the properties described in EntityTopicOperatorSpec schema reference.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   Use oc apply:

   oc apply -f <your-file>

Procedure

1. Edit the entityOperator properties of the Kafka resource to include userOperator:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     #...
     entityOperator:
       topicOperator: {}
       userOperator: {}

2. Configure the User Operator spec using the properties described in EntityUserOperatorSpec schema reference in the Using AMQ Streams on OpenShift guide.

   Use an empty object ({}) if you want all properties to use their default values.

3. Create or update the resource:

   oc apply -f <your-file>

Procedure

1. Edit the env properties in the install/topic-operator/05-Deployment-strimzi-topic-operator.yaml standalone deployment file.

   Example standalone Topic Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-topic-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-topic-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 

   1

   
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 

   2

   
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_RESOURCE_LABELS 

   3

   
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_ZOOKEEPER_CONNECT 

   4

   
                 value: my-cluster-zookeeper-client:2181
               - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS 

   5

   
                 value: "18000"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 

   6

   
                 value: "120000"
               - name: STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS 

   7

   
                 value: "6"
               - name: STRIMZI_LOG_LEVEL 

   8

   
                 value: INFO
               - name: STRIMZI_TLS_ENABLED 

   9

   
                 value: "false"
               - name: STRIMZI_JAVA_OPTS 

   10

   
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 

   11

   
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_PUBLIC_CA 

   12

   
                 value: "false"
               - name: STRIMZI_TLS_AUTH_ENABLED 

   13

   
                 value: "false"
               - name: STRIMZI_SASL_ENABLED 

   14

   
                 value: "false"
               - name: STRIMZI_SASL_USERNAME 

   15

   
                 value: "admin"
               - name: STRIMZI_SASL_PASSWORD 

   16

   
                 value: "password"
               - name: STRIMZI_SASL_MECHANISM 

   17

   
                 value: "scram-sha-512"
               - name: STRIMZI_SECURITY_PROTOCOL 

   18

   
                 value: "SSL"

   1

   The OpenShift namespace for the Topic Operator to watch for KafkaTopic resources. Specify the namespace of the Kafka cluster.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The label selector to identify the KafkaTopic resources managed by the Topic Operator.

   4

   The host and port pair of the address to connect to the ZooKeeper cluster. This must be the same ZooKeeper cluster that your Kafka cluster is using.

   5

   The ZooKeeper session timeout, in milliseconds. The default is 18000 (18 seconds).

   6

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes).

   7

   The number of attempts at getting topic metadata from Kafka. The time between each attempt is defined as an exponential backoff. Consider increasing this value when topic creation takes more time due to the number of partitions or replicas. The default is 6 attempts.

   8

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   9

   Enables TLS support for encrypted communication with the Kafka brokers.

   10

   (Optional) The Java options used by the JVM running the Topic Operator.

   11

   (Optional) The debugging (-D) options set for the Topic Operator.

   12

   (Optional) Skips the generation of trust store certificates if TLS is enabled through STRIMZI_TLS_ENABLED. If this environment variable is enabled, the brokers must use a public trusted certificate authority for their TLS certificates. The default is false.

   13

   (Optional) Generates key store certificates for mutual TLS authentication. Setting this to false disables client authentication with TLS to the Kafka brokers. The default is true.

   14

   (Optional) Enables SASL support for client authentication when connecting to Kafka brokers. The default is false.

   15

   (Optional) The SASL username for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   16

   (Optional) The SASL password for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED.

   17

   (Optional) The SASL mechanism for client authentication. Mandatory only if SASL is enabled through STRIMZI_SASL_ENABLED. You can set the value to plain, scram-sha-256, or scram-sha-512.

   18

   (Optional) The security protocol used for communication with Kafka brokers. The default value is "PLAINTEXT". You can set the value to PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL.

2. If you want to connect to Kafka brokers that are using certificates from a public certificate authority, set STRIMZI_PUBLIC_CA to true. Set this property to true, for example, if you are using Amazon AWS MSK service.

3. If you enabled TLS with the STRIMZI_TLS_ENABLED environment variable, specify the keystore and truststore used to authenticate connection to the Kafka cluster.

   Example TLS configuration

   # ....
   env:
     - name: STRIMZI_TRUSTSTORE_LOCATION 

   1

   
       value: "/path/to/truststore.p12"
     - name: STRIMZI_TRUSTSTORE_PASSWORD 

   2

   
       value: "TRUSTSTORE-PASSWORD"
     - name: STRIMZI_KEYSTORE_LOCATION 

   3

   
       value: "/path/to/keystore.p12"
     - name: STRIMZI_KEYSTORE_PASSWORD 

   4

   
       value: "KEYSTORE-PASSWORD"
   # ...

   1

   The truststore contains the public keys of the Certificate Authorities used to sign the Kafka and ZooKeeper server certificates.

   2

   The password for accessing the truststore.

   3

   The keystore contains the private key for TLS client authentication.

   4

   The password for accessing the keystore.

4. Deploy the Topic Operator.

   oc create -f install/topic-operator

5. Verify that the Topic Operator has been deployed successfully.

   oc describe deployment strimzi-topic-operator

   The Topic Operator is deployed when the Replicas entry shows 1 available.

   Note

   You may experience a delay with the deployment if you have a slow connection to the OpenShift cluster and the Topic Operator images have not been downloaded before.

Procedure

1. Edit the following env properties in the install/user-operator/05-Deployment-strimzi-user-operator.yaml standalone deployment file.

   Example standalone User Operator deployment configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: strimzi-user-operator
     labels:
       app: strimzi
   spec:
     # ...
     template:
       # ...
       spec:
         # ...
         containers:
           - name: strimzi-user-operator
             # ...
             env:
               - name: STRIMZI_NAMESPACE 

   1

   
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS 

   2

   
                 value: my-kafka-bootstrap-address:9092
               - name: STRIMZI_CA_CERT_NAME 

   3

   
                 value: my-cluster-clients-ca-cert
               - name: STRIMZI_CA_KEY_NAME 

   4

   
                 value: my-cluster-clients-ca
               - name: STRIMZI_LABELS 

   5

   
                 value: "strimzi.io/cluster=my-cluster"
               - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS 

   6

   
                 value: "120000"
               - name: STRIMZI_LOG_LEVEL 

   7

   
                 value: INFO
               - name: STRIMZI_GC_LOG_ENABLED 

   8

   
                 value: "true"
               - name: STRIMZI_CA_VALIDITY 

   9

   
                 value: "365"
               - name: STRIMZI_CA_RENEWAL 

   10

   
                 value: "30"
               - name: STRIMZI_JAVA_OPTS 

   11

   
                 value: "-Xmx=512M -Xms=256M"
               - name: STRIMZI_JAVA_SYSTEM_PROPERTIES 

   12

   
                 value: "-Djavax.net.debug=verbose -DpropertyName=value"
               - name: STRIMZI_SECRET_PREFIX 

   13

   
                 value: "kafka-"
               - name: STRIMZI_ACLS_ADMIN_API_SUPPORTED 

   14

   
                 value: "true"

   1

   The OpenShift namespace for the User Operator to watch for KafkaUser resources. Only one namespace can be specified.

   2

   The host and port pair of the bootstrap broker address to discover and connect to all brokers in the Kafka cluster. Use a comma-separated list to specify two or three broker addresses in case a server is down.

   3

   The OpenShift Secret that contains the public key (ca.crt) value of the Certificate Authority that signs new user certificates for TLS client authentication.

   4

   The OpenShift Secret that contains the private key (ca.key) value of the Certificate Authority that signs new user certificates for TLS client authentication.

   5

   The label selector used to identify the KafkaUser resources managed by the User Operator.

   6

   The interval between periodic reconciliations, in milliseconds. The default is 120000 (2 minutes). The default is 18000 (18 seconds).

   7

   The level for printing logging messages. You can set the level to ERROR, WARNING, INFO, DEBUG, or TRACE.

   8

   Enables garbage collection (GC) logging. The default is true.

   9

   The validity period for the Certificate Authority. The default is 365 days.

   10

   The renewal period for the Certificate Authority. The renewal period is measured backwards from the expiry date of the current certificate. The default is 30 days to initiate certificate renewal before the old certificates expire.

   11

   (Optional) The Java options used by the JVM running the User Operator

   12

   (Optional) The debugging (-D) options set for the User Operator

   13

   (Optional) Prefix for the names of OpenShift secrets created by the User Operator.

   14

   (Optional) Indicates whether the Kafka cluster supports management of authorization ACL rules using the Kafka Admin API. When set to false, the User Operator will reject all resources with simple authorization ACL rules. This helps to avoid unnecessary exceptions in the Kafka cluster logs. The default is true.

2. If you are using TLS to connect to the Kafka cluster, specify the secrets used to authenticate connection. Otherwise, go to the next step.

   Example TLS configuration

   # ....
   env:
     - name: STRIMZI_CLUSTER_CA_CERT_SECRET_NAME 

   1

   
       value: my-cluster-cluster-cert
     - name: STRIMZI_EO_KEY_SECRET_NAME 

   2

   
       value: my-cluster-entity-operator-certs
   # ..."

   1

   The OpenShift Secret that contains the public key (ca.crt) value of the Certificate Authority that signs Kafka broker certificates for TLS client authentication.

   2

   The OpenShift Secret that contains the keystore (entity-operator.p12) with the private key and certificate for TLS authentication against the Kafka cluster. The Secret must also contain the password (entity-operator.password) for accessing the keystore.

3. Deploy the User Operator.

   oc create -f install/user-operator

4. Verify that the User Operator has been deployed successfully.

   oc describe deployment strimzi-user-operator

   The User Operator is deployed when the Replicas entry shows 1 available.

   Note

   You might experience a delay with the deployment if you have a slow connection to the OpenShift cluster and the User Operator images have not been downloaded before.

Procedure

1. Configure the KafkaConnect custom resource by specifying the container registry in .spec.build.output, and additional connectors in .spec.build.plugins:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnect
   metadata:
     name: my-connect-cluster
   spec: 

   1

   
     #...
     build:
       output: 

   2

   
         type: docker
         image: my-registry.io/my-org/my-connect-cluster:latest
         pushSecret: my-registry-credentials
       plugins: 

   3

   
         - name: debezium-postgres-connector
           artifacts:
             - type: tgz
               url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.3.1.Final/debezium-connector-postgres-1.3.1.Final-plugin.tar.gz
               sha512sum: 962a12151bdf9a5a30627eebac739955a4fd95a08d373b86bdcea2b4d0c27dd6e1edd5cb548045e115e33a9e69b1b2a352bee24df035a0447cb820077af00c03
         - name: camel-telegram
           artifacts:
             - type: tgz
               url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-telegram-kafka-connector/0.7.0/camel-telegram-kafka-connector-0.7.0-package.tar.gz
               sha512sum: a9b1ac63e3284bea7836d7d24d84208c49cdf5600070e6bd1535de654f6920b74ad950d51733e8020bf4187870699819f54ef5859c7846ee4081507f48873479
     #...

   1

   The specification for the Kafka Connect cluster.

   2

   (Required) Configuration of the container registry where new images are pushed.

   3

   (Required) List of connector plugins and their artifacts to add to the new container image. Each plugin must be configured with at least one artifact.

2. Create or update the resource:

   $ oc apply -f KAFKA-CONNECT-CONFIG-FILE

3. Wait for the new container image to build, and for the Kafka Connect cluster to be deployed.
4. Use the Kafka Connect REST API or the KafkaConnector custom resources to use the connector plugins you added.

Procedure

1. Create a new Dockerfile using registry.redhat.io/amq7/amq-streams-kafka-30-rhel8:2.0.1 as the base image:

   FROM registry.redhat.io/amq7/amq-streams-kafka-30-rhel8:2.0.1
   USER root:root
   COPY ./my-plugins/ /opt/kafka/plugins/
   USER 1001

   Example plug-in file

   $ tree ./my-plugins/
   ./my-plugins/
   ├── debezium-connector-mongodb
   │   ├── bson-3.4.2.jar
   │   ├── CHANGELOG.md
   │   ├── CONTRIBUTE.md
   │   ├── COPYRIGHT.txt
   │   ├── debezium-connector-mongodb-0.7.1.jar
   │   ├── debezium-core-0.7.1.jar
   │   ├── LICENSE.txt
   │   ├── mongodb-driver-3.4.2.jar
   │   ├── mongodb-driver-core-3.4.2.jar
   │   └── README.md
   ├── debezium-connector-mysql
   │   ├── CHANGELOG.md
   │   ├── CONTRIBUTE.md
   │   ├── COPYRIGHT.txt
   │   ├── debezium-connector-mysql-0.7.1.jar
   │   ├── debezium-core-0.7.1.jar
   │   ├── LICENSE.txt
   │   ├── mysql-binlog-connector-java-0.13.0.jar
   │   ├── mysql-connector-java-5.1.40.jar
   │   ├── README.md
   │   └── wkb-1.0.2.jar
   └── debezium-connector-postgres
       ├── CHANGELOG.md
       ├── CONTRIBUTE.md
       ├── COPYRIGHT.txt
       ├── debezium-connector-postgres-0.7.1.jar
       ├── debezium-core-0.7.1.jar
       ├── LICENSE.txt
       ├── postgresql-42.0.0.jar
       ├── protobuf-java-2.6.1.jar
       └── README.md

   Note

   This example uses the Debezium connectors for MongoDB, MySQL, and PostgreSQL. Debezium running in Kafka Connect looks the same as any other Kafka Connect task.

2. Build the container image.
3. Push your custom image to your container registry.

4. Point to the new container image.

   You can either:

   • Edit the KafkaConnect.spec.image property of the KafkaConnect custom resource.

     If set, this property overrides the STRIMZI_KAFKA_CONNECT_IMAGES variable in the Cluster Operator.

     apiVersion: kafka.strimzi.io/v1beta2
     kind: KafkaConnect
     metadata:
       name: my-connect-cluster
     spec: 

     1

     
       #...
       image: my-new-container-image 

     2

     
       config: 

     3

     
         #...

     1

     The specification for the Kafka Connect cluster.

     2

     The docker image for the pods.

     3

     Configuration of the Kafka Connect workers (not connectors).

     or

   • In the install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml file, edit the STRIMZI_KAFKA_CONNECT_IMAGES variable to point to the new container image, and then reinstall the Cluster Operator.