Chapter 2. Installing Debezium connectors

Procedure

1. Deploy your Kafka cluster. If you already have a Kafka cluster deployed, skip the following three sub-steps.

   1. Install the AMQ Streams operator by following the steps in Installing AMQ Streams and deploying components.
   2. Select the desired configuration and deploy your Kafka Cluster.
   3. Deploy Kafka Connect.

   You now have a working Kafka cluster that is running in OpenShift with Kafka Connect.

2. Check that your pods are running. The pod names correspond with your AMQ Streams deployment.

   $ oc get pods
   
   NAME                                               READY STATUS
   <cluster-name>-entity-operator-7b6b9d4c5f-k7b92    3/3   Running
   <cluster-name>-kafka-0                             2/2   Running
   <cluster-name>-zookeeper-0                         2/2   Running
   <cluster-name>-operator-97cd5cf7b-l58bq            1/1   Running

   In addition to running pods, you should have a DeploymentConfig associated with Kafka Connect.

3. Go to the Red Hat Integration download site.
4. Download the Debezium connector archive(s) for your database(s).

5. Extract the archive(s) to create a directory structure for the connector plug-in(s). If you downloaded and extracted multiple archives, the structure looks like this:

   $ tree ./my-plugins/
   ./my-plugins/
   ├── debezium-connector-db2
   |   ├── ...
   ├── debezium-connector-mongodb
   |   ├── ...
   ├── debezium-connector-mysql
   │   ├── ...
   ├── debezium-connector-postgres
   │   ├── ...
   └── debezium-connector-sqlserver
       ├── ...

6. Create a new Dockerfile by using registry.redhat.io/amq7/amq-streams-kafka-28-rhel8:1.8.0 as the base image:

   FROM registry.redhat.io/amq7/amq-streams-kafka-28-rhel8:1.8.0
   USER root:root
   COPY ./my-plugins/ /opt/kafka/plugins/
   USER 1001

7. Build the container image. If the Dockerfile you created in the previous step is in the current directory, enter one of the following commands:

   podman build -t my-new-container-image:latest .

   docker build -t my-new-container-image:latest .

8. Push your custom image to your container registry. Enter one of the following commands:

   podman push my-new-container-image:latest

   docker push my-new-container-image:latest

9. Point to the new container image. Complete one of the following tasks to specify the name of the image that you created to run your Debezium connector:

   • Edit the spec.image field of the KafkaConnect custom resource.

     If you set this property, the value overrides the STRIMZI_DEFAULT_KAFKA_CONNECT_IMAGE variable in the cluster Operator. For example:

     apiVersion: kafka.strimzi.io/v1beta2
     kind: KafkaConnect
     metadata:
       name: my-connect-cluster
       annotations: strimzi.io/use-connector-resources: "true"
     spec:
       #...
       image: my-new-container-image

   • In the install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml file, edit the STRIMZI_DEFAULT_KAFKA_CONNECT_IMAGE variable to point to the new container image and reinstall the Cluster Operator. If you edit this file you will need to apply it to your OpenShift cluster.

   The Kafka Connect deployment starts to use the new image.