Chapter 2. Starting the services
Using Debezium requires AMQ Streams and the Debezium connector service. To start the services needed for this tutorial, you must:
2.1. Setting up a Kafka cluster
You use AMQ Streams to set up a Kafka cluster. This procedure deploys a single-node Kafka cluster.
Procedure
In your OpenShift 4.x cluster, create a new project:
$ oc new-project cdc-tutorial
- Change to the directory where you downloaded the AMQ Streams 1.4 OpenShift installation and example files.
Deploy the AMQ Streams Cluster Operator.
The Cluster Operator is responsible for deploying and managing Kafka clusters within an OpenShift cluster. This command deploys the Cluster Operator to watch just the project that you created:
$ sed -i 's/namespace: .*/namespace: cdc-tutorial/' install/cluster-operator/*RoleBinding*.yaml $ oc apply -f install/cluster-operator -n cdc-tutorial
Verify that the Cluster Operator is running.
This command shows that the Cluster Operator is running, and that all of the Pods are ready:
$ oc get pods NAME READY STATUS RESTARTS AGE strimzi-cluster-operator-5c6d68c54-l4gdz 1/1 Running 0 46s
Deploy the Kafka cluster.
This command uses the
kafka-ephemeral-single.yaml
Custom Resource to create an ephemeral Kafka cluster with three ZooKeeper nodes and one Kafka node:$ oc apply -f examples/kafka/kafka-ephemeral-single.yaml
Verify that the Kafka cluster is running.
This command shows that the Kafka cluster is running, and that all of the Pods are ready:
$ oc get pods NAME READY STATUS RESTARTS AGE my-cluster-entity-operator-5b5d4f7c58-8gnq5 3/3 Running 0 41s my-cluster-kafka-0 2/2 Running 0 70s my-cluster-zookeeper-0 2/2 Running 0 107s my-cluster-zookeeper-1 2/2 Running 0 107s my-cluster-zookeeper-2 2/2 Running 0 107s strimzi-cluster-operator-5c6d68c54-l4gdz 1/1 Running 0 8m53s
2.2. Deploying Kafka Connect
After setting up a Kafka cluster, you deploy the Kafka Connect Source-to-Image (S2I) service. This service provides a framework for managing the Debezium MySQL connector.
Procedure
Deploy the Kafka Connect Source-to-Image (S2I) service:
This command deploys the Kafka Connect S2I service using the example YAML file for a single-node Kafka cluster:
$ oc apply -f examples/kafka-connect/kafka-connect-s2i-single-node-kafka.yaml
Verify that the Kafka Connect service is running.
This command shows that the Kafka Connect service is running, and that the Pod is ready:
$ oc get pods -l strimzi.io/name=my-connect-cluster-connect NAME READY STATUS RESTARTS AGE my-connect-cluster-connect-1-dxcs9 1/1 Running 0 7m
Start a new build of the Kafka Connect image using the Debezium MySQL Connector plugin.
This command uses the Debezium MySQL Connector plugin that you previously downloaded:
$ oc start-build my-connect-cluster-connect --from-dir ./my-plugins/
Verify that the build has completed.
This command shows that the new build is complete (
my-connect-cluster-connect-2
). The Debezium MySQL Connector is installed:$ oc get build NAME TYPE FROM STATUS STARTED DURATION my-connect-cluster-connect-1 Source Complete 9 minutes ago 2m10s my-connect-cluster-connect-2 Source Binary Complete 4 minutes ago 2m2s
2.3. Deploying a MySQL database
At this point, you have deployed a Kafka cluster and the Kafka Connect service with the Debezium MySQL Database Connector. However, you still need a database server from which Debezium can capture changes. In this procedure, you will start a MySQL server with an example database.
Procedure
Start a MySQL database.
This command starts a MySQL database server preconfigured with an example
inventory
database:$ oc new-app --name=mysql debezium/example-mysql:1.0
Configure credentials for the MySQL database.
This command updates the deployment configuration for the MySQL database to add the user name and password:
$ oc set env dc/mysql MYSQL_ROOT_PASSWORD=debezium MYSQL_USER=mysqluser MYSQL_PASSWORD=mysqlpw
Verify that the MySQL database is running.
This command shows that the MySQL database is running, and that the Pod is ready:
$ oc get pods -l app=mysql NAME READY STATUS RESTARTS AGE mysql-1-2gzx5 1/1 Running 1 23s
Open a new terminal and log into the sample
inventory
database.This command opens a MySQL command line client in the Pod that is running the MySQL database. It uses the user name and password that you previously configured:
$ oc exec mysql-1-2gzx5 -it -- mysql -u mysqluser -p mysqlpw inventory mysql: [Warning] Using a password on the command line interface can be insecure. Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 7 Server version: 5.7.29-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
List the tables in the
inventory
database.mysql> show tables; +---------------------+ | Tables_in_inventory | +---------------------+ | addresses | | customers | | geom | | orders | | products | | products_on_hand | +---------------------+ 6 rows in set (0.00 sec)
Explore the database and view the pre-loaded data.
This example shows the customers table:
mysql> select * from customers; +------+------------+-----------+-----------------------+ | id | first_name | last_name | email | +------+------------+-----------+-----------------------+ | 1001 | Sally | Thomas | sally.thomas@acme.com | | 1002 | George | Bailey | gbailey@foobar.com | | 1003 | Edward | Walker | ed@walker.com | | 1004 | Anne | Kretchmar | annek@noanswer.org | +------+------------+-----------+-----------------------+ 4 rows in set (0.00 sec)