이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 2. Installing Debezium connectors


Install Debezium connectors through AMQ Streams by extending Kafka Connect with connector plugins. Following a deployment of AMQ Streams, you can deploy Debezium as a connector configuration through Kafka Connect.

2.1. Prerequisites

A Debezium installation requires the following:

  • Red Hat Enterprise Linux version 7.x or 8.x with an x86_64 architecture.
  • Administrative privileges (sudo access).
  • AMQ Streams 1.5 on Red Hat Enterprise Linux is installed on the host machine.

  • Credentials for the kafka user that was created when AMQ Streams was installed.
  • An AMQ Streams cluster is running.

Note

If you have an earlier version of AMQ Streams, you need to upgrade to AMQ Streams 1.5. For upgrade instructions, see AMQ Streams and Kafka upgrades.

Additional resources

2.2. Kafka topic creation recommendations

Debezium uses multiple Kafka topics for storing data. The topics have to be either created by an administrator, or by Kafka itself by enabling auto-creation for topics using the auto.create.topics.enable broker configuration.

The following list describes limitations and recommendations to consider when creating topics:

Database history topics (for MySQL and SQL Server connectors)
  • Infinite (or very long retention).
  • Replication factor of at least 3 in production.
  • Single partition.
Other topics
  • Optionally, log compaction enabled (if you wish to only keep the last change event for a given record).

    In this case, the min.compaction.lag.ms and delete.retention.ms topic-level settings in Apache Kafka should be configured so that consumers have enough time to receive all events and delete markers. Specifically, these values should be larger than the maximum downtime you anticipate for the sink connectors (for example, when you update them).

  • Replicated in production.
  • Single partition.

    You can relax the single partition rule, but your application must handle out-of-order events for different rows in the database (events for a single row are still totally ordered). If multiple partitions are used, Kafka will determine the partition by hashing the key by default. Other partition strategies require using Simple Message Transforms (SMTs) to set the partition number for each record.

2.3. Deploying Debezium with AMQ Streams on RHEL

This procedure describes how to set up connectors for Debezium on Red Hat Enterprise Linux. Connectors are deployed to an AMQ Streams cluster using Kafka Connect, a framework for streaming data between Apache Kafka and external systems. Kafka Connect must be run in distributed mode rather than standalone mode.

This procedure assumes that AMQ Streams is installed and ZooKeeper and Kafka are running.

Procedure

  1. Visit the Red Hat Integration download site on the Red Hat Customer Portal and download the Debezium connector or connectors that you want to use. For example, download the Debezium 1.1.0 MySQL Connector to use Debezium with a MySQL database.
  2. In /opt/kafka, create the connector-plugins directory if not already created for other Kafka Connect plugins:

    $ sudo mkdir /opt/kafka/connector-plugins
  3. Extract the contents of the Debezium connector archive to the /opt/kafka/connector-plugins directory.

    This example extracts the contents of the MySQL connector:

    $ sudo unzip debezium-connector-mysql-1.1.0-plugin.zip -d /opt/kafka/connector-plugins
  4. Repeat the above step for each connector that you want to install.
  5. Switch to the kafka user:

    $ su - kafka
    $ Password:
  6. Check whether Kafka Connect is already running in distributed mode. If it is running, stop the associated process on all Kafka Connect worker nodes. For example:

    $ jcmd | grep ConnectDistributed
    18514 org.apache.kafka.connect.cli.ConnectDistributed /opt/kafka/config/connect-distributed.properties
    $ kill 18514
  7. Edit the connect-distributed.properties file in /opt/kafka/config/ and specify the location of the Debezium connector:

    plugin.path=/opt/kafka/connector-plugins
  8. Run Kafka Connect in distributed mode:

    $ /opt/kafka/bin/connect-distributed.sh /opt/kafka/config/connect-distributed.properties

    Kafka Connect runs. During startup, Debezium connectors are loaded from the connector-plugins directory.

  9. Repeat steps 6–8 for each Kafka Connect worker node.

Updating Kafka Connect

If you need to update your deployment, amend the Debezium connector JAR files in the /opt/kafka/connector-plugins directory, and then restart Kafka Connect.

Next Steps

The Debezium User Guide describes how to configure each connector and its source database for change data capture. Once configured, a connector will connect to the source database and produce events for each inserted, updated, and deleted row or document.

Red Hat logoGithubRedditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

© 2024 Red Hat, Inc.