此内容没有您所选择的语言版本。

Chapter 1. Introduction to Camel Kafka Connector

This chapter introduces the features, concepts, and distributions provided by Camel Kafka Connector:

Important

Camel Kafka Connector is a Technology Preview feature only. There are no plans for a release of Camel Kafka Connector beyond the Technology Preview release. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview.

1.1. Camel Kafka Connector overview

Apache Camel is a highly flexible open source integration framework for connecting a wide range of different systems, which is based on standard Enterprise Integration Patterns (EIPs). Apache Kafka Connect is the Kafka-native approach for connecting to external systems, which is specifically designed for event-driven architectures.

Camel Kafka Connector enables you to use standard Camel components as Kafka Connect connectors. This widens the scope of possible integrations beyond the external systems supported by Kafka Connect connectors alone. Camel Kafka Connector works as an adapter that makes the popular Camel component ecosystem available in Kafka-based AMQ Streams on OpenShift.

Camel Kafka Connector provides a user-friendly way to configure Camel components directly in the Kafka Connect framework. Using Camel Kafka Connector, you can leverage Camel components for integration with different systems by connecting to or from Camel Kafka sink or source connectors. You do not need to write any code, and can include the appropriate connector JARs in your Kafka Connect image and configure connector options using custom resources.

Camel Kafka Connector is built on Apache Camel Kafka Connector, which is a subproject of the Apache Camel open source community. Camel Kafka Connector is fully integrated with AMQ Streams and Kafka Connect, and is available on both OpenShift Container Platform and Red Hat Enterprise Linux.

Camel Kafka Connector is available along with the Red Hat Integration - Camel K distribution for cloud-native integration on OpenShift. Camel K is a lightweight integration framework built from Apache Camel K that runs natively in the cloud on OpenShift. Camel K is specifically designed for serverless and microservice architectures.

Additional resources

1.2. Camel Kafka Connector features

The Camel Kafka Connector Technology Preview includes the following main features:

1.2.1. Platforms and components

OpenShift Container Platform 4.6 or 4.7
Red Hat Enterprise Linux 8.x
AMQ Streams 1.6
Apache Kafka Connect 2.6
Apache Camel Kafka Connector 0.7.1
Apache Camel 3.7
OpenJDK 11

1.2.2. Technology Preview features

Selected Camel Kafka connectors
Marshaling/unmarshalling of Camel data formats for sink and source connectors
Aggregation for sink connectors
Maven archetypes for extending connectors

1.2.3. Camel Kafka connectors

Table 1.1. Camel Kafka connectors in Technology Preview
Connector	Sink/source
Amazon Web Services (AWS2) Kinesis	Sink and source
Amazon Web Services (AWS2) S3	Sink and source
Amazon Web Services (AWS2) SNS	Sink only
Amazon Web Services (AWS2) SQS	Sink and source
Azure Storage Blob	Sink only
Azure Storage Queue	Sink only
Cassandra Query Language (CQL)	Sink and source
Elasticsearch	Sink only
File	Sink only
Hadoop Distributed File System (HDFS)	Sink only
Hypertext Transfer Protocol (HTTP)	Sink only
Java Database Connectivity (JDBC)	Sink only
Java Message Service (JMS)	Sink and source
MongoDB	Sink and source
RabbitMQ	Sink and source
SQL	Sink and source
SSH	Sink and source
Syslog	Sink and source
Timer	Source only

1.2.4. Camel data formats

The Camel Kafka Connector Technology Preview includes marshaling and unmarshaling of Camel data formats. For example, these formats include Apache Avro, Base64, Google Protobuf, JSON, SOAP, Zip file, and many more. You can configure marshaling and unmarshaling of Camel data formats using properties in your Camel Kafka Connector, configuration.

1.3. Camel Kafka Connector architecture

AMQ Streams is a distributed and scalable streaming platform based on Apache Kafka that includes a publish/subscribe messaging broker. Kafka Connect provides a framework to integrate Kafka-based systems with external systems. Using Kafka Connect, you can configure source and sink connectors to stream data from external systems into and out of a Kafka broker.

Camel Kafka Connector reuses the flexibility of Camel components and makes them available in Kafka Connect as source and sink connectors that you can use to stream data into and out of AMQ Streams. For example, you can ingest data from Amazon Web Services for processing using an AWS S3 source connector, or consolidate events stored in Kafka into an Elasticsearch instance for analytics using an Elasticsearch sink connector.

The following diagram shows a simplified view of the Camel Kafka Connector cloud-native integration architecture based on AMQ Streams:

Figure 1.1. Camel Kafka Connector architecture

Kafka Connect concepts

Source connector: Source connectors work like consumers and pull data from external systems into Kafka topics to make the data available for stream processing. For example, these external source systems include Amazon Web Services or Java Message Service.
Sink connector: Sink connectors work like producers and push data from Kafka topics into external systems for offline analysis. For example, these external sink systems include Cassandra, Syslog, or Elasticsearch.
Sink/source task: Tasks are typically created by a sink or source connector and are responsible for handling the data.
Key/value converter: Key/value converters can serialize/deserialize the key or value of a Kafka message in various formats.
Transformer: Transformers can manipulate Kafka message content, for example, renaming fields or routing to topics based on values.
Aggregator: Sink connectors can use an aggregator to batch up records before sending them to an external system.

Camel Kafka Connector configuration

You can use Camel Kafka Connector configuration to specify the following:

Kafka Connect configuration options
Camel route definitions
Camel configuration options

Additional resources

Apache Kafka Connect 2.6 user documentation

1.4. Camel Kafka Connector distributions

The Camel Kafka Connector distributions are available as part of Red Hat Integration:

Table 1.2. Camel Kafka Connector available distributions
Distribution	Description	Location
Camel Kafka connectors	JAR files for each Camel Kafka connector in `.zip` or `.tar.gz` format	Early-access Maven repository downloads
Maven repository	All Maven artifacts for Camel Kafka Connector	Software Downloads > Red Hat Integration
Source code	All source code for Camel Kafka Connector	Software Downloads > Red Hat Integration
Demonstration examples	Camel Kafka Connector examples and Debezium community example	Camel Kafka Connector examples Debezium PostgreSQL connector (Debezium community)

Note

You must have a subscription for Red Hat Integration and be logged into the Red Hat Customer Portal to access the Camel Kafka Connector distributions available with Red Hat Integration.