Search

Chapter 1. Introduction to Camel Kafka Connector

download PDF

This chapter introduces the features, concepts, and distributions provided by Camel Kafka Connector:

Important

Camel Kafka Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production.

These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview.

1.1. Camel Kafka Connector overview

Apache Camel is a highly flexible open source integration framework for connecting a wide range of different systems, which is based on standard Enterprise Integration Patterns (EIPs). Apache Kafka Connect is the Kafka-native approach for connecting to external systems, which is specifically designed for event-driven architectures.

Camel Kafka Connector enables you to use standard Camel components as Kafka Connect connectors. This widens the scope of possible integrations beyond the external systems supported by Kafka Connect connectors alone. Camel Kafka Connector works as an adapter that makes the popular Camel component ecosystem available in Kafka-based AMQ Streams on OpenShift.

Camel Kafka Connector provides a user-friendly way to configure Camel components directly in the Kafka Connect framework. Using Camel Kafka Connector, you can leverage Camel components for integration with different systems by connecting to or from Camel Kafka sink or source connectors. You do not need to write any code, and can include the appropriate connector JARs in your Kafka Connect image and configure connector options using custom resources.

Camel Kafka Connector is built on Apache Camel Kafka Connector, which is a subproject of the Apache Camel open source community. Camel Kafka Connector is fully integrated with OpenShift Container Platform, AMQ Streams, and Kafka Connect.

Camel Kafka Connector is available with the Red Hat Integration - Camel K distribution for cloud-native integration on OpenShift. Camel K is a lightweight integration framework built from Apache Camel K that runs natively in the cloud on OpenShift. Camel K is specifically designed for serverless and microservice architectures.

1.2. Camel Kafka Connector features

The Camel Kafka Connector Technology Preview includes the following main features:

  • OpenShift Container Platform 4.4 or 4.3
  • AMQ Streams 1.5
  • Kafka Connect 2.5
  • Camel 3.1
  • Selected Camel Kafka connectors
Table 1.1. Camel Kafka connectors in Technology Preview
ConnectorSink/source

Amazon AWS Kinesis

Sink and source

Amazon AWS S3

Sink and source

Cassandra Query Language (CQL)

Sink only

Elasticsearch

Sink only

Java Message Service (JMS)

Sink and source

Salesforce

Source only

Syslog

Source only

1.3. Camel Kafka Connector architecture

AMQ Streams is a distributed and scalable streaming platform based on Apache Kafka that includes a publish/subscribe messaging broker. Kafka Connect provides a framework to integrate Kafka-based systems with external systems. Using Kafka Connect, you can configure source and sink connectors to stream data from external systems into and out of a Kafka broker.

Camel Kafka Connector reuses the flexibility of Camel components and makes them available in Kafka Connect as source and sink connectors that you can use to stream data into and out of AMQ Streams. For example, you can ingest data from Amazon Web Services for processing using an AWS S3 source connector, or consolidate events stored in Kafka into an Elasticsearch instance for analytics using an Elasticsearch sink connector.

The following diagram shows a simplified view of the Camel Kafka Connector cloud-native integration architecture based on AMQ Streams on OpenShift:

Figure 1.1. Camel Kafka Connector architecture

Camel K Kafka Connect architecture

Kafka Connect concepts

The main Kafka Connect concepts include:

Source connector
Source connectors work like consumers and pull data from external systems into Kafka topics to make the data available for stream processing. For example, these external source systems include Amazon Web Services or Java Message Service.
Sink connector
Sink connectors work like producers and push data from Kafka topics into external systems for offline analysis. For example, these external sink systems include Cassandra, Syslog, or Elasticsearch.
Sink/source task
Tasks are typically created by a sink or source connector and are responsible for handling the data.
Key/value converter
Key/value converters can serialize/deserialize the key or value of a Kafka message in various formats.
Transformer
Transformers can manipulate Kafka message content.

Camel Kafka Connector configuration

You can use Camel Kafka Connector configuration to specify the following:

  • Kafka Connect configuration options
  • Camel route definitions
  • Camel configuration options

1.4. Camel Kafka Connector distributions

The Camel Kafka Connector distributions are bundled with Red Hat Integration - Camel K:

Table 1.2. Camel Kafka Connector available distributions
DistributionDescriptionLocation

Maven repository

Maven artifacts for Camel Kafka Connector

Software Downloads > Red Hat Integration - Camel K

Source code

Source code for Camel Kafka Connector

Software Downloads > Red Hat Integration - Camel K

Demonstration examples

Camel Kafka Connector examples and Debezium and Apache Camel Kafka Connector community example

Note

You must have a subscription for Red Hat Integration and be logged into the Red Hat Customer Portal to access the Camel Kafka Connector distributions available with Red Hat Integration - Camel K.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.