Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 8. AWS Kinesis Source
Receive data from AWS Kinesis.
8.1. Authentication methods
In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.
The order of evaluation for Default Credentials Provider is the following:
-
Java system properties -
aws.accessKeyIdandaws.secretKey. -
Environment variables -
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. - Web Identity Token from AWS STS.
- The shared credentials and config files.
-
Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIis set. - Amazon EC2 Instance profile credentials.
You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.
Only one of access key/secret key or default credentials provider could be used
For more information, see the AWS credentials documentation
8.2. Usage example with plain consumer
You could consume the stream content directly
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
useDefaultCredentialsProvider: true
region: "eu-west-1"
stream: "kamelets"
steps:
- to:
uri: "kamelet:log-sink"8.3. Usage example with KCL Consumer
You could consume the stream content with the KCL support
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
stream: "kamelets"
useDefaultCredentialsProvider: true
region: "eu-west-1"
asyncClient: true
useKclConsumers: true
steps:
- to:
uri: "kamelet:log-sink"
parameters:
showHeaders: true
With the useKclConsumers enabled, you won’t have to deal with shard iteration directly. Everything is managed by the AWS Kinesis client library and the KCL layer.
As a side note you need to remember that the KCL consumer will need access to DynamoDB and Cloudwatch services from AWS, so it will create clients to these services under the hood and it will use them.
8.4. Configuration Options
The following table summarizes the configuration options available for the aws-kinesis-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
| region * | AWS Region | The AWS region to access. | string | ||
| stream * | Stream Name | The Kinesis stream that you want to access. The Kinesis stream that you specify must already exist. | string | ||
| accessKey | Access Key | The access key obtained from AWS. | string | ||
| asyncClient | Async Client | If we want a KinesisAsyncClient instance set it to true. | boolean | False | |
| delay | Delay | The number of milliseconds before the next poll of the selected stream. | integer | 500 | |
| kclDisableCloudwatchMetricsExport | KCL Disable Cloudwatch Metrics Export | Define if we want to use a KCL Consumer and disable the CloudWatch Metrics Export | boolean | False | |
| overrideEndpoint | Endpoint Overwrite |
Select this option to override the endpoint URI. To use this option, you must also provide a URI for the | boolean | False | |
| profileCredentialsName | Profile Credentials Name | If using a profile credentials provider this parameter sets the profile name. | string | ||
| secretKey | Secret Key | The secret key obtained from AWS. | string | ||
| sessionToken | Session Token | Amazon AWS Session Token used when the user needs to assume a IAM role. | string | ||
| uriEndpointOverride | Overwrite Endpoint URI |
The overriding endpoint URI. To use this option, you must also select the | string | ||
| useDefaultCredentialsProvider | Default Credentials Provider | If true, the Kinesis client loads credentials through a default credentials provider. If false, it uses the basic authentication method (access key and secret key). | boolean | False | |
| useKclConsumers | KCL Consumer | If we want to a KCL Consumer set it to true | boolean | False | |
| useProfileCredentialsProvider | Profile Credentials Provider | Set whether the Kinesis client should expect to load credentials through a profile credentials provider. | boolean | False | |
| useSessionCredentials | Session Credentials | Set whether the Kinesis client should expect to use Session Credentials. This is useful in situation in which the user needs to assume a IAM role for doing operations in Kinesis. | boolean | False |
* = Fields marked with an asterisk are mandatory.
8.5. Dependencies
8.5.1. Quarkus dependencies
<dependencies>
<dependency>
<groupId>org.apache.camel.kamelets</groupId>
<artifactId>camel-kamelets-utils</artifactId>
<version>4.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-aws-kinesis-source</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-aws2-kinesis</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-kamelet</artifactId>
</dependency>
</dependencies>8.6. Usage
8.6.1. Camel JBang usage
8.6.1.1. Prerequisites for JBang
- Install JBang.
You have executed the following command:
jbang app install camel@apache/camel
8.6.1.2. Running a route with JBang
Suppose you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:log-sink"You can now run it directly through the following command.
camel run route.yaml8.6.2. Knative Source
You can use the aws-kinesis-source Kamelet as a Knative source by binding it to a Knative object.
aws-kinesis-source-binding.yaml
apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
name: aws-kinesis-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-kinesis-source
properties:
accessKey: "The Access Key"
region: "eu-west-1"
secretKey: "The Secret Key"
stream: "The Stream Name"
sink:
ref:
kind: Channel
apiVersion: messaging.knative.dev/v1
name: mychannel8.6.3. Kafka Source
You can use the aws-kinesis-source Kamelet as a Kafka source by binding it to a Kafka topic.
aws-kinesis-source-binding.yaml
apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
name: aws-kinesis-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-kinesis-source
properties:
accessKey: "The Access Key"
region: "eu-west-1"
secretKey: "The Secret Key"
stream: "The Stream Name"
sink:
ref:
kind: KafkaTopic
apiVersion: kafka.strimzi.io/v1beta1
name: my-topic
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}