SupportConsoleDevelopersStart a trialContact

Chapter 17. AWS S3 Streaming upload Sink

Upload data to AWS S3 in streaming upload mode.

17.1. Authentication methods
Copy link

In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.

The order of evaluation for Default Credentials Provider is the following:

  • Java system properties - aws.accessKeyId and aws.secretKey.
  • Environment variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  • Web Identity Token from AWS STS.
  • The shared credentials and config files.
  • Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set.
  • Amazon EC2 Instance profile credentials.

You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.

Only one of access key/secret key or default credentials provider could be used

For more information, see the AWS credentials documentation

17.2. Optional Headers
Copy link

In the header, you can optionally set the file / ce-file property to specify the name of the file to upload.

If you do not set the property in the header, the Kamelet uses the exchange ID for the file name.

17.3. Configuration Options
Copy link

The following table summarizes the configuration options available for the aws-s3-streaming-upload-sink Kamelet:

Expand
PropertyNameDescriptionTypeDefaultExample

bucketNameOrArn *

Bucket Name

The S3 Bucket name or Amazon Resource Name (ARN)..

string

  

keyName *

Key Name

Setting the key name for an element in the bucket through endpoint parameter. In Streaming Upload, with the default configuration, this is the base for the progressive creation of files.

string

  

region *

AWS Region

The AWS region to access.

string

  

accessKey

Access Key

The access key obtained from AWS.

string

  

autoCreateBucket

Autocreate Bucket

Setting the autocreation of the S3 bucket bucketName.

boolean

False

 

batchMessageNumber

Batch Message Number

The number of messages composing a batch in streaming upload mode.

integer

10

 

batchSize

Batch Size

The batch size (in bytes) in streaming upload mode.

integer

1000000

 

forcePathStyle

Force Path Style

Forces path style when accessing AWS S3 buckets.

boolean

False

 

namingStrategy

Naming Strategy

The naming strategy to use in streaming upload mode. There are 2 enums and the value can be one of progressive, random.

string

progressive

 

overrideEndpoint

Endpoint Overwrite

Select this option to override the endpoint URI. To use this option, you must also provide a URI for the uriEndpointOverride option.

boolean

False

 

profileCredentialsName

Profile Credentials Name

If using a profile credentials provider this parameter sets the profile name.

string

  

restartingPolicy

Restarting Policy

The restarting policy to use in streaming upload mode. There are 2 enums and the value can be one of override, lastPart.

string

lastPart

 

secretKey

Secret Key

The secret key obtained from AWS.

string

  

sessionToken

Session Token

Amazon AWS Session Token used when the user needs to assume a IAM role.

string

  

streamingUploadTimeout

Streaming Upload Timeout

While streaming upload mode is true, this option set the timeout to complete upload.

integer

  

uriEndpointOverride

Overwrite Endpoint URI

The overriding endpoint URI. To use this option, you must also select the overrideEndpoint option.

string

  

useDefaultCredentialsProvider

Default Credentials Provider

Set whether the S3 client should expect to load credentials through a default credentials provider or to expect static credentials to be passed in.

boolean

False

 

useProfileCredentialsProvider

Profile Credentials Provider

Set whether the S3 client should expect to load credentials through a profile credentials provider.

boolean

False

 

useSessionCredentials

Session Credentials

Set whether the S3 client should expect to use Session Credentials. This is useful in situation in which the user needs to assume a IAM role for doing operations in S3.

boolean

False

 

* = Fields marked with an asterisk are mandatory.

17.4. Dependencies
Copy link

17.4.1. Quarkus dependencies
Copy link

<dependencies>
  <dependency>
    <groupId>org.apache.camel.kamelets</groupId>
    <artifactId>camel-kamelets-utils</artifactId>
    <version>4.8.5</version>
  </dependency>
  <dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-aws-s3-streaming-upload</artifactId>
  </dependency>
  <dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-aws2-s3</artifactId>
  </dependency>
  <dependency>
    <groupId>org.apache.camel.quarkus</groupId>
    <artifactId>camel-quarkus-kamelet</artifactId>
  </dependency>
</dependencies>

17.5. Usage
Copy link

17.5.1. Camel JBang usage
Copy link

17.5.1.1. Prerequisites for JBang
Copy link

  • Install JBang.

  • You have executed the following command: 

    jbang app install camel@apache/camel

17.5.1.2. Running a route with JBang
Copy link

Suppose you have a file named route.yaml with this content: 

- route:
    from:
      uri: "kamelet:timer-source"
      parameters:
        period: 10000
        message: 'test'
      steps:
        - to:
            uri: "kamelet:log-sink"

You can now run it directly through the following command. 

camel run route.yaml

17.5.2. Knative Sink
Copy link

You can use the aws-s3-streaming-upload-sink Kamelet as a Knative sink by binding it to a Knative object.

aws-s3-streaming-upload-sink-binding.yaml

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: aws-s3-streaming-upload-sink-binding
spec:
  source:
    ref:
      kind: Channel
      apiVersion: messaging.knative.dev/v1
      name: mychannel
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: aws-s3-streaming-upload-sink
    properties:
      accessKey: "The Access Key"
      bucketNameOrArn: "The Bucket Name"
      keyName: "The Key Name"
      region: "eu-west-1"
      secretKey: "The Secret Key"

17.5.3. Kafka Sink
Copy link

You can use the aws-s3-streaming-upload-sink Kamelet as a Kafka sink by binding it to a Kafka topic.

aws-s3-streaming-upload-sink-binding.yaml

apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  name: aws-s3-streaming-upload-sink-binding
spec:
  source:
    ref:
      kind: KafkaTopic
      apiVersion: kafka.strimzi.io/v1beta1
      name: my-topic
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1
      name: aws-s3-streaming-upload-sink
    properties:
      accessKey: "The Access Key"
      bucketNameOrArn: "The Bucket Name"
      keyName: "The Key Name"
      region: "eu-west-1"
      secretKey: "The Secret Key"

17.6. Kamelets source file
Copy link

https://github.com/jboss-fuse/camel-kamelets/blob/camel-kamelets-4.10.3-branch/kamelets/aws-s3-streaming-upload-sink.kamelet.yaml

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top