Chapter 22. Elasticsearch
Since Camel 3.18.3
Only producer is supported
The ElasticSearch component allows you to interface with an ElasticSearch 8.x API using the Java API Client library.
Add the following dependency to your pom.xml
for this component:
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-elasticsearch</artifactId> <version>3.20.1.redhat-00050</version> <!-- use the same version as your Camel core version --> </dependency>
22.1. URI format
elasticsearch://clusterName[?options]
22.2. Configuring Options
Camel components are configured on two separate levels:
- component level
- endpoint level
22.2.1. Configuring Component Options
The component level is the highest level which holds general and common configurations that are inherited by the endpoints. For example a component may have security settings, credentials for authentication, urls for network connection and so forth.
Some components only have a few options, and others may have many. Because components typically have pre configured defaults that are commonly used, then you may often only need to configure a few options on a component; or none at all.
Configuring components can be done with the Component DSL, in a configuration file (application.properties|yaml), or directly with Java code.
22.2.2. Configuring Endpoint Options
Where you find yourself configuring the most is on endpoints, as endpoints often have many options, which allows you to configure what you need the endpoint to do. The options are also categorized into whether the endpoint is used as consumer (from) or as a producer (to), or used for both.
Configuring endpoints is most often done directly in the endpoint URI as path and query parameters. You can also use the Endpoint DSL as a type safe way of configuring endpoints.
A good practice when configuring options is to use Property Placeholders, which allows to not hardcode urls, port numbers, sensitive information, and other settings. In other words placeholders allows to externalize the configuration from your code, and gives more flexibility and reuse.
The following two sections lists all the options, firstly for the component followed by the endpoint.
22.3. Component Options
The Elasticsearch component supports 14 options, which are listed below.
Name | Description | Default | Type |
---|---|---|---|
connectionTimeout (producer) | The time in ms to wait before connection will timeout. | 30000 | int |
hostAddresses (producer) | Comma separated list with ip:port formatted remote transport addresses to use. The ip and port options must be left blank for hostAddresses to be considered instead. | String | |
lazyStartProducer (producer) | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | boolean |
maxRetryTimeout (producer) | The time in ms before retry. | 30000 | int |
socketTimeout (producer) | The timeout in ms to wait before the socket will timeout. | 30000 | int |
autowiredEnabled (advanced) | Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. | true | boolean |
client (advanced) | Autowired To use an existing configured Elasticsearch client, instead of creating a client per endpoint. This allow to customize the client with specific settings. | RestClient | |
enableSniffer (advanced) | Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it’s managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). | false | boolean |
sniffAfterFailureDelay (advanced) | The delay of a sniff execution scheduled after a failure (in milliseconds). | 60000 | int |
snifferInterval (advanced) | The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. | 300000 | int |
certificatePath (security) | The path of the self-signed certificate to use to access to Elasticsearch. | String | |
enableSSL (security) | Enable SSL. | false | boolean |
password (security) | Password for authenticate. | String | |
user (security) | Basic authenticate user. | String |
22.4. Endpoint Options
The Elasticsearch endpoint is configured using URI syntax:
elasticsearch:clusterName
with the following path and query parameters:
22.4.1. Path Parameters (1 parameters)
Name | Description | Default | Type |
---|---|---|---|
clusterName (producer) | Required Name of the cluster. | String |
22.4.2. Query Parameters (19 parameters)
Name | Description | Default | Type |
---|---|---|---|
connectionTimeout (producer) | The time in ms to wait before connection will timeout. | 30000 | int |
disconnect (producer) | Disconnect after it finish calling the producer. | false | boolean |
from (producer) | Starting index of the response. | Integer | |
hostAddresses (producer) | Comma separated list with ip:port formatted remote transport addresses to use. | String | |
indexName (producer) | The name of the index to act against. | String | |
maxRetryTimeout (producer) | The time in ms before retry. | 30000 | int |
operation (producer) | What operation to perform. Enum values:
| ElasticsearchOperation | |
scrollKeepAliveMs (producer) | Time in ms during which elasticsearch will keep search context alive. | 60000 | int |
size (producer) | Size of the response. | Integer | |
socketTimeout (producer) | The timeout in ms to wait before the socket will timeout. | 30000 | int |
useScroll (producer) | Enable scroll usage. | false | boolean |
waitForActiveShards (producer) | Index creation waits for the write consistency number of shards to be available. | 1 | int |
lazyStartProducer (producer (advanced)) | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | boolean |
documentClass (advanced) | The class to use when deserializing the documents. | ObjectNode | Class |
enableSniffer (advanced) | Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it’s managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). | false | boolean |
sniffAfterFailureDelay (advanced) | The delay of a sniff execution scheduled after a failure (in milliseconds). | 60000 | int |
snifferInterval (advanced) | The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. | 300000 | int |
certificatePath (security) | The path of the self-signed certificate to use to access to Elasticsearch. | String | |
enableSSL (security) | Enable SSL. | false | boolean |
22.5. Message Headers
The Elasticsearch component supports 9 message header(s), which is/are listed below:
Name | Description | Default | Type |
---|---|---|---|
operation (producer) Constant: PARAM_OPERATION | The operation to perform. Enum values:
| ElasticsearchOperation | |
indexId (producer) Constant: PARAM_INDEX_ID | The id of the indexed document. | String | |
indexName (producer) Constant: PARAM_INDEX_NAME | The name of the index to act against. | String | |
documentClass (producer) Constant: PARAM_DOCUMENT_CLASS | The full qualified name of the class of the document to unmarshall. | ObjectNode | Class |
waitForActiveShards (producer) Constant: PARAM_WAIT_FOR_ACTIVE_SHARDS | The index creation waits for the write consistency number of shards to be available. | Integer | |
scrollKeepAliveMs (producer) Constant: PARAM_SCROLL_KEEP_ALIVE_MS | The starting index of the response. | Integer | |
useScroll (producer) Constant: PARAM_SCROLL | Set to true to enable scroll usage. | Boolean | |
size (producer) Constant: PARAM_SIZE | The size of the response. | Integer | |
from (producer) Constant: PARAM_FROM | The starting index of the response. | Integer |
22.6. Message Operations
The following ElasticSearch operations are currently supported. Simply set an endpoint URI option or exchange header with a key of "operation" and a value set to one of the following. Some operations also require other parameters or the message body to be set.
operation | message body | description |
---|---|---|
Index | Map, String, byte[], Reader, InputStream or IndexRequest.Builder content to index | Adds content to an index and returns the content’s indexId in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the indexId by setting the message header with the key "indexId". |
GetById | String or GetRequest.Builder index id of content to retrieve | Retrieves the document corresponding to the given index id and returns a GetResponse object in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the type of document by setting the message header with the key "documentClass". |
Delete | String or DeleteRequest.Builder index id of content to delete | Deletes the specified indexName and returns a Result object in the body. You can set the name of the target index by setting the message header with the key "indexName". |
DeleteIndex | String or DeleteIndexRequest.Builder index name of the index to delete | Deletes the specified indexName and returns a status code in the body. You can set the name of the target index by setting the message header with the key "indexName". |
Bulk | Iterable or BulkRequest.Builder of any type that is already accepted (DeleteOperation.Builder for delete operation, UpdateOperation.Builder for update operation, CreateOperation.Builder for create operation, byte[], InputStream, String, Reader, Map or any document type for index operation) | Adds/Updates/Deletes content from/to an index and returns a List<BulkResponseItem> object in the body You can set the name of the target index by setting the message header with the key "indexName". |
Search | Map, String or SearchRequest.Builder | Search the content with the map of query string. You can set the name of the target index by setting the message header with the key "indexName". You can set the number of hits to return by setting the message header with the key "size". You can set the starting document offset by setting the message header with the key "from". |
MultiSearch | MsearchRequest.Builder | Multiple search in one |
MultiGet | Iterable<String> or MgetRequest.Builder the id of the document to retrieve | Multiple get in one You can set the name of the target index by setting the message header with the key "indexName". |
Exists | None | Checks whether the index exists or not and returns a Boolean flag in the body. You must set the name of the target index by setting the message header with the key "indexName". |
Update | byte[], InputStream, String, Reader, Map or any document type content to update | Updates content to an index and returns the content’s indexId in the body. You can set the name of the target index by setting the message header with the key "indexName". You can set the indexId by setting the message header with the key "indexId". |
Ping | None | Pings the Elasticsearch cluster and returns true if the ping succeeded, false otherwise |
22.7. Configure the component and enable basic authentication
To use the Elasticsearch component it has to be configured with a minimum configuration.
ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); camelContext.addComponent("elasticsearch", elasticsearchComponent);
For basic authentication with elasticsearch or using reverse http proxy in front of the elasticsearch cluster, simply setup basic authentication and SSL on the component like the example below
ElasticsearchComponent elasticsearchComponent = new ElasticsearchComponent(); elasticsearchComponent.setHostAddresses("myelkhost:9200"); elasticsearchComponent.setUser("elkuser"); elasticsearchComponent.setPassword("secure!!"); elasticsearchComponent.setEnableSSL(true); elasticsearchComponent.setCertificatePath(certPath); camelContext.addComponent("elasticsearch", elasticsearchComponent);
22.8. Index Example
Below is a simple INDEX example
from("direct:index") .to("elasticsearch://elasticsearch?operation=Index&indexName=twitter");
<route> <from uri="direct:index"/> <to uri="elasticsearch://elasticsearch?operation=Index&indexName=twitter"/> </route>
For this operation you need to specify a indexId header.
A client would simply need to pass a body message containing a Map to the route. The result body contains the indexId created.
Map<String, String> map = new HashMap<String, String>(); map.put("content", "test"); String indexId = template.requestBody("direct:index", map, String.class);
22.9. Search Example
Searching on specific field(s) and value use the Operation ´Search´. Pass in the query JSON String or the Map
from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter");
<route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&indexName=twitter"/> </route>
String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class);
Search on specific field(s) using Map.
Map<String, Object> actualQuery = new HashMap<>(); actualQuery.put("doc.content", "new release of ApacheCamel"); Map<String, Object> match = new HashMap<>(); match.put("match", actualQuery); Map<String, Object> query = new HashMap<>(); query.put("query", match); HitsMetadata<?> response = template.requestBody("direct:search", query, HitsMetadata.class);
Search using Elasticsearch scroll api in order to fetch all results.
from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000");
<route> <from uri="direct:search"/> <to uri="elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000"/> </route>
String query = "{\"query\":{\"match\":{\"doc.content\":\"new release of ApacheCamel\"}}}"; try (ElasticsearchScrollRequestIterator response = template.requestBody("direct:search", query, ElasticsearchScrollRequestIterator.class)) { // do something smart with results }
can also be used.
from("direct:search") .to("elasticsearch://elasticsearch?operation=Search&indexName=twitter&useScroll=true&scrollKeepAliveMs=30000") .split() .body() .streaming() .to("mock:output") .end();
22.10. MultiSearch Example
MultiSearching on specific field(s) and value use the Operation ´MultiSearch´. Pass in the MultiSearchRequest instance
from("direct:multiSearch") .to("elasticsearch://elasticsearch?operation=MultiSearch");
<route> <from uri="direct:multiSearch"/> <to uri="elasticsearch://elasticsearch?operation=MultiSearch"/> </route>
MultiSearch on specific field(s)
MsearchRequest.Builder builder = new MsearchRequest.Builder().index("twitter").searches( new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build(), new RequestItem.Builder().header(new MultisearchHeader.Builder().build()) .body(new MultisearchBody.Builder().query(b -> b.matchAll(x -> x)).build()).build()); List<MultiSearchResponseItem<?>> response = template.requestBody("direct:multiSearch", builder, List.class);
22.11. Document type
For all the search operations, it is possible to indicate the type of document to retrieve in order to get the result already unmarshalled with the expected type.
The document type can be set using the header "documentClass" or via the uri parameter of the same name.
22.12. Using Camel Elasticsearch with Spring Boot
When you use camel-elasticsearch-starter
with Spring Boot v2, then you must declare the following dependency in your own pom.xml
.
<dependency> <groupId>jakarta.json</groupId> <artifactId>jakarta.json-api</artifactId> <version>2.0.2</version> </dependency>
This is needed because Spring Boot v2 provides jakarta.json-api:1.1.6, and Elasticsearch requires to use json-api v2.
22.12.1. Use RestClient provided by Spring Boot
By default Spring Boot will auto configure an Elasticsearch RestClient that will be used by camel, it is possible to customize the client with the following basic properties:
spring.elasticsearch.uris=myelkhost:9200 spring.elasticsearch.username=elkuser spring.elasticsearch.password=secure!!
More information can be found in application-properties.data.spring.elasticsearch.connection-timeout.
22.12.2. Disable Sniffer when using Spring Boot
When Spring Boot is on the classpath the Sniffer client for Elasticsearch is enabled by default. This option can be disabled in the Spring Boot Configuration:
spring: autoconfigure: exclude: org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration
22.13. Spring Boot Auto-Configuration
When using elasticsearch with Spring Boot make sure to use the following Maven dependency to have support for auto configuration:
<dependency> <groupId>org.apache.camel.springboot</groupId> <artifactId>camel-elasticsearch-starter</artifactId> </dependency>
The component supports 15 options, which are listed below.
Name | Description | Default | Type |
---|---|---|---|
camel.component.elasticsearch.autowired-enabled | Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. | true | Boolean |
camel.component.elasticsearch.certificate-path | The path of the self-signed certificate to use to access to Elasticsearch. | String | |
camel.component.elasticsearch.client | To use an existing configured Elasticsearch client, instead of creating a client per endpoint. This allow to customize the client with specific settings. The option is a org.elasticsearch.client.RestClient type. | RestClient | |
camel.component.elasticsearch.connection-timeout | The time in ms to wait before connection will timeout. | 30000 | Integer |
camel.component.elasticsearch.enable-s-s-l | Enable SSL. | false | Boolean |
camel.component.elasticsearch.enable-sniffer | Enable automatically discover nodes from a running Elasticsearch cluster. If this option is used in conjunction with Spring Boot then it’s managed by the Spring Boot configuration (see: Disable Sniffer in Spring Boot). | false | Boolean |
camel.component.elasticsearch.enabled | Whether to enable auto configuration of the elasticsearch component. This is enabled by default. | Boolean | |
camel.component.elasticsearch.host-addresses | Comma separated list with ip:port formatted remote transport addresses to use. The ip and port options must be left blank for hostAddresses to be considered instead. | String | |
camel.component.elasticsearch.lazy-start-producer | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | Boolean |
camel.component.elasticsearch.max-retry-timeout | The time in ms before retry. | 30000 | Integer |
camel.component.elasticsearch.password | Password for authenticate. | String | |
camel.component.elasticsearch.sniff-after-failure-delay | The delay of a sniff execution scheduled after a failure (in milliseconds). | 60000 | Integer |
camel.component.elasticsearch.sniffer-interval | The interval between consecutive ordinary sniff executions in milliseconds. Will be honoured when sniffOnFailure is disabled or when there are no failures between consecutive sniff executions. | 300000 | Integer |
camel.component.elasticsearch.socket-timeout | The timeout in ms to wait before the socket will timeout. | 30000 | Integer |
camel.component.elasticsearch.user | Basic authenticate user. | String |