Chapter 13. Duplicate Message Detection
AMQ Broker includes automatic duplicate message detection, which filters out any duplicate messages it receives so you do not have to code your own duplicate detection logic.
Without duplicate detection, a client cannot determine whether a message it sent was successful whenever the target broker or the connection to it fails. For example, if the broker or connection fails before the message was received and processed by the broker, the message never arrives at its address, and the client does not receive a response from the broker due to the failure. On the other hand, if the broker or connection failed after a message was received and processed by the broker, the message is routed correctly, but the client still does not receive a response.
Moreover, using a transaction to determine success does not help in these cases. If the broker or connection fails while the transaction commit is being processed, for example, the client is still unable to determine whether it successfully sent the message.
If the client resends the last message in an effort to correct the assumed failure, the result could be a duplicate message being sent to the address, which could negatively impact your system. Sending a duplicate message could mean that a purchase order is fulfilled twice, for example. Fortunately, {AMQ Broker} provides automatic duplicate messages detection as a way to prevent these kind of issues from happening.
13.1. Using the Duplicate ID Message Property
To enable duplicate message detection provide a unique value for the message property _AMQ_DUPL_ID. When a broker receives a message, it checks if _AMQ_DUPL_ID has a value. If it does, the broker then checks in its memory cache to see if it has already received a message with that value. If a message with the same value is found, the incoming message is ignored.
Procedure
The examples below illustrate how to set the duplicate detection property using a Core JMS Client. Note that for convenience, the clients use the value of the constant org.apache.activemq.artemis.api.core.Message.HDR_DUPLICATE_DETECTION_ID for the name of the duplicate ID property, _AMQ_DUPL_ID.
Set the value for
_AMQ_DUPL_IDto a uniqueString.Message jmsMessage = session.createMessage(); String myUniqueID = "This is my unique id"; message.setStringProperty(HDR_DUPLICATE_DETECTION_ID.toString(), myUniqueID);
13.2. Configuring the Duplicate ID Cache
The broker maintains caches of received values of the _AMQ_DUPL_ID property. Each address has its own distinct cache. The cache is circular and fixed. New entries replace the oldest ones as cache space demands.
Be sure to size the cache appropriately. If a previous message arrived more than id-cache-size messages before the arrival of a new message with the same _AMQ_DUPL_ID, the broker cannot detect the duplicate. This results in both messages being processed by the broker.
Procedure
The example configuration below illustrates how to configure the ID cache by adding elements to BROKER_INSTANCE_DIR/etc/broker.xml.
<configuration>
<core>
...
<id-cache-size>5000</id-cache-size> 1
<persist-id-cache>false</persist-id-cache> 2
</core>
</configuration>- 1
- The maximum size of the cache is configured by the parameter
id-cache-size. The default value is20000entries. In the example above, the cache size is set to5000entries. - 2
- Set
persist-id-cachetotrueto have each ID persisted to disk as they are received. The default value istrue. In the example above, persistence is disabled by setting the value tofalse.
13.3. Duplicate Detection and Transactions
Using duplicate detection to move messages between brokers can give you the same once and only once delivery guarantees as using an XA transaction to consume messages, but with less overhead and much easier configuration than using XA.
If you are sending messages in a transaction, you do not have to set _AMQ_DUPL_ID for every message in the transaction, but only in one of them. If the broker detects a duplicate message for any message in the transaction, it ignores the entire transaction.
13.4. Duplicate Detection and Cluster Connections
You can configure cluster connections to insert a duplicate ID for each message they move across the cluster.
Procedure
Add the element
use-duplicate-detectionto the configuration of the desired cluster connection found inBROKER_INSTANCE_DIR/etc/broker.xml. Note that the default value for this parameter istrue. In the example below, the element is added to the configuration for the cluster connectionmy-cluster.<configuration> <core> ... <cluster-connection> <cluster-connection name="my-cluster"> 2 <use-duplicate-detection>true</use-duplicate-detection> ... </cluster-connection> ... </cluster-connections> </core> </configuration>
Related Information
For more information about broker clusters, see Section 16.2, “Creating a broker cluster”.
