Chapter 8. Handling large messages
Clients might send large messages that can exceed the size of the broker’s internal buffer, causing unexpected errors. To prevent this situation, you can configure the broker to store messages as files when the messages are larger than a specified minimum value. Handling large messages in this way means that the broker does not hold the messages in memory. Instead, you specify a directory on disk or in a database table in which the broker stores large message files.
When the broker stores a message as a large message, the queue retains a reference to the file in the large messages directory or database table.
Large message handling is available for the Core Protocol, AMQP, OpenWire and STOMP protocols.
For the Core Protocol and OpenWire protocols, clients specify the minimum large message size in their connection configurations. For the AMQP and STOMP protocols, you specify the minimum large message size in the acceptor defined for each protocol in the broker configuration.
It is recommended that you do not use different protocols for producing and consuming large messages. To do this, the broker might need to perform several conversions of the message. For example, say that you want to send a message using the AMQP protocol and receive it using OpenWire. In this situation, the broker must first read the entire body of the large message and convert it to use the Core protocol. Then, the broker must perform another conversion, this time to the OpenWire protocol. Message conversions such as these cause significant processing overhead on the broker.
The minimum large message size that you specify for any of the preceding protocols is affected by system resources such as the amount of disk space available, as well as the sizes of the messages. It is recommended that you run performance tests using several values to determine an appropriate size.
The procedures in this section show how to:
- Configure the broker to store large messages
- Configure acceptors for the AMQP and STOMP protocols for large message handling
This section also links to additional resources about configuring AMQ Core Protocol and AMQ OpenWire JMS clients to work with large messages.
8.1. Configuring the broker for large message handling
The following procedure shows how to specify a directory on disk or a database table in which the broker stores large message files.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xmlconfiguration file. Specify where you want the broker to store large message files.
If you are storing large messages on disk, add the
large-messages-directoryparameter within thecoreelement and specify a file system location. For example:<configuration> <core> ... <large-messages-directory>/path/to/my-large-messages-directory</large-messages-directory> ... </core> </configuration>NoteIf you do not explicitly specify a value for
large-messages-directory, the broker uses a default value of<broker_instance_dir>/data/largemessagesIf you are storing large messages in a database table, add the
large-message-tableparameter to thedatabase-storeelement and specify a value. For example:<store> <database-store> ... <large-message-table>MY_TABLE</large-message-table> ... </database-store> </store>NoteIf you do not explicitly specify a value for
large-message-table, the broker uses a default value ofLARGE_MESSAGE_TABLE.
Additional resources
- For more information about configuring a database store, see Section 6.2, “Persisting message data in a database”.
8.2. Configuring AMQP acceptors for large message handling
The following procedure shows how to configure an AMQP acceptor to handle an AMQP message larger than a specified size as a large message.
Procedure
Open the
<broker_instance_dir>/etc/broker.xmlconfiguration file.The default AMQP acceptor in the broker configuration file looks as follows:
<acceptors> ... <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor> ... </acceptors>In the default AMQP acceptor (or another AMQP acceptor that you have configured), add the
amqpMinLargeMessageSizeproperty and specify a value. For example:<acceptors> ... <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=204800</acceptor> ... </acceptors>In the preceding example, the broker is configured to accept AMQP messages on port 5672. Based on the value of
amqpMinLargeMessageSize, if the acceptor receives an AMQP message with a body larger than or equal to 204800 bytes (that is, 200 kilobytes), the broker stores the message as a large message. If you do not explicitly specify a value for this property, the broker uses a default value of 102400 (that is, 100 kilobytes).
-
If you set
amqpMinLargeMessageSizeto -1, large message handling for AMQP messages is disabled. -
If the broker receives a persistent AMQP message that does not exceed the value of
amqpMinLargeMessageSize, but which does exceed the size of the messaging journal buffer (specified using thejournal-buffer-sizeconfiguration parameter), the broker converts the message to a large Core Protocol message, before storing it in the journal.
8.3. Configuring STOMP acceptors for large message handling
The following procedure shows how to configure a STOMP acceptor to handle a STOMP message larger than a specified size as a large message.
Procedure
Open the
<broker_instance_dir>/etc/broker.xmlconfiguration file.The default AMQP acceptor in the broker configuration file looks as follows:
<acceptors> ... <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor> ... </acceptors>In the default STOMP acceptor (or another STOMP acceptor that you have configured), add the
stompMinLargeMessageSizeproperty and specify a value. For example:<acceptors> ... <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true;stompMinLargeMessageSize=204800</acceptor> ... </acceptors>
In the preceding example, the broker is configured to accept STOMP messages on port 61613. Based on the value of stompMinLargeMessageSize, if the acceptor receives a STOMP message with a body larger than or equal to 204800 bytes (that is, 200 kilobytes), the broker stores the message as a large message. If you do not explicitly specify a value for this property, the broker uses a default value of 102400 (that is, 100 kilobytes).
To deliver a large message to a STOMP consumer, the broker automatically converts the message from a large message to a normal message before sending it to the client. If a large message is compressed, the broker decompresses it before sending it to STOMP clients.
8.4. Large messages and Java clients
There are two options available to Java developers who are writing clients that use large messages.
One option is to use instances of InputStream and OutputStream. For example, a FileInputStream can be used to send a message taken from a large file on a physical disk. A FileOutputStream can then be used by the receiver to stream the message to a location on its local file system.
Another option is to stream a JMS BytesMessage or StreamMessage directly. For example:
BytesMessage rm = (BytesMessage)cons.receive(10000);
byte data[] = new byte[1024];
for (int i = 0; i < rm.getBodyLength(); i += 1024)
{
int numberOfBytes = rm.readBytes(data);
// Do whatever you want with the data
}Additional resources
To learn about working with large messages in the AMQ Core Protocol JMS client, see:
- For an example of working with large messages, see the large message example. To learn more about running example programs, see Running the AMQ Broker examples.
