Chapter 7. Configuring memory usage for addresses
AMQ Broker transparently supports huge queues containing millions of messages, even if the machine that is hosting the broker is running with limited memory.
In these situations, it might be not possible to store all of the queues in memory at any one time. To protect against excess memory consumption, you can configure the maximum memory usage that is allowed for each address on the broker. In addition, you can configure the broker to take one of the following actions when memory usage for an address reaches the configured limit:
- Page messages
- Silently drop messages
- Drop messages and notify the sending clients
- Block clients from sending messages
If you configure the broker to page messages when the maximum memory usage for an address is reached, you can configure limits for specific addresses to:
- Limit the disk space used to page incoming messages
- Limit the memory used for paged messages that the broker transfers from disk back to memory when clients are ready to consume messages.
You can also set a disk usage threshold, which overrides all the configured paging limits. If the disk usage threshold is reached, the broker stops paging and blocks all incoming messages.
When you use transactions, the broker might allocate extra memory to ensure transactional consistency. In this case, the memory usage reported by the broker might not reflect the total number of bytes being used in memory. Therefore, if you configure the broker to page, drop, or block messages based on a specified maximum memory usage, you should not also use transactions.
7.1. Configuring message paging
For any address that has a maximum memory usage limit specified, you can also specify what action the broker takes when that usage limit is reached. One of the options that you can configure is paging.
If you configure the paging option, when the maximum size of an address is reached, the broker starts to store messages for that address on disk, in files known as page files. Each page file has a maximum size that you can configure. Each address that you configure in this way has a dedicated folder in your file system to store paged messages.
Both queue browsers and consumers can navigate through page files when inspecting messages in a queue. However, a consumer that is using a very specific filter might not be able to consume a message that is stored in a page file until existing messages in the queue have been consumed first. For example, suppose that a consumer filter includes a string expression such as "color='red'"
. If a message that meets this condition follows one million messages with the property "color='blue'"
, the consumer cannot consume the message until those with "color='blue'"
have been consumed first.
The broker transfers (that is, depages) messages from disk into memory when clients are ready to consume them. The broker removes a page file from disk when all messages in that file have been acknowledged.
The procedures that follow show how to configure message paging.
7.1.1. Specifying a paging directory
The following procedure shows how to specify the location of the paging directory.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. Within the
core
element, add thepaging-directory
element. Specify a location for the paging directory in your file system.<configuration ...> <core ...> ... <paging-directory>/path/to/paging-directory</paging-directory> ... </core> </configuration>
For each address that you subsequently configure for paging, the broker adds a dedicated directory within the paging directory that you have specified.
7.1.2. Configuring an address for paging
The following procedure shows how to configure an address for paging.
Prerequisites
- You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. For an
address-setting
element that you have configured for a matching address or set of addresses, add configuration elements to specify maximum memory usage and define paging behavior. For example:<address-settings> <address-setting match="my.paged.address"> ... <max-size-bytes>104857600</max-size-bytes> <max-size-messages>20000</max-size-messages> <page-size-bytes>10485760</page-size-bytes> <address-full-policy>PAGE</address-full-policy> ... </address-setting> </address-settings>
max-size-bytes
-
Maximum size, in bytes, of the memory allowed for the address before the broker executes the action specified for the
address-full-policy
attribute. The default value is-1
, which means that there is no limit. The value that you specify also supports byte notation such as "K", "MB", and "GB". max-size-messages
-
Maximum number of messages allowed for the address before the broker executes the action specified for the
address-full-policy
attribute. The default value is -1, which means that there is no message limit. page-size-bytes
-
Size, in bytes, of each page file used on the paging system. The default value is
10485760
(that is, 10 MiB). The value that you specify also supports byte notation such as "K", "MB", and "GB". address-full-policy
Action that the broker takes when the maximum size for an address has been reached. The default value is
PAGE
. Valid values are:PAGE
- The broker pages any further messages to disk.
DROP
- The broker silently drops any further messages.
FAIL
- The broker drops any further messages and issues exceptions to client message producers.
BLOCK
- Client message producers block when they try to send further messages.
If you set limits for the
max-size-bytes
andmax-size-message
attributes, the broker executes the action specified for theaddress-full-policy
attribute when either limit is reached. With the configuration in the previous example, the broker starts paging messages for themy.paged.address
address when the total messages for the address in memory exceeds 20,000 or uses 104857600 bytes of available memory.Additional paging configuration elements that are not shown in the preceding example are described below.
page-sync-timeout
-
Time, in nanoseconds, between periodic page synchronizations. If you are using an asynchronous IO journal (that is,
journal-type
is set toASYNCIO
in thebroker.xml
configuration file), the default value is3333333
. If you are using a standard Java NIO journal (that is,journal-type
is set toNIO
), the default value is the configured value of thejournal-buffer-timeout
parameter.
In the preceding example , when messages sent to the address my.paged.address
exceed 104857600 bytes in memory, the broker begins paging.
If you specify max-size-bytes
in an address-setting
element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value of max-size-bytes
.
7.1.3. Configuring a global paging size
Sometimes, configuring a memory limit per address is not practical, for example, when a broker manages many addresses that have different usage patterns. In these situations, you can specify a global memory limit. The global limit is the total amount of memory that the broker can use for all addresses. When this memory limit is reached, the broker executes the action specified for the address-full-policy
attribute for the address associated with each new incoming message.
The following procedure shows how to configure a global paging size.
Prerequisites
- You should be familiar with how to configure an address for paging. For more information, see Section 7.1.2, “Configuring an address for paging”.
Procedure
Stop the broker.
On Linux:
<broker_instance_dir>/bin/artemis stop
On Windows:
<broker_instance_dir>\bin\artemis-service.exe stop
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. Within the
core
element, add theglobal-max-size
element and specify a value. For example:<configuration> <core> ... <global-max-size>1GB</global-max-size> <global-max-messages>900000</global-max-messages> ... </core> </configuration>
global-max-size
Total amount of memory, in bytes, that the broker can use for all addresses. When this limit is reached, the broker executes the action specified for the
address-full-policy
attribute for the address associated with each incoming message. The default value ofglobal-max-size
is half of the maximum memory available to the Java virtual machine (JVM) that is hosting the broker.The value for
global-max-size
is in bytes, but also supports byte notation (for example, "K", "Mb", "GB").In the preceding example, the broker is configured to use a maximum of one gigabyte of available memory when processing messages.
global-max-messages
The total number of messages allowed for all addresses. When this limit is reached, the broker executes the action specified for the
address-full-policy
attribute for the address associated with each incoming message. The default value is -1, which means that there is no message limit.If you set limits for the
global-max-size
andglobal-max-messages
attributes, the broker executes the action specified for theaddress-full-policy
attribute when either limit is reached. With the configuration in the previous example, the broker starts paging messages for all addresses when the number of messages in memory exceeds 900,000 or uses 1 GB of available memory.NoteIf limits that are set for an individual address, by using the
max-size-bytes
ormax-size-message
attributes, are reached before the limits set for theglobal-max-size
orglobal-max-messages
attributes, the broker executes the action specified for theaddress-full-policy
attribute for that address.
Start the broker.
On Linux:
<broker_instance_dir>/bin/artemis run
On Windows:
<broker_instance_dir>\bin\artemis-service.exe start
7.1.4. Limiting disk usage during paging for specific addresses
You can limit the amount of disk space that the broker can use before it stops paging incoming messages for an individual address or set of addresses.
Procedure
Stop the broker.
On Linux:
<broker_instance_dir>/bin/artemis stop
On Windows:
<broker_instance_dir>\bin\artemis-service.exe stop
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. Within the
core
element, add attributes to specify paging limits based on disk usage or number of messages, or both, and specify the action to take if either limit is reached. For example:<address-settings> <address-setting match="match="my.paged.address""> ... <page-limit-bytes>10G</page-limit-bytes> <page-limit-messages>1000000</page-limit-messages> <page-full-policy>FAIL</page-full-policy> ... </address-setting> </address-settings>
page-limit-bytes
-
Maximum size, in bytes, of the disk space allowed for paging incoming messages for the address before the broker executes the action specified for the
page-full-policy
attribute. The value that you specify supports byte notation such as "K", "MB", and "GB". The default value is -1, which means that there is no limit. page-limit-messages
-
Maximum number of incoming message that can be paged for the address before the broker executes the action specified for the
page-full-policy
attribute. The default value is -1, which means that there is no message limit. page-full-policy
Action that the broker takes when a limit set in the
page-limit-bytes
orpage-limit-messages
attributes is reached for an address. Valid values are:DROP The broker silently drops any further messages.
FAIL The broker drops any further messages and notifies the sending clients
In the preceding example, the broker pages message for the
my.paged.address
address until paging uses 10GB of disk space or until a total of one million messages are paged.
Start the broker.
On Linux:
<broker_instance_dir>/bin/artemis run
On Windows:
<broker_instance_dir>\bin\artemis-service.exe start
7.1.5. Controlling the flow of paged messages into memory
If AMQ Broker is configured to page messages to disk, the broker reads paged messages and transfers the messages into memory when clients are ready to consume messages. To prevent messages from consuming excess memory, you can limit the memory used by each address for messages that the broker transfers from disk to memory.
If client applications leave too many messages pending acknowledgment, the broker does not read paged messages until the pending messages are acknowledged, which can cause message starvation on the broker.
For example, if the limit for the transfer of paged messages into memory, which is 20MB by default, is reached, the broker waits for an acknowledgment from a client before it reads any more messages. If, at the same time, clients are waiting to receive sufficient messages before they send an acknowledgment to the broker, which is determined by the batch size used by clients, the broker is starved of messages.
To avoid starvation, either increase the broker limits that control the transfer of paged message into memory or reduce the number of delivering messages. You can reduce the number of delivering messages by ensuring that clients either commit message acknowledgments sooner or use a timeout and commit acknowledgments when no more messages are received from the broker.
You can see the number and size of delivering messages in a queue’s Delivering Count
and Delivering Bytes
metrics in AMQ Management Console.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. -
For an
address-settings
element that you have configured for a matching address or set of addresses, specify limits on the transfer of paged messages into memory. For example:
address-settings> <address-setting match="my.paged.address"> ... <max-read-page-messages>104857600</max-read-page-messages> <max-read-page-bytes>20MB</max-read-page-bytes> ... </address-setting> </address-settings>
Max-read-page-messages
Maximum number of paged messages that the broker can read from disk into memory per-address. The default value is -1, which means that no limit applies.
Max-read-page-bytes
Maximum size in bytes of paged messages that the broker can read from disk into memory per-address. The default value is 20MB.
If you specify limits for both the max-read-page-messages
and max-read-page-bytes
attributes, the broker stops reading messages when either limit is reached.
7.1.6. Setting a disk usage threshold
You can set a disk usage threshold which, if reached, causes the broker to stop paging and block all incoming messages.
Procedure
Stop the broker.
On Linux:
<broker_instance_dir>/bin/artemis stop
On Windows:
<broker_instance_dir>\bin\artemis-service.exe stop
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. Within the
core
element add themax-disk-usage
configuration element and specify a value. For example:<configuration> <core> ... <max-disk-usage>80</max-disk-usage> ... </core> </configuration>
max-disk-usage
Maximum percentage of the available disk space that the broker can use. When this limit is reached, the broker blocks incoming messages. The default value is
90
.In the preceding example, the broker is limited to using eighty percent of available disk space.
Start the broker.
On Linux:
<broker_instance_dir>/bin/artemis run
On Windows:
<broker_instance_dir>\bin\artemis-service.exe start
7.2. Configuring message dropping
Section 7.1.2, “Configuring an address for paging” shows how to configure an address for paging. As part of that procedure, you set the value of address-full-policy
to PAGE
.
To drop messages (rather than paging them) when an address reaches its specified maximum size, set the value of the address-full-policy
to one of the following:
DROP
- When the maximum size of a given address has been reached, the broker silently drops any further messages.
FAIL
- When the maximum size of a given address has been reached, the broker drops any further messages and issues exceptions to producers.
7.3. Configuring message blocking
The following procedures show how to configure message blocking when a given address reaches the maximum size limit that you have specified.
You can configure message blocking only for the Core, OpenWire, and AMQP protocols.
7.3.1. Blocking Core and OpenWire producers
The following procedure shows how to configure message blocking for Core and OpenWire message producers when a given address reaches the maximum size limit that you have specified.
Prerequisites
- You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. For an
address-setting
element that you have configured for a matching address or set of addresses, add configuration elements to define message blocking behavior. For example:<address-settings> <address-setting match="my.blocking.address"> ... <max-size-bytes>300000</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> ... </address-setting> </address-settings>
max-size-bytes
Maximum size, in bytes, of the memory allowed for the address before the broker executes the policy specified for
address-full-policy
. The value that you specify also supports byte notation such as "K", "MB", and "GB".NoteIf you specify
max-size-bytes
in anaddress-setting
element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value ofmax-size-bytes
.address-full-policy
- Action that the broker takes when then the maximum size for an address has been reached.
In the preceding example, when messages sent to the address my.blocking.address
exceed 300000 bytes in memory, the broker begins blocking further messages from Core or OpenWire message producers.
7.3.2. Blocking AMQP producers
Protocols such as Core and OpenWire use a window-size flow control system. In this system, credits represent bytes and are allocated to producers. If a producer wants to send a message, the producer must wait until it has sufficient credits for the size of the message.
By contrast, AMQP flow control credits do not represent bytes. Instead, AMQP credits represent the number of messages a producer is permitted to send, regardless of message size. Therefore, it is possible, in some situations, for AMQP producers to significantly exceed the max-size-bytes
value of an address.
Therefore, to block AMQP producers, you must use a different configuration element, max-size-bytes-reject-threshold
. For a matching address or set of addresses, this element specifies the maximum size, in bytes, of all AMQP messages in memory. When the total size of all messages in memory reaches the specified limit, the broker blocks AMQP producers from sending further messages.
The following procedure shows how to configure message blocking for AMQP message producers.
Prerequisites
- You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues.
Procedure
-
Open the
<broker_instance_dir>/etc/broker.xml
configuration file. For an
address-setting
element that you have configured for a matching address or set of addresses, specify the maximum size of all AMQP messages in memory. For example:<address-settings> <address-setting match="my.amqp.blocking.address"> ... <max-size-bytes-reject-threshold>300000</max-size-bytes-reject-threshold> ... </address-setting> </address-settings>
max-size-bytes-reject-threshold
Maximum size, in bytes, of the memory allowed for the address before the broker blocks further AMQP messages. The value that you specify also supports byte notation such as "K", "MB", and "GB". By default,
max-size-bytes-reject-threshold
is set to-1
, which means that there is no maximum size.NoteIf you specify
max-size-bytes-reject-threshold
in anaddress-setting
element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value ofmax-size-bytes-reject-threshold
.
In the preceding example, when messages sent to the address my.amqp.blocking.address
exceed 300000 bytes in memory, the broker begins blocking further messages from AMQP producers.
7.4. Understanding memory usage on multicast addresses
When a message is routed to an address that has multicast queues bound to it, there is only one copy of the message in memory. Each queue has only a reference to the message. Because of this, the associated memory is released only after all queues referencing the message have delivered it.
In this type of situation, if you have a slow consumer, the entire address might experience a negative performance impact.
For example, consider this scenario:
- An address has ten queues that use the multicast routing type.
- Due to a slow consumer, one of the queues does not deliver its messages. The other nine queues continue to deliver messages and are empty.
- Messages continue to arrive to the address. The queue with the slow consumer continues to accumulate references to the messages, causing the broker to keep the messages in memory.
- When the maximum size of the address is reached, the broker starts to page messages.
In this scenario because of a single slow consumer, consumers on all queues are forced to consume messages from the page system, requiring additional IO.
Additional resources
- To learn how to configure flow control to regulate the flow of data between the broker and producers and consumers, see Flow control in the AMQ Core Protocol JMS documentation.