Chapter 8. Message Persistence and Paging
AMQ Broker 7 provides persistence through either a message journal or a JDBC store. The method by which the broker stores messages and pages them to disk is different than AMQ 6, and the configuration properties you use to configure message persistence are changed.
8.1. Message Persistence Changes
AMQ Broker 7 uses a different type of message journal than AMQ 6, and it does not use a journal index.
AMQ 6 used KahaDB for a message store, and it maintained a message journal index to track the position of each message inside the journal. This index enabled the broker to pull paged messages from its journal in batches and place them in its cache.
By default, AMQ Broker 7 uses an in-memory message journal from which the broker can dispatch messages. Therefore, AMQ Broker 7 does not use a message journal index. If a broker instance runs out of memory, messages are paged as they arrive at the broker, but before they are queued. These message page files are stored on disk sequentially in the same order in which they arrived. Then, when memory is freed on the broker, the messages are moved from the page file to the journal on the broker. Because the journal is read sequentially, there is no need to keep an index of messages in the journal.
In addition, AMQ Broker 7 also offers a different JDBC-based message journal option that was not available in AMQ 6.
The AMQ Broker 7 message journal supports the following shared file systems:
- NFSv4
- GFS2
Related Information
- For more information about the default in-memory message journal, see About Journal-based Persistence in Configuring AMQ Broker.
- For more information about the new JDBC-based persistence option, see Configuring JDBC Persistence in Configuring AMQ Broker.
8.2. How Message Persistence is Configured
You use the BROKER_INSTANCE_DIR/etc/broker.xml configuration file to configure the broker instance’s message journal.
The broker.xml configuration file contains the following default message journal configuration properties:
<core>
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>./data/paging</paging-directory>
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>-1</journal-pool-files>
<journal-buffer-timeout>744000</journal-buffer-timeout>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<global-max-size>104857600</global-max-size>
...
</core>To configure the message journal, you can change the default values for any of the journal configuration properties. You can also add additional configuration properties.
8.3. Message Persistence Configuration Property Changes
AMQ 6 and AMQ Broker 7 both offer a number of configuration properties to control how the broker persists messages. This section compares the configuration properties in the AMQ 6 KahaDB journal to the equivalent properties in the AMQ Broker 7 in-memory message journal.
For complete details on each message persistence configuration property for the in-memory message journal, see the following:
- The Bindings Journal in Configuring AMQ Broker
- Messaging Journal Configuration Elements in Configuring AMQ Broker
8.3.1. Journal Size and Management
The following table compares the journal size and management configuration properties in AMQ 6 to the equivalent properties in AMQ Broker 7:
| To set… | In AMQ 6 | In AMQ Broker 7 |
|---|---|---|
| The time interval between cleaning up data logs that are no longer used |
The default is 30000 ms. | No equivalent. In AMQ Broker 7, journal files that exceed the pool size are no longer used. |
| The number of message store GC cycles that must be completed without cleaning up other files before compaction is triggered |
| No equivalent. In AMQ Broker 7, compaction is not related to particular record types. |
| Whether compaction should be run when the message store is still growing, or if it should only occur when it has stopped growing |
The default is | No equivalent. |
| The minimum number of journal files that can be stored on the broker before it will compact them | No equivalent. |
The default is 10. If you set this value to 0, compaction will be deactivated. |
| The threshold to reach before compaction starts | No equivalent. |
The default is 30%. When less than this percentage is considered to be live data, compaction will start. |
| The path to the top-level folder that holds the message store’s data files |
| AMQ Broker 7 has a separate directory for each type of journal:
|
| Whether the bindings directory should be created automatically if it does not already exist | No equivalent. |
The default is |
| Whether the journal directory should be created automatically if it does not already exist | No equivalent. |
The default is |
| Whether the message store should periodically compact older journal log files that contain only message acknowledgements |
| No equivalent. |
| The maximum size of the data log files |
The default is 32 MB. |
The default is 10485760 bytes (10 MiB). |
| The policy that the broker should use to preallocate the journal files when a new journal file is needed |
The default is | No equivalent. By default, preallocated journal files are typically filled with zeroes, but it can vary depending on the file system. |
| The policy the broker should use to preallocate the journal files |
The default is |
AMQ Broker 7 automatically preallocates the journal files specified by |
| The journal type (either NIO or AIO) | No equivalent. |
You can choose either |
| The minimum number of files that the journal should maintain | No equivalent. |
|
| The number of journal files the broker should keep when reclaiming files | No equivalent. |
The default is -1, which means the broker instance will never delete files on the journal once created. |
8.3.2. Write Boundaries
The following table compares the write boundary configuration properties in AMQ 6 to the equivalent properties in AMQ Broker 7:
| To set… | In AMQ 6 | In AMQ Broker 7 |
|---|---|---|
| The time interval between writing the metadata cache to disk |
The default is 5000 ms. | No equivalent. |
| Whether the message store should dispatch queue messages to clients concurrently with message storage |
The default is | No equivalent. |
| Whether the message store should dispatch topic messages to interested clients concurrently with message storage |
The default is | No equivalent. |
| Whether a disk sync should be performed after each non-transactional journal write |
The default is |
|
| When to flush the entire journal buffer | No equivalent. |
The default for NIO is 3,333,333 nanoseconds, and the default for AIO is 500,000 nanoseconds. |
| The amount of data to buffer between journal disk writes |
The default is 4000 bytes. | No equivalent. |
| The size of the task queue used to buffer the journal’s write requests |
The default is 10000. |
This property controls the maximum number of write requests that can be in the I/O queue at any given point. The default for NIO is 1, and the default for AIO is 500. |
|
Whether to use | No equivalent. |
The default is |
8.3.3. Index Configuration
AMQ 6 has a number of configuration properties for configuring the journal index. Because AMQ Broker 7 does not use journal indexes, you do not need to configure any of these properties for your broker instance.
8.3.4. Journal Archival
AMQ 6 has several configuration properties for controlling which files are archived and where the archives are stored. In AMQ Broker 7, however, when old journal files are no longer needed, the broker reuses them instead of archiving them. Therefore, you do not need to configure any journal archival properties for your broker instance.
8.3.5. Journal Recovery
AMQ 6 has several configuration properties for controlling how the broker checks for corrupted journal files and what to do when it encounters a missing journal file.
In AMQ Broker 7, however, you do not need to configure any journal recovery properties for your broker instance. Journal files have a different format in AMQ Broker 7, which should prevent a corrupted entry in the journal from corrupting the entire journal file. Even if the journal is partially damaged, the broker should still be able to extract data from the undamaged entries.
