Chapter 5. Using the Replicated LevelDB Persistence Adapter
Abstract
The Replicated LevelDB persistence adapter uses Apache ZooKeeper to select a master from a cluster of broker nodes that are configured to replicate a levelDB store. The Replicated LevelDB persistence adapter then synchronizes all slave LevelDB stores with the master LevelDB store by replicating the master broker's updates to all slave brokers in the cluster.
Because the Replicated LevelDB store uses the same data files as the regular LevelDB store, you can switch a broker's LevelDB configuration between replicated and regular at any time.
Warning
The Replicated LevelDB persistence adapter is a technical preview only, and is not supported in JBoss A-MQ 6.1.
Overview
Important
The Replicated LevelDB persistence adapter is provided for technical preview only. It is not ready for use in the production environment.
The Replicated LevelDB message store uses the same file-based store implemented using Google's LevelDB library. As such, it provides the same advantages and runs on the same platforms as the LevelDB persistence adapter (for details, see Using the LevelDB Persistence Adapter).
The Replicated LevelDB store uses Apache ZooKeeper to coordinate and select which broker node in the cluster becomes master. Only the master accepts and starts client connections. All other broker nodes enter slave mode and connect to the master, synchronizing their persistence state with it. The master node then replicates all persistent operations to the connected slaves.
When the master dies, Apache ZooKeeper selects a slave that has the latest updates to be the new master. Once the new master is activated, the old master can be brought back online, at which time it enters slave mode.
All messaging operations that require a sync-to-disk wait for the update to be replicated to a quorum of slave nodes before the operations complete. For example, a store configured with
replicas="3" has a quorum size of (3/2)+1=2. In this case, the master stores the update locally, then waits for at least one slave to store the update before it reports success.
To select a new master, a quorum of nodes must be online for ZooKeeper to find a slave with the latest updates. Therefore, it's recommend that you run with at least three replica nodes, so you can take one down without suffering a service outage.
Deployment tips
- Clients should use the Failover Transport to connect to the broker nodes in the replication cluster; for example, using a URL like this:
failover:(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)
- To enable highly available ZooKeeper service, run at least three ZooKeeper server nodes.NoteAvoid overcommitting ZooKeeper servers. An overworked ZooKeeper server might infer that a live broker replication node has gone offline due to delayed processing of keep-alive messages.For details on setting up and running a distributed cluster of Apache ZooKeeper servers, see the ZooKeeper Getting Started document.
- To enable highly available ActiveMQ service, run at least three replicated broker nodes.
Note
Though Example 5.1, “Configuring the Replicated LevelDB Message Store” configures three replicated broker nodes and three ZooKeeper servers, having the same number of ZooKeeper nodes as replicated broker nodes is not required. Both the ZooKeeper service and the messaging service operate under the same outage probability formula, wherein running three nodes allows one node to fail without incurring a service outage, running five nodes allows two nodes to fail simultaneously without incurring a service outage, and so on. Applications that must meet stringent high availability requirements might configure more ZooKeeper nodes than replicated broker nodes, as the messaging service depends on the ZooKeeper service and is limited by its availability.
Basic configuration
To configure the Replicated LevelDB message store, place a
replicatedLevelDB element in the persistenceAdapter element of your broker's configuration, and use the replicatedLevelDB element's attributes to configure the message store.
Important
All broker nodes in the same replication cluster must use the same value in the
brokerName attribute.
Example 5.1, “Configuring the Replicated LevelDB Message Store” shows a basic configuration of the Replicated LevelDB message store. The Replicated LevelDB files are stored under the
activemq-data directory.
Example 5.1. Configuring the Replicated LevelDB Message Store
<broker brokerName="broker" persistent="true" ... >
...
<persistenceAdapter>
<replicatedLevelDB
directory="activemq-data" />
replicas="3"
bind="tcp://0.0.0.0;0"
zkAddress="zoo1.example.org:2181,zoo2.example.org:2181,zoo3.example.org:2181"
zkPassword="password"
zkPath="/activemq/leveldb-stores"
/>
</persistenceAdapter>
...
</broker>Configuration attributes
The attributes listed in Table 5.1, “Configuration Properties of the Replicated LevelDB Message Store—attributes which must be identical on all broker nodes in a replication cluster” must be configured with the same value on all broker nodes in a replication cluster.
Important
The broker uses
zkSessionTimeout to detect when it has been disconnected from the ZooKeeper server due to a network failure. When set to 2s, the replicated broker nodes will detect a disconnect within two seconds of a network failure. Once the disconnect is detected, the master broker gives up the master role, and the slave brokers begin the election process. The lower the timeout value, the faster the process to select a new master. However, setting the timeout value too low can result in false positives, causing masters to switch when no disconnect has occurred.
The attributes listed in Table 5.2, “Configuration Properties of the Replicated LevelDB Message Store—attributes which can be unique for each broker node in a replication cluster” can be unique per broker node in a replication cluster.
Note
Except for the Pluggable Storage Lockers, the Replicated LevelDB store supports all of the standard LevelDB store configuration attributes. For details, see Table 4.1, “Configuration Properties of the LevelDB Message Store—standard LevelDB attributes” in Using the LevelDB Persistence Adapter.
