Red Hat Summit18.15. HornetQ Clustering
HornetQ clusters are used to create groups of HornetQ servers in order to share message processing load. Each active node in the cluster acts as an independent HornetQ server and manages its own messages and connections.
To form a cluster, each node (independent HornetQ server) declares cluster connections with another node with configuration parameters in server configuration files (
standalone.xml and domain.xml).
In clustering, core bridges are used for bridging/routing messages from one cluster to another. Core bridges consume messages from a source queue and then forward these messages to a target HornetQ server (node) which may or may not be in the same cluster.
When a node forms a cluster connection with another node, it creates a core bridge internally. Each node creates an explicit core bridge and you do not need to declare it. These cluster connections allow transmission of messages between nodes in various clusters for balancing message processing load.
You can configure cluster nodes in server configuration files (
standalone.xml and domain.xml).
Important
You can configure a node through server configuration files (
standalone.xml and domain.xml) and copy this configuration to other nodes to generate a symmetric cluster. However you must be careful when you are copying the server configuration files. You must not copy the HornetQ data (i.e. the bindings, journal, and large messages directories) from one node to another. When a node is started for the first time it persists a unique identifier to the journal directory which is needed for proper formation of clusters.
18.15.1. About Server Discovery
Servers use a mechanism called "server discovery" to:
- Forward their connection details to messaging clients: Messaging clients intend to connect to servers of a cluster without specific details on the servers which are up and running at a given point of time
- Connect to other servers: Servers in a cluster want to establish cluster connections with other servers without specific details on of all other servers in a cluster
Information about servers is sent to messaging clients via normal HornetQ connections and to other servers via cluster connections.
The initial first connection needs to be established and it can be established using dynamic server discovery techniques like UDP (User Datagram Protocol), JGroups or by providing a list of connectors.
18.15.2. Broadcast Groups
Connectors are used on the client to define how and in what ways it connects to the server. Servers use broadcast groups to broadcast connectors over the network. The broadcast group takes a set of connector pairs and broadcasts them on the network. Each connector pair contains connection settings for a live and backup server.
You can define broadcast groups in
broadcast-groups element of server configuration files (standalone.xml and domain.xml). A single HornetQ server can have many broadcast groups. You can define either a User Datagram Protocol (UDP) or a JGroup broadcast group.
18.15.2.1. User Datagram Protocol (UDP) Broadcast Group
The example shown below defines a UDP broadcast group:
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>172.16.9.3</local-bind-address>
<local-bind-port>5432</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty</connector-ref>
</broadcast-group>
</broadcast-groups>
Note
In the configuration example shown above, the attributes "local-bind-address", "local-bind-port", "group-address" and "group-port" are deprecated. Instead of these attributes you can choose to use the attribute "socket-binding".
The example shown below defines a UDP broadcast group replacing all the deprecated attributes with the attribute "socket-binding":
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<socket-binding>messaging-group</socket-binding>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty</connector-ref>
</broadcast-group>
</broadcast-groups>
The table shown below describes all the important parameters used in the above examples and in general to define a UDP broadcast group:
| Attribute | Description |
|---|---|
| name attribute |
Denotes the name of each broadcast group in a server. Each broadcast group must have a unique name.
|
| local-bind-address |
[Deprecated] This is a UDP specific attribute and specifies the local bind address which the datagram packet binds to. You must set this property to define the interface which you wish to use for your broadcasts. If this property is not specified then the socket binds to a wildcard address (a random kernel generated address).
|
| local-bind-port |
[Deprecated] This is a UDP specific attribute and is used to specify a local port which the datagram socket binds to. A default value of "-1" specifies an anonymous port to be used.
|
| group-address |
[Deprecated] This is a multicast address specific to UDP where messages are broadcast. This IP address has a range of 224.0.0.0 to 239.255.255.255, inclusive. The IP address 224.0.0 is reserved and can not be used.
|
| group-port |
[Deprecated] This denotes the UDP port number for broadcasting.
|
| socket-binding |
This denotes the broadcast group socket binding
|
| broadcast-period |
This parameter specifies the time between two broadcasts (milliseconds). It is optional.
|
| connector-ref |
This refers to the connector which will be broadcasted.
|
18.15.2.2. JGroups Broadcast Group
You can use JGroups to broadcast by specifying two attributes namely
jgroups-stack and jgroups-channel. The example shown below defines a JGroups broadcast group:
<broadcast-groups>
<broadcast-group name="bg-group1">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>udp</jgroups-channel>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty</connector-ref>
</broadcast-group>
</broadcast-groups>
The JGroups broadcast group definition uses two main attributes:
jgroups-stackattribute: This denotes the name of a stack defined in theorg.jboss.as.clustering.jgroupssubsystemjgroups-channelattribute: This denotes the channel which JGroups channels connect to for broadcasting
18.15.3. Discovery Groups
Broadcast groups are used for broadcasting connectors over a network. On the other hand discovery groups define how connector information is received from broadcast endpoints (UDP or JGroups broadcast group). A discovery group maintains a list of connector pair- one for each broadcast by a different server.
When a discovery group receives broadcasts on a broadcast endpoint for a specific server, it accordingly updates the connector pairs entry in the list for the specific server. If it does not receive a broadcast from a specific server for a long time, it removes the server's entry from the list altogether.
Discovery groups are mainly used by cluster connections and Java Messaging Service (JMS) clients to obtain initial connection information in order to download the required topology.
Note
You must configure each discovery group with an appropriate broadcast endpoint which matches its broadcast group counterpart (UDP or JGroups).
18.15.3.1. Configuring User Datagram Protocol (UDP) Discovery Group on the Server
The example shown below defines a UDP discovery group:
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>172.16.9.7</local-bind-address>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
Note
In the configuration example shown above, the attributes "local-bind-address", "group-address" and "group-port" are deprecated. Instead of these attributes you can choose to use the attribute "socket-binding".
The example shown below defines a UDP discovery group replacing all the deprecated attributes with the attribute "socket-binding":
<discovery-groups>
<discovery-group name="my-discovery-group">
<socket-binding>messaging-group</socket-binding>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
The table shown below describes all the important parameters used in the above example and in general to define a discovery group:
| Attribute | Description |
|---|---|
| name attribute |
This attribute denotes the name of your discovery group. Each discovery name must have a unique name per server.
|
| local-bind-address |
[Deprecated] This is an optional UDP specific attribute. It is used to configure a discovery group to listen on a specific interface when using multiple interfaces on the same machine.
|
| group-address |
[Deprecated] This is a compulsory UDP specific attribute. It is used to configure a discovery group to listen on the multicast IP address of a group. The value of this attribute must match the
group-address attribute of the broadcast group that you wish to listen from.
|
| group-port |
[Deprecated] This is a compulsory UDP specific attribute. It is used to configure the UDP port of the multicast group. The value of this attribute must match the
group-port attribute of the multicast group that you wish to listen from.
|
| socket-binding |
This denotes the discovery group socket binding
|
| refresh-timeout |
This is an optional UDP specific attribute. It is used to configure the time period (in milliseconds) for which the discovery group waits before removing a server's connector pair entry from the list after receiving the last broadcast from that server. The value of
refresh-timeout must be set significantly higher than the value of broadcast-period attribute on the broadcast group to prevent quick removal servers from the list when the broadcast process is still on. The default value of this attribute is 10,000 milliseconds.
|
18.15.3.2. Configuring JGroups Discovery Group on the Server
The example shown below defines a JGroups discovery group:
<discovery-groups>
<discovery-group name="dg-group1">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>udp</jgroups-channel>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
The JGroups discovery group definition uses two main attributes:
jgroups-stackattribute: This denotes the name of a stack defined in theorg.jboss.as.clustering.jgroupssubsystemjgroups-channelattribute: This attribute denotes the channel which JGroups channels connect to for receiving broadcasts
Note
JGroup attributes and UDP specific attributes are exclusive. You can use either JGroup or UDP set of attributes in the configuration of a discovery group or a broadcast group
18.15.3.3. Configuring Discovery Groups for Java Messaging Service (JMS) Clients
Discovery groups can be configured for JMS and core clients. You can specify the discovery group to be used for a JMS connection factory in server configuration files (
standalone.xml and domain.xml):
<connection-factory name="ConnectionFactory">
<discovery-group-ref discovery-group-name="my-discovery-group"/>
<entries>
<entry name="ConnectionFactory"/>
</entries>
</connection-factory>
The element discovery-group-ref is used to specify the name of a discovery group. When a client application downloads this connection factory from Java Naming and Directory Interface (JNDI) and creates JMS connections, these connections are load balanced across all the servers which the discovery group maintains by listening on the multicast address specified in the discovery group configuration.
If you are using JMS but not JNDI to lookup for a connection factory then you can specify the discovery group parameters directly when creating the JMS connection factory:
final String groupAddress = "231.7.7.7"; final int groupPort = 9876; ConnectionFactory jmsConnectionFactory = HornetQJMSClient.createConnectionFactory(new DiscoveryGroupConfiguration(groupAddress, groupPort, new UDPBroadcastGroupConfiguration(groupAddress, groupPort, null, -1)), JMSFactoryType.CF); Connection jmsConnection1 = jmsConnectionFactory.createConnection(); Connection jmsConnection2 = jmsConnectionFactory.createConnection();
The default value of
refresh-timeout attribute can be set on DiscoveryGroupConfiguration by using the setter method setDiscoveryRefreshTimeout(). For the connection factory to wait for a specific amount of time before creating the first connection, you can use the setter method setDiscoveryInitialWaitTimeout() on DiscoveryGroupConfiguration.
Doing this ensures that the connection factory has enough time to receive broadcasts from all the nodes in the cluster. The default value for this parameter is 10000 milliseconds.
18.15.3.4. Configuring discovery for Core API
If you are using the core API to directly instantiate
ClientSessionFactory instances, then you can specify the discovery group parameters directly when creating the session factory:
final String groupAddress = "231.7.7.7"; final int groupPort = 9876; ServerLocator factory = HornetQClient.createServerLocatorWithHA(new DiscoveryGroupConfiguration(groupAddress, groupPort, new UDPBroadcastGroupConfiguration(groupAddress, groupPort, null, -1)))); ClientSessionFactory factory = locator.createSessionFactory(); ClientSession session1 = factory.createSession(); ClientSession session2 = factory.createSession();The default value of
refresh-timeout attribute can be set on DiscoveryGroupConfiguration by using the setter method setDiscoveryRefreshTimeout(). You can use setDiscoveryInitialWaitTimeout() on DiscoveryGroupConfiguration for the session factory to wait for a specific amount of time before creating a session.
18.15.4. Server Side Load Balancing
There is one important cluster topology:
- Symmetric Cluster: In a symmetric cluster every cluster node is connected directly to every other node in the cluster. To create a symmetric cluster every node in the cluster defines a cluster connection with the attribute
max-hopsset to 1.Note
In a symmetric cluster each node knows about all the queues that exist on all the other nodes and what consumers they have. With this knowledge it can determine how to load balance and redistribute messages around the nodes.
18.15.4.1. Configuring Cluster Connections
Cluster connections are configured in server configuration files (
standalone.xml and domain.xml) in the element cluster-connection. There can be zero or more cluster connections defined per HornetQ server.
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>netty-connector</connector-ref>
<check-period>1000</check-period>
<connection-ttl>5000</connection-ttl>
<min-large-message-size>50000</min-large-message-size>
<call-timeout>5000</call-timeout>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.0</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<reconnect-attempts>-1</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<max-hops>1</max-hops>
<confirmation-window-size>32000</confirmation-window-size>
<call-failover-timeout>30000</call-failover-timeout>
<notification-interval>1000</notification-interval>
<notification-attempts>2</notification-attempts>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
The following table defines the configurable attributes:
| Attribute | Description | Default |
|---|---|---|
address | Each cluster connection only applies to messages sent to an address that starts with this value. The address can be any value and you can have many cluster connections with different values of addresses, simultaneously balancing messages for those addresses, potentially to different clusters of servers. This does not use wild card matching. | |
connector-ref | This is a compulsory attribute which refers to the connector sent to other nodes in the cluster so that they have the correct cluster topology | |
check-period | This refers to the time period (in milliseconds) which is used to verify if a cluster connection has failed to receive pings from another server | 30,000 milliseconds |
connection-ttl | This specifies how long a cluster connection must stay alive if it stops receiving messages from a specific node in the cluster | 60,000 milliseconds |
min-large-message-size | If the message size (in bytes) is larger than this value then it will be split into multiple segments when sent over the network to other cluster members | 102400 bytes |
call-timeout | This specifies the time period (milliseconds) for which a packet sent over a cluster connection waits (for a reply) before throwing an exception | 30,000 milliseconds |
retry-interval | If the cluster connection is created between nodes of a cluster and the target node has not been started, or is being rebooted, then the cluster connections from other nodes will retry connecting to the target until it comes back up. The parameter retry-interval defines the interval (milliseconds) between retry attempts | 500 milliseconds |
retry-interval-multiplier | This is used to increment the retry-interval after each retry attempt | 1 |
max-retry-interval | This refers to the maximum delay (in milliseconds) for retries | 2000 milliseconds |
reconnect-attempts | This defines the number of times the system will try to connect a node on the cluster | -1 (infinite retries) |
use-duplicate-detection | Cluster connections use bridges to link the nodes, and bridges can be configured to add a duplicate id property in each message that is forwarded. If the target node of the bridge crashes and then recovers, messages might be resent from the source node. By enabling duplicate detection any duplicate messages will be filtered out and ignored on receipt at the target node. | True |
forward-when-no-consumers | This parameter determines whether or not messages will be distributed in a round robin fashion between other nodes of the cluster regardless of whether there are matching or indeed any consumers on other nodes | False |
max-hops | This determines how messages are load balanced to other HornetQ serves which are connected to this server | -1 |
confirmation-window-size | The size (in bytes) of the window used for sending confirmations from the server connected to | 1048576 |
call-failover-timeout | This is used when a call is made during a failover attempt | -1 (no timeout) |
notification-interval | This determines how often (in milliseconds) the cluster connection must broadcast itself when attaching to the cluster | 1000 milliseconds |
notification-attempts | This defines as to how many times the cluster connection must broadcast itself when connecting to the cluster | 2 |
discovery-group-ref | This parameter determines which discovery group is used to obtain the list of other servers in the cluster which the current cluster connection will make connections to |
standalone.xml and domain.xml):
<cluster-user>HORNETQ.CLUSTER.ADMIN.USER</cluster-user> <cluster-password>NEW USER</cluster-password>
Warning
It is important to change the default values of these credentials to prevent remote clients from making connections to the server using the default values.
