Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 14. Setting up a broker cluster

A cluster consists of multiple broker instances that have been grouped together. Broker clusters enhance performance by distributing the message processing load across multiple brokers. In addition, broker clusters can minimize downtime through high availability.

You can connect brokers together in many different cluster topologies. Within the cluster, each active broker manages its own messages and handles its own connections.

You can also balance client connections across the cluster and redistribute messages to avoid broker starvation.

14.1. Understanding broker clusters
Copy link

Before creating a broker cluster, you should understand some important clustering concepts.

14.1.1. How broker clusters balance message load
Copy link

When brokers are connected to form a cluster, AMQ Broker automatically balances the message load between the brokers. This ensures that the cluster can maintain high message throughput.

Consider a symmetric cluster of four brokers. Each broker is configured with a queue named OrderQueue. The OrderProducer client connects to Broker1 and sends messages to OrderQueue. Broker1 forwards the messages to the other brokers in round-robin fashion. The OrderConsumer clients connected to each broker consume the messages.

Figure 14.1. Message load balancing in a cluster

Messages are load balanced across the brokers in the cluster
View larger image
Messages are load balanced across the brokers in the cluster

Without message load balancing, the messages sent to Broker1 would stay on Broker1 and only OrderConsumer1 would be able to consume them.

The AMQ Broker automatically load balances messages by default, distributing the first group of messages to the first broker and the second group of messages to the second broker. The order in which the brokers were started determines which broker is first, second and so on.

You can configure:

  • the cluster to load balance messages to brokers that have a matching queue.
  • the cluster to load balance messages to brokers that have a matching queue with active consumers.
  • the cluster to not load balance, but to perform redistribution of messages from queues that do not have any consumers to queues that do have consumers.
  • an address to automatically redistribute messages from queues that do not have any consumers to queues that do have consumers.

Additional resources

14.1.2. How broker clusters improve reliability
Copy link

Broker clusters make high availability and failover possible, which makes them more reliable than standalone brokers. By configuring high availability, you can ensure that client applications can continue to send and receive messages even if a broker encounters a failure event.

With high availability, the brokers in the cluster are grouped into primary-backup groups. A primary-backup group consists of an active broker that serves client requests, and one or more backup brokers that wait passively to replace the active broker if it fails. If a failure occurs, a passive broker replaces the active broker in its primary-backup group, and the clients reconnect and continue their work. For more information, see Section 15.1, “Configuring primary-backup groups for high availability”.

14.1.3. Cluster limitations
Copy link

The following limitation applies when you use AMQ broker in a clustered environment.

Temporary Queues
During a failover, if a client has consumers that use temporary queues, these queues are automatically recreated. The recreated queue name does not match the original queue name, which causes message redistribution to fail and can leave messages stranded in existing temporary queues. Red Hat recommends that you avoid using temporary queues in a cluster. For example, applications that use a request/reply pattern should use fixed queues for the JMSReplyTo address.

14.1.4. Understanding node IDs
Copy link

The broker node ID is a Globally Unique Identifier (GUID) generated programmatically when the journal for a broker instance is first created and initialized. The node ID is stored in the server.lock file. The node ID is used to uniquely identify a broker instance, regardless of whether the broker is a standalone instance, or part of a cluster. Primary-backup broker pairs share the same node ID, since they share the same journal.

In a broker cluster, broker instances (nodes) connect to each other and create bridges and internal "store-and-forward" queues. The names of these internal queues are based on the node IDs of the other broker instances. Broker instances also monitor cluster broadcasts for node IDs that match their own. A broker produces a warning message in the log if it identifies a duplicate ID.

When you are using the replication high availability (HA) policy, a primary broker that starts and has check-for-active-server set to true searches for a broker that is using its node ID. If the primary broker finds another broker using the same node ID, it either does not start, or initiates failback, based on the HA configuration.

The node ID is durable, meaning that it survives restarts of the broker. However, if you delete a broker instance (including its journal), then the node ID is also permanently deleted.

Additional resources

14.1.5. Common broker cluster topologies
Copy link

You can connect brokers to form either a symmetric or chain cluster topology. The topology you implement depends on your environment and messaging requirements.

Symmetric clusters

In a symmetric cluster, every broker is connected to every other broker. This means that every broker is no more than one hop away from every other broker.

Figure 14.2. Symmetric cluster topology

In a four-broker symmetric cluster each broker is connected to every other broker
View larger image
In a four-broker symmetric cluster each broker is connected to every other broker

Each broker in a symmetric cluster is aware of all of the queues that exist on every other broker in the cluster and the consumers that are listening on those queues. Therefore, symmetric clusters are able to load balance and redistribute messages more optimally than a chain cluster.

Symmetric clusters are easier to set up than chain clusters, but they can be difficult to use in environments in which network restrictions prevent brokers from being directly connected.

Chain clusters

In a chain cluster, each broker in the cluster is not connected to every broker in the cluster directly. Instead, the brokers form a chain with a broker on each end of the chain and all other brokers just connecting to the previous and next brokers in the chain.

Figure 14.3. Chain cluster topology

In a four-broker chain cluster the brokers are connected in a chain
View larger image
In a four-broker chain cluster the brokers are connected in a chain

Chain clusters are more difficult to set up than symmetric clusters, but can be useful when brokers are on separate networks and cannot be directly connected. By using a chain cluster, an intermediary broker can indirectly connect two brokers to enable messages to flow between them even though the two brokers are not directly connected.

14.1.6. Broker discovery methods
Copy link

Discovery is the mechanism by which brokers in a cluster propagate their connection details to each other. AMQ Broker supports both dynamic discovery and static discovery.

Dynamic discovery

Each broker in the cluster broadcasts its connection settings to the other members through either UDP multicast or JGroups. In this method, each broker uses:

  • A broadcast group to push information about its cluster connection to other potential members of the cluster.
  • A discovery group to receive and store cluster connection information about the other brokers in the cluster.
Static discovery

If you are not able to use UDP or JGroups in your network, or if you want to manually specify each member of the cluster, you can use static discovery. In this method, a broker "joins" the cluster by connecting to a second broker and sending its connection details. The second broker then propagates those details to the other brokers in the cluster.

14.1.7. Cluster sizing considerations
Copy link

Before creating a broker cluster, consider your messaging throughput, topology, and high availability requirements. These factors affect the number of brokers to include in the cluster.

Note

After creating the cluster, you can adjust the size by adding and removing brokers. You can add and remove brokers without losing any messages.

Messaging throughput

The cluster should contain enough brokers to provide the messaging throughput that you require. The more brokers in the cluster, the greater the throughput. However, large clusters can be complex to manage.

Topology

You can create either symmetric clusters or chain clusters. The type of topology you choose affects the number of brokers you may need.

For more information, see Section 14.1.5, “Common broker cluster topologies”.

High availability

If you require high availability (HA), consider choosing an HA policy before creating the cluster. The HA policy affects the size of the cluster, because each primary broker should have at least one backup broker.

For more information, see Chapter 15, Implementing high availability.

14.2. Creating a broker cluster
Copy link

You create a broker cluster by configuring a cluster connection on each broker that should participate in the cluster. The cluster connection defines how the broker should connect to the other brokers.

You can create a broker cluster that uses static discovery or dynamic discovery (either UDP multicast or JGroups).

Prerequisites

You can create a broker cluster by specifying a static list of brokers. Use this static discovery method if you are unable to use UDP multicast or JGroups on your network.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file.

  2. Within the <core> element, add the following connectors:

    • A connector that defines how other brokers can connect to this one
    • One or more connectors that define how this broker can connect to other brokers in the cluster 
    <configuration>
    <core>
        ...
        <connectors>
            <connector name="netty-connector">tcp://localhost:61617</connector>
    1 
    
            <connector name="broker2">tcp://localhost:61618</connector>
    2 
    
            <connector name="broker3">tcp://localhost:61619</connector>
        </connectors>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    1
    This connector defines connection information that other brokers can use to connect to this one. This information will be sent to other brokers in the cluster during discovery.
    2
    The broker2 and broker3 connectors define how this broker can connect to two other brokers in the cluster, one of which will always be available. If there are other brokers in the cluster, they will be discovered by one of these connectors when the initial connection is made.

    For more information about connectors, see Section 2.3, “About connectors”.

  3. Add a cluster connection and configure it to use static discovery.

    By default, the cluster connection will load balance messages for all addresses in a symmetric topology. 

    <configuration>
    <core>
        ...
        <cluster-connections>
            <cluster-connection name="my-cluster">
                <connector-ref>netty-connector</connector-ref>
                <static-connectors>
                    <connector-ref>broker2-connector</connector-ref>
                    <connector-ref>broker3-connector</connector-ref>
                </static-connectors>
            </cluster-connection>
        </cluster-connections>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    cluster-connection
    Use the name attribute to specify the name of the cluster connection.
    connector-ref
    The connector that defines how other brokers can connect to this one.
    static-connectors
    One or more connectors that this broker can use to make an initial connection to another broker in the cluster. After making this initial connection, the broker will discover the other brokers in the cluster. You only need to configure this property if the cluster uses static discovery.

  4. Configure any additional properties for the cluster connection.

    These additional cluster connection properties have default values that are suitable for most common use cases. Therefore, you only need to configure these properties if you do not want the default behavior. For more information, see Appendix C, Cluster Connection Configuration Elements.

  5. Create the cluster user and password.

    AMQ Broker ships with default cluster credentials, but you should change them to prevent unauthorized remote clients from using these default credentials to connect to the broker.

    Important

    The cluster password must be the same on every broker in the cluster. 

    <configuration>
    <core>
        ...
        <cluster-user>cluster_user</cluster-user>
        <cluster-password>cluster_user_password</cluster-password>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  6. Repeat this procedure on each additional broker.

    You can copy the cluster configuration to each additional broker. However, do not copy any of the other AMQ Broker data files (such as the bindings, journal, and large messages directories). These files must be unique among the nodes in the cluster or the cluster will not form properly.

Additional resources

You can create a broker cluster in which the brokers discover each other dynamically through UDP multicast.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file.

  2. Within the <core> element, add a connector.

    This connector defines connection information that other brokers can use to connect to this one. This information will be sent to other brokers in the cluster during discovery. 

    <configuration>
    <core>
        ...
        <connectors>
            <connector name="netty-connector">tcp://localhost:61617</connector>
        </connectors>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  3. Add a UDP broadcast group.

    The broadcast group enables the broker to push information about its cluster connection to the other brokers in the cluster. This broadcast group uses UDP to broadcast the connection settings: 

    <configuration>
    <core>
        ...
        <broadcast-groups>
            <broadcast-group name="my-broadcast-group">
                <local-bind-address>172.16.9.3</local-bind-address>
                <local-bind-port>-1</local-bind-port>
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <broadcast-period>2000</broadcast-period>
                <connector-ref>netty-connector</connector-ref>
            </broadcast-group>
        </broadcast-groups>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

    The following parameters are required unless otherwise noted:

    broadcast-group
    Use the name attribute to specify a unique name for the broadcast group.
    local-bind-address
    The address to which the UDP socket is bound. If you have multiple network interfaces on your broker, you should specify which one you want to use for broadcasts. If this property is not specified, the socket will be bound to an IP address chosen by the operating system. This is a UDP-specific attribute.
    local-bind-port
    The port to which the datagram socket is bound. In most cases, use the default value of -1, which specifies an anonymous port. This parameter is used in connection with local-bind-address. This is a UDP-specific attribute.
    group-address
    The multicast address to which the data will be broadcast. It is a class D IP address in the range 224.0.0.0 - 239.255.255.255 inclusive. The address 224.0.0.0 is reserved and is not available for use. This is a UDP-specific attribute.
    group-port
    The UDP port number used for broadcasting. This is a UDP-specific attribute.
    broadcast-period (optional)
    The interval in milliseconds between consecutive broadcasts. The default value is 2000 milliseconds.
    connector-ref
    The previously configured cluster connector that should be broadcasted.

  4. Add a UDP discovery group.

    The discovery group defines how this broker receives connector information from other brokers. The broker maintains a list of connectors (one entry for each broker). As it receives broadcasts from a broker, it updates its entry. If it does not receive a broadcast from a broker for a length of time, it removes the entry.

    This discovery group uses UDP to discover the brokers in the cluster: 

    <configuration>
    <core>
        ...
        <discovery-groups>
            <discovery-group name="my-discovery-group">
                <local-bind-address>172.16.9.7</local-bind-address>
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
        <discovery-groups>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

    The following parameters are required unless otherwise noted:

    discovery-group
    Use the name attribute to specify a unique name for the discovery group.
    local-bind-address (optional)
    If the machine on which the broker is running uses multiple network interfaces, you can specify the network interface to which the discovery group should listen. This is a UDP-specific attribute.
    group-address
    The multicast address of the group on which to listen. It should match the group-address in the broadcast group that you want to listen from. This is a UDP-specific attribute.
    group-port
    The UDP port number of the multicast group. It should match the group-port in the broadcast group that you want to listen from. This is a UDP-specific attribute.
    refresh-timeout (optional)

    The amount of time in milliseconds that the discovery group waits after receiving the last broadcast from a particular broker before removing that broker’s connector pair entry from its list. The default is 10000 milliseconds (10 seconds).

    Set this to a much higher value than the broadcast-period on the broadcast group. Otherwise, brokers might periodically disappear from the list even though they are still broadcasting (due to slight differences in timing).

  5. Create a cluster connection and configure it to use dynamic discovery.

    By default, the cluster connection will load balance messages for all addresses in a symmetric topology. 

    <configuration>
    <core>
        ...
        <cluster-connections>
            <cluster-connection name="my-cluster">
                <connector-ref>netty-connector</connector-ref>
                <discovery-group-ref discovery-group-name="my-discovery-group"/>
            </cluster-connection>
        </cluster-connections>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    cluster-connection
    Use the name attribute to specify the name of the cluster connection.
    connector-ref
    The connector that defines how other brokers can connect to this one.
    discovery-group-ref
    The discovery group that this broker should use to locate other members of the cluster. You only need to configure this property if the cluster uses dynamic discovery.

  6. Configure any additional properties for the cluster connection.

    These additional cluster connection properties have default values that are suitable for most common use cases. Therefore, you only need to configure these properties if you do not want the default behavior. For more information, see Appendix C, Cluster Connection Configuration Elements.

  7. Create the cluster user and password.

    AMQ Broker ships with default cluster credentials, but you should change them to prevent unauthorized remote clients from using these default credentials to connect to the broker.

    Important

    The cluster password must be the same on every broker in the cluster. 

    <configuration>
    <core>
        ...
        <cluster-user>cluster_user</cluster-user>
        <cluster-password>cluster_user_password</cluster-password>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  8. Repeat this procedure on each additional broker.

    You can copy the cluster configuration to each additional broker. However, do not copy any of the other AMQ Broker data files (such as the bindings, journal, and large messages directories). These files must be unique among the nodes in the cluster or the cluster will not form properly.

Additional resources

  • For an example of a broker cluster configuration that uses dynamic discovery with UDP, see the clustered-queue example.

If you are already using JGroups in your environment, you can use it to create a broker cluster in which the brokers discover each other dynamically.

Prerequisites

  • JGroups must be installed and configured.

    For an example of a JGroups configuration file, see the clustered-jgroups example.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file.

  2. Within the <core> element, add a connector.

    This connector defines connection information that other brokers can use to connect to this one. This information will be sent to other brokers in the cluster during discovery. 

    <configuration>
    <core>
        ...
        <connectors>
            <connector name="netty-connector">tcp://localhost:61617</connector>
        </connectors>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  3. Within the <core> element, add a JGroups broadcast group.

    The broadcast group enables the broker to push information about its cluster connection to the other brokers in the cluster. This broadcast group uses JGroups to broadcast the connection settings: 

    <configuration>
    <core>
        ...
        <broadcast-groups>
            <broadcast-group name="my-broadcast-group">
                <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
                <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
                <broadcast-period>2000</broadcast-period>
                <connector-ref>netty-connector</connector-ref>
            </broadcast-group>
        </broadcast-groups>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

    The following parameters are required unless otherwise noted:

    broadcast-group
    Use the name attribute to specify a unique name for the broadcast group.
    jgroups-file
    The name of JGroups configuration file to initialize JGroups channels. The file must be in the Java resource path so that the broker can load it.
    jgroups-channel
    The name of the JGroups channel to connect to for broadcasting.
    broadcast-period (optional)
    The interval, in milliseconds, between consecutive broadcasts. The default value is 2000 milliseconds.
    connector-ref
    The previously configured cluster connector that should be broadcasted.

  4. Add a JGroups discovery group.

    The discovery group defines how connector information is received. The broker maintains a list of connectors (one entry for each broker). As it receives broadcasts from a broker, it updates its entry. If it does not receive a broadcast from a broker for a length of time, it removes the entry.

    This discovery group uses JGroups to discover the brokers in the cluster: 

    <configuration>
    <core>
        ...
        <discovery-groups>
            <discovery-group name="my-discovery-group">
                <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>
                <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
                <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
        <discovery-groups>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

    The following parameters are required unless otherwise noted:

    discovery-group
    Use the name attribute to specify a unique name for the discovery group.
    jgroups-file
    The name of JGroups configuration file to initialize JGroups channels. The file must be in the Java resource path so that the broker can load it.
    jgroups-channel
    The name of the JGroups channel to connect to for receiving broadcasts.
    refresh-timeout (optional)

    The amount of time in milliseconds that the discovery group waits after receiving the last broadcast from a particular broker before removing that broker’s connector pair entry from its list. The default is 10000 milliseconds (10 seconds).

    Set this to a much higher value than the broadcast-period on the broadcast group. Otherwise, brokers might periodically disappear from the list even though they are still broadcasting (due to slight differences in timing).

  5. Create a cluster connection and configure it to use dynamic discovery.

    By default, the cluster connection will load balance messages for all addresses in a symmetric topology. 

    <configuration>
    <core>
        ...
        <cluster-connections>
            <cluster-connection name="my-cluster">
                <connector-ref>netty-connector</connector-ref>
                <discovery-group-ref discovery-group-name="my-discovery-group"/>
            </cluster-connection>
        </cluster-connections>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    cluster-connection
    Use the name attribute to specify the name of the cluster connection.
    connector-ref
    The connector that defines how other brokers can connect to this one.
    discovery-group-ref
    The discovery group that this broker should use to locate other members of the cluster. You only need to configure this property if the cluster uses dynamic discovery.

  6. Configure any additional properties for the cluster connection.

    These additional cluster connection properties have default values that are suitable for most common use cases. Therefore, you only need to configure these properties if you do not want the default behavior. For more information, see Appendix C, Cluster Connection Configuration Elements.

  7. Create the cluster user and password.

    AMQ Broker ships with default cluster credentials, but you should change them to prevent unauthorized remote clients from using these default credentials to connect to the broker.

    Important

    The cluster password must be the same on every broker in the cluster. 

    <configuration>
    <core>
        ...
        <cluster-user>cluster_user</cluster-user>
        <cluster-password>cluster_user_password</cluster-password>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  8. Repeat this procedure on each additional broker.

    You can copy the cluster configuration to each additional broker. However, do not copy any of the other AMQ Broker data files (such as the bindings, journal, and large messages directories). These files must be unique among the nodes in the cluster or the cluster will not form properly.

Additional resources

14.3. Enabling message redistribution
Copy link

If your broker cluster is configured with message-load-balancing set to ON_DEMAND or OFF_WITH_REDISTRIBUTION, you can configure message redistribution to prevent messages from being "stuck" in a queue that does not have a consumer to consume the messages.

This section contains information about:

14.3.1. Understanding message redistribution
Copy link

Broker clusters use load balancing to distribute the message load across the cluster. When configuring load balancing in the cluster connection, you can enable redistribution using the following message-load-balancing settings:

  • ON_DEMAND - enable load balancing and allow redistribution
  • OFF_WITH_REDISTRIBUTION - disable load balancing but allow redistribution

In both cases, the broker forwards messages only to other brokers that have matching consumers. This behavior ensures that messages are not moved to queues that do not have any consumers to consume the messages. However, if the consumers attached to a queue close after the messages are forwarded to the broker, those messages become "stuck" in the queue and are not consumed. This issue is sometimes called starvation.

Message redistribution prevents starvation by automatically redistributing the messages from queues that have no consumers to brokers in the cluster that do have matching consumers.

With OFF_WITH_REDISTRIBUTION, the broker only forwards messages to other brokers that have matching consumers if there are no active local consumers, enabling you to prioritize a broker while providing an alternative when consumers are not available.

Message redistribution supports the use of filters (also know as selectors), that is, messages are redistributed when they do not match the selectors of the available local consumers.

Additional resources

14.3.2. Configuring message redistribution
Copy link

This procedure shows how to configure message redistribution with load balancing. If you want message redistribution without load balancing, set <message-load-balancing> is set to OFF_WITH_REDISTRIBUTION.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file.

  2. In the <cluster-connection> element, verify that <message-load-balancing> is set to ON_DEMAND. 

    <configuration>
    <core>
        ...
        <cluster-connections>
            <cluster-connection name="my-cluster">
                ...
                <message-load-balancing>ON_DEMAND</message-load-balancing>
                ...
            </cluster-connection>
        </cluster-connections>
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap

  3. Within the <address-settings> element, set the redistribution delay for a queue or set of queues.

    In this example, messages load balanced to my.queue will be redistributed 5000 milliseconds after the last consumer closes. 

    <configuration>
    <core>
        ...
        <address-settings>
            <address-setting match="my.queue">
                <redistribution-delay>5000</redistribution-delay>
            </address-setting>
        </address-settings>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    address-setting
    Set the match attribute to be the name of the queue for which you want messages to be redistributed. You can use the broker wildcard syntax to specify a range of queues. For more information, see Section 4.2, “Applying address settings to sets of addresses”.
    redistribution-delay
    The amount of time (in milliseconds) that the broker should wait after this queue’s final consumer closes before redistributing messages to other brokers in the cluster. If you set this to 0, messages will be redistributed immediately. However, you should typically set a delay before redistributing - it is common for a consumer to close but another one to be quickly created on the same queue.
  4. Repeat this procedure for each additional broker in the cluster.

Additional resources

14.4. Configuring clustered message grouping
Copy link

Message grouping enables clients to send groups of messages of a particular type to be processed serially by the same consumer. By adding a grouping handler to each broker in the cluster, you ensure that clients can send grouped messages to any broker in the cluster and still have those messages consumed in the correct order by the same consumer.

Note

Grouping and clustering techniques can be summarized as follows:

  • Message grouping imposes an order on message consumption. In a group, each message must be fully consumed and acknowledged prior to proceeding with the next message. This methodology leads to serial message processing, where concurrency is not an option.
  • Clustering aims to horizontally scale brokers to boost message throughput. Horizontal scaling is achieved by adding additional consumers that can process messages concurrently.

Because these techniques contradict each other, avoid using clustering and grouping together.

There are two types of grouping handlers: local handlers and remote handlers. They enable the broker cluster to route all of the messages in a particular group to the appropriate queue so that the intended consumer can consume them in the correct order.

Prerequisites

  • There should be at least one consumer on each broker in the cluster.

    When a message is pinned to a consumer on a queue, all messages with the same group ID will be routed to that queue. If the consumer is removed, the queue will continue to receive the messages even if there are no consumers.

Procedure

  1. Configure a local handler on one broker in the cluster.

    If you are using high availability, this should be a primary broker.

    1. Open the broker’s <broker_instance_dir>/etc/broker.xml configuration file.

    2. Within the <core> element, add a local handler:

      The local handler serves as an arbiter for the remote handlers. It stores route information and communicates it to the other brokers. 

      <configuration>
    <core>
        ...
        <grouping-handler name="my-grouping-handler">
            <type>LOCAL</type>
            <timeout>10000</timeout>
        </grouping-handler>
        ...
    </core>
</configuration>
      Copy to Clipboard Toggle word wrap
      grouping-handler
      Use the name attribute to specify a unique name for the grouping handler.
      type
      Set this to LOCAL.
      timeout

      The amount of time to wait (in milliseconds) for a decision to be made about where to route the message. The default is 5000 milliseconds. If the timeout is reached before a routing decision is made, an exception is thrown, which ensures strict message ordering.

      When the broker receives a message with a group ID, it proposes a route to a queue to which the consumer is attached. If the route is accepted by the grouping handlers on the other brokers in the cluster, then the route is established: all brokers in the cluster will forward messages with this group ID to that queue. If the broker’s route proposal is rejected, then it proposes an alternate route, repeating the process until a route is accepted.

  2. If you are using high availability, copy the local handler configuration to the primary broker’s backup broker.

    Copying the local handler configuration to the backup broker prevents a single point of failure for the local handler.

  3. On each remaining broker in the cluster, configure a remote handler.

    1. Open the broker’s <broker_instance_dir>/etc/broker.xml configuration file.

    2. Within the <core> element, add a remote handler: 

      <configuration>
    <core>
        ...
        <grouping-handler name="my-grouping-handler">
            <type>REMOTE</type>
            <timeout>5000</timeout>
        </grouping-handler>
        ...
    </core>
</configuration>
      Copy to Clipboard Toggle word wrap
      grouping-handler
      Use the name attribute to specify a unique name for the grouping handler.
      type
      Set this to REMOTE.
      timeout
      The amount of time to wait (in milliseconds) for a decision to be made about where to route the message. The default is 5000 milliseconds. Set this value to at least half of the value of the local handler.

Additional resources

14.5. Connecting clients to a broker cluster
Copy link

You can use the Red Hat build of Apache Qpid JMS clients to connect to the cluster. By using JMS, you can configure your messaging clients to discover the list of brokers dynamically or statically. You can also configure client-side load balancing to distribute the client sessions created from the connection across the cluster.

Procedure

14.6. Partitioning client connections
Copy link

Partitioning client connections involves routing connections for individual clients to the same broker each time the client initiates a connection.

Two use cases for partitioning client connections are:

  • Partitioning clients of durable subscriptions to ensure that a subscriber always connects to the broker where the durable subscriber queue is located.
  • Minimizing the need to move data by attracting clients to data where it originates, also known as data gravity.

Durable subscriptions

A durable subscription is represented as a queue on a broker and is created when a durable subscriber first connects to the broker. This queue remains on the broker and receives messages until the client unsubscribes. Therefore, you want the client to connect to the same broker repeatedly to consume the messages that are in the subscriber queue.

To partition clients for durable subscription queues, you can filter the client ID in client connections.

Data gravity

If you scale up the number of brokers in your environment without considering data gravity, some of the performance benefits are lost because of the need to move messages between brokers. To support date gravity, you should partition your client connections so that client consumers connect to the broker on which the messages that they need to consume are produced.

To partition client connections to support data gravity, you can filter any of the following attributes of a client connection:

  • a role assigned to the connecting user (ROLE_NAME)
  • the username of the user (USER_NAME)
  • the hostname of the client (SNI_HOST)
  • the IP address of the client (SOURCE_IP)

To partition clients for durable subscriptions, you can filter client IDs in incoming connections by using a consistent hashing algorithm or a regular expression.

Prerequisites

Clients are configured so they can connect to all of the brokers in the cluster, for example, by using a load balancer or by having all of the broker instances configured in the connection URL. If a broker rejects a connection because the client details do not match the partition configuration for that broker, the client must be able to connect to the other brokers in the cluster to find a broker that accepts connections from it.

You can configure each broker in a cluster to use a consistent hashing algorithm to hash the client ID in each client connection. After the broker hashes the client ID, it performs a modulo operation on the hashed value to return an integer value, which identifies the target broker for the client connection. The broker compares the integer value returned to a unique value configured on the broker. If there is a match, the broker accepts the connection. If the values don’t match, the broker rejects the connection. This process is repeated on each broker in the cluster until a match is found and a broker accepts the connection.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file for the first broker.

  2. Create a connection-routers element and create a connection-route to filter client IDs by using a consistent hashing algorithm. For example: 

    <configuration>
   <core>
      ...
      <connection-routers>
         <connection-route name=”consistent-hash-routing”>
            <key>CLIENT_ID</target-key>
            <local-target-filter>NULL|0</local-target-filter>
            <policy name="CONSISTENT_HASH_MODULO">
               <property key="modulo" value="<number_of_brokers_in_cluster>">
               </property>
            </policy>
         </connection-route>
      </connection-routers>
      ...
   </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    connection-route
    For the connection-route name, specify an identifying string for this connection routing configuration. You must add this name to each broker acceptor that you want to apply the consistent hashing filter to.
    key
    The key type to apply the filter to. To filter the client ID, specify CLIENT_ID in the key field.
    local-target-filter
    The value that the broker compares to the integer value returned by the modulo operation to determine if there is a match and the broker can accept the connection. The value of NULL|0 in the example provides a match for connections that have no client ID (NULL) and connections where the number returned by the modulo operation is 0.
    policy

    Accepts a modulo property key, which performs a modulo operation on the hashed client ID to identify the target broker. The value of the modulo property key must equal the number of brokers in the cluster.

    Important

    The policy name must be CONSISTENT_HASH_MODULO.

  3. Open the <broker_instance_dir>/etc/broker.xml configuration file for the second broker.

  4. Create a connection-routers element and create a connection route to filter client IDs by using a consistent hashing algorithm.

    In the following example, the local-target-filter value of NULL|1 provides a match for connections that have no client ID (NULL) and connections where the value returned by the modulo operation is 1. 

    <configuration>
   <core>
      ...
      <connection-routers>
         <connection-route name=”consistent-hash-routing”>
            <key>CLIENT_ID</target-key>
            <local-target-filter>NULL|1</local-target-filter>
            <policy name="CONSISTENT_HASH_MODULO">
               <property key="modulo" value="<number_of_brokers_in_cluster>">
               </property>
            </policy>
         </connection-route>
      </connection-routers>
      ...
   </core>
</configuration>
    Copy to Clipboard Toggle word wrap
  5. Repeat this procedure to create a consistent hash filter for each additional broker in the cluster.

You can partition client connections by configuring brokers to apply a regular expression filter to a part of the client ID in client connections. A broker only accepts a connection if the result of the regular expression filter matches the local target filter configured for the broker. If a match is not found, the broker rejects the connection. This process is repeated on each broker in the cluster until a match is found and a broker accepts the connection.

Prerequisites

  • A common string in each client ID that can be filtered by a regular expression.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file for the first broker.

  2. Create a connection-routers element and create a connection-route to filter part of the client ID. For example: 

    <configuration>
    <core>
        ...
        <connection-routers>
            <connection-route name=”regex-routing”>
                <key>CLIENT_ID</target-key>
                <key-filter>^.{3}</key-filter>
                <local-target-filter>NULL|CL1</local-target-filter>
            </connection-route>
    </connection-routers>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    connection-route
    For the connection-route name, specify an identifying string for this routing configuration. You must add this name to each broker acceptor that you want to apply the regular expression filter to.
    key
    The key to apply the filter to. To filter the client ID, specify CLIENT_ID in the key field.
    key-filter

    The part of the client ID string to which the regular expression is applied to extract a key value. In the example for the first broker above, the broker extracts a key value that is the first 3 characters of the client ID. If, for example, the client ID string is CL100.consumer, the broker extracts a key value of CL1. After the broker extracts the key value, it compares it to the value of the local-target-filter.

    If an incoming connection does not have a client ID, or if the broker is unable to extract a key value by using the regular expression specified for the key-filter, the key value is set to NULL.

    local-target-filter
    The value that the broker compares to the key value to determine if there is a match and the broker can accept the connection. A value of NULL|CL1, as shown in the example for the first broker above, matches connections that have no client ID (NULL) or have a 3-character prefix of CL1 in the client ID.
  3. Open the <broker_instance_dir>/etc/broker.xml configuration file for the second broker.

  4. Create a connection-routers element and create a connection route to filter connections based on a part of the client ID.

    In the following filter example, the broker uses a regular expression to extract a key value that is the first 3 characters of the client ID. The broker compares the values of NULL and CL2 to the key value to determine if there is a match and the broker can accept the connection. 

    <configuration>
    <core>
        ...
        <connection-routers>
            <connection-route name=”regex-routing”>
                <key>CLIENT_ID</target-key>
                <key-filter>^.{3}</key-filter>
                <local-target-filter>NULL|CL2</local-target-filter>
            </connection-route>
        </connection-routers>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
  5. Repeat this procedure and create the appropriate connection routing filter for each additional broker in the cluster.

To support date gravity, you can partition client connections so that client consumers connect to the broker where the messages that they need to consume are produced. For example, if you have a set of addresses that are used by producer and consumer applications, you can configure the addresses on a specific broker. You can then partition client connections for both producers and consumers that use those addresses so they can only connect to that broker.

You can partition client connections based on attributes such as the role assigned to the connecting user, the username of the user, or the hostname or IP address of the client. This section shows an example of how to partition client connections by filtering user roles assigned to client users. If clients are required to authenticate to connect to brokers, you can assign roles to client users and filter connections so only users that match the role criteria can connect to a broker.

Prerequisites

  • Clients are configured so they can connect to all of the brokers in the cluster, for example, by using a load balancer or by having all of the broker instances configured in the connection URL. If a broker rejects a connection because the client does not match the partitioning filter criteria configured for that broker, the client must be able to connect to the other brokers in the cluster to find a broker that accepts connections from it.

Procedure

  1. Open the <broker_instance_dir>/etc/artemis-roles.properties file for the first broker. Add a broker1users role and add users to the role.
  2. Open the <broker_instance_dir>/etc/broker.xml configuration file for the first broker.

  3. Create a connection-routers element and create a connection-route to filter connections based on the roles assigned to users. For example: 

    <configuration>
    <core>
        ...
        <connection-routers>
            <connection-route name=”role-based-routing”>
                <key>ROLE_NAME</target-key>
                <key-filter>broker1users</key-filter>
                <local-target-filter>broker1users</local-target-filter>
            </connection-route>
        </connection-routers>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
    connection-route
    For the connection-route name, specify an identifying string for this routing configuration. You must add this name to each broker acceptor that you want to apply the role filter to.
    key
    The key to apply the filter to. To configure role-based filtering, specify ROLE_NAME in the key field.
    key-filter
    The string or regular expression that the broker uses to filter the user’s roles and extract a key value. If the broker finds a matching role, it sets the key value to that role. If it does not find a matching role, the broker sets the key value to NULL. In the above example, the broker applies a filter of broker1users to the client user’s roles. After the broker extracts the key value, it compares it to the value of the local-target-filter.
    local-target-filter
    The value that the broker compares to the key value to determine if there is a match and the broker can accept the connection. In the example, the broker compares a value of broker1users to the key value. It there is a match, which means that the user has a broker1users role, the broker accepts the connection.
  4. Repeat this procedure and specify the appropriate role in the filter to partition clients on other brokers in the cluster.

14.6.3. Adding connection routes to acceptors
Copy link

After you configure a connection route on a broker, you must add the route to one or more of the broker’s acceptors to partition client connections. After you add a connection route to an acceptor, the broker applies the filter configured in the connection route to connections received by the acceptor.

Procedure

  1. Open the <broker_instance_dir>/etc/broker.xml configuration file for the first broker.

  2. For each acceptor on which you want to enable partitioning, append the router key and specify the connection-route name. In the following example, a connection-route name of consistent-hash-routing is added to the artemis acceptor. 

    <configuration>
    <core>
        ...
        <acceptors>
        ...
        <!-- Acceptor for every supported protocol -->
        <acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;router="consistent-hash-routing" </acceptor>
      </acceptors>
        ...
    </core>
</configuration>
    Copy to Clipboard Toggle word wrap
  3. Repeat this procedure to specify the appropriate connection route filter for each broker in the cluster.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat