SupportConsoleDevelopersStart a trialContact

Chapter 5. Routing messages through the router network

Routing is the process by which messages are delivered to their destinations. To accomplish this, AMQ Interconnect provides two routing mechanisms: message routing and link routing.

Message routing

Routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its settlement based on the message’s address and routing pattern.

Figure 5.1. Message Routing

Message Routing

In this diagram, the message producer attaches a link to the router, and then sends a message over the link. When the router receives the message, it identifies the message’s destination based on the message’s address, and then uses its routing table to determine the best route to deliver the message either to its destination or to the next hop in the route. All dispositions (including settlement) are propagated along the same path that the original message transfer took. Flow control is handled between the sender and the router, and then between the router and the receiver.

Link routing

Routing is performed on link-attach frames, which are chained together to form a virtual messaging path that directly connects a sender and receiver. Once a link route is established, the transfer of message deliveries, flow frames, and dispositions is performed across the link route.

Figure 5.2. Link Routing

Link Routing

In this diagram, a router is connected to clients and to a broker, and it provides a link route to a queue on the broker (my_queue). The sender connects to the router, and the router propagates the link-attaches to the broker to form a direct link between the sender and the broker. The sender can begin sending messages to the queue, and the router passes the deliveries along the link route directly to the broker queue.

5.2. Configuring Message Routing

With message routing, routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its settlement based on the message’s address and routing pattern.

With message routing, you can do the following:

  • Route messages between clients (direct-routed, or brokerless messaging)

    This involves configuring an address with a routing pattern. All messages sent to the address will be routed based on the routing pattern.

  • Route messages through a broker queue (brokered messaging)

    This involves configuring a waypoint address to identify the broker queue and then connecting the router to the broker. All messages sent to the waypoint address will be routed to the broker queue.

5.2.1. Addresses

Addresses determine how messages flow through your router network. An address designates an endpoint in your messaging network, such as:

  • Endpoint processes that consume data or offer a service
  • Topics that match multiple consumers to multiple producers

  • Entities within a messaging broker:

    • Queues
    • Durable Topics
    • Exchanges

When a router receives a message, it uses the message’s address to determine where to send the message (either its destination or one step closer to its destination).

5.2.1.1. Mobile Addresses

Routers consider addresses to be mobile such that any users of an address may be directly connected to any router in a network and may move around the topology. In cases where messages are broadcast to or balanced across multiple consumers, the address users may be connected to multiple routers in the network.

Mobile addresses are rendezvous points for senders and receivers. Messages arrive at the mobile address and are dispatched to their destinations according to the routing defined for the mobile address. The details of these routing patterns are discussed later.

Mobile addresses may be discovered during normal router operation or configured through management settings.

5.2.1.1.1. Discovered Mobile Addresses

Mobile addresses are created when a client creates a link to a source or destination address that is unknown to the router network.

Suppose a service provider wants to offer my-service that clients may use. The service provider must open a receiver link with source address my-service. The router creates a mobile address my-service and propagates the address so that it is known to every router in the network.

Later a client wants to use the service and creates a sending link with target address my-service. The router matches the service provider’s receiver having source address my-service to the client’s sender having target address my-service and routes messages between the two.

Any number of other clients can create links to the service as well. The clients do not have to know where in the router network the service provider is physically located nor are the clients required to connect to a specific router to use the service. Regardless of how many clients are using the service the service provider needs only a single connection and link into the router network.

Another view of this same scenario is when a client tries to use the service before service provider has connected to the network. In this case the router network creates the mobile address my-service as before. However, since the mobile address has only client sender links and no receiver links the router stalls the clients and prevents them from sending any messages. Later, after the service provider connects and creates the receiver link, the router will issue credits to the clients and the messages will begin to flow between the clients and the service.

The service provider can connect, disconnect, and reconnect from a different location without having to change any of the clients or their connections. Imagine having the service running on a laptop. One day the connection is from corporate headquarters and the next day the connection is from some remote location. In this case the service provider’s computer will typically have different host IP addresses for each connection. Using the router network the service provider connects to the router network and offers the named service and the clients connect to the router network and consume from the named service. The router network routes messages between the mobile addresses effectively masking host IP addresses of the service provider and the client systems.

5.2.1.1.2. Configured Mobile Addresses

Mobile addresses may be configured using the router autoLink object. An address created via an autoLink represents a queue, topic, or other service in an external broker. Logically the autoLink addresses are treated by the router network as if the broker had connected to the router and offered the services itself.

For each configured mobile address the router will create a single link to the external resource. Messages flow between sender links and receiver links the same regardless if the mobile address was discovered or configured.

Multiple autoLink objects may define the same address on multiple brokers. In this case the router network creates a sharded resource split between the brokers. Any client can seamlessly send and receive messages from either broker.

Note that the brokers do not need to be clustered or federated to receive this treatment. The brokers may even be from different vendors or be different versions of the same broker yet still work together to provide a larger service platform.

5.2.2. Routing Patterns

Routing patterns define the paths that a message with a mobile address can take across a network. These routing patterns can be used for both direct routing, in which the router distributes messages between clients without a broker, and indirect routing, in which the router enables clients to exchange messages through a broker.

Routing patterns fall into two categories: Anycast (Balanced and Closest) and Multicast. There is no concept of "unicast" in which there is only one consumer for an address.

Anycast distribution delivers each message to one consumer whereas multicast distribution delivers each message to all consumers.

Each address has one of the following routing patterns, which define the path that a message with the address can take across the messaging network:

Balanced

An anycast method that allows multiple consumers to use the same address. Each message is delivered to a single consumer only, and AMQ Interconnect attempts to balance the traffic load across the router network.

If multiple consumers are attached to the same address, each router determines which outbound path should receive a message by considering each path’s current number of unsettled deliveries. This means that more messages will be delivered along paths where deliveries are settled at higher rates.

Note

AMQ Interconnect neither measures nor uses message settlement time to determine which outbound path to use.

In this scenario, the messages are spread across both receivers regardless of path length:

Figure 5.3. Balanced Message Routing

Balanced Message Routing
Closest

An anycast method in which every message is sent along the shortest path to reach the destination, even if there are other consumers for the same address.

AMQ Interconnect determines the shortest path based on the topology cost to reach each of the consumers. If there are multiple consumers with the same lowest cost, messages will be spread evenly among those consumers.

In this scenario, all messages sent by Sender will be delivered to Receiver 1:

Figure 5.4. Closest Message Routing

Closest Message Routing
Multicast

Messages are sent to all consumers attached to the address. Each consumer will receive one copy of the message.

In this scenario, all messages are sent to all receivers:

Figure 5.5. Multicast Message Routing

Multicast Message Routing

5.2.3. Message Settlement

Message settlement is negotiated between the producer and the router when the producer establishes a link to the router. Depending on the settlement pattern, messages might be delivered with any of the following degrees of reliability:

  • At most once
  • At least once
  • Exactly once

AMQ Interconnect treats all messages as either pre-settled or unsettled, and it is responsible for propagating the settlement of each message it routes.

Pre-settled
Sometimes called fire and forget, the router settles the incoming and outgoing deliveries and propagates the settlement to the message’s destination. However, it does not guarantee delivery.
Unsettled

The router propagates the settlement between the sender and receiver, and guarantees one of the following outcomes:

  • The message is delivered and settled, with the consumer’s disposition indicated.

  • The delivery is settled with a disposition of RELEASED.

    This means that the message did not reach its destination.

  • The delivery is settled with a disposition of MODIFIED.

    This means that the message might or might not have reached its destination. The delivery is considered to be "in-doubt" and should be re-sent if "at least once" delivery is required.

  • The link, session, or connection to AMQ Interconnect was dropped, and all deliveries are "in-doubt".

5.2.4. Routing Pattern Reliability

The following table describes the levels of reliability provided by each routing pattern:

Routing patternReliable?

Anycast (Balanced or Closest)

Yes, when the message deliveries are unsettled.

There is a reliability contract that the router network abides by when delivering unsettled messages to anycast addresses. For every such delivery sent by a producer, the router network guarantees that one of the following outcomes will occur:

  • The delivery shall be settled with ACCEPTED or REJECTED disposition where the disposition is supplied by the consumer.
  • The delivery shall be settled with RELEASED disposition, meaning that the message was not delivered to any consumer.
  • The delivery shall be settled with MODIFIED disposition, meaning that the message may have been delivered to a consumer but should be considered in-doubt and re-sent.
  • The connection to the producer shall be dropped, signifying that all unsettled deliveries should now be considered in-doubt by the producer and later re-sent.

Multicast

No.

If a producer sends an unsettled delivery, the disposition may be ACCEPTED or RELEASED.

  • If ACCEPTED, there is no guarantee that the message was delivered to any consumer.
  • If RELEASED, the message was definitely not delivered to any consumer.

5.2.5. Configuring Addresses for Prioritized Message Delivery

You can set the priority level of an address to control how AMQ Interconnect processes messages sent to that address. Within the scope of a connection, AMQ Interconnect attempts to process messages based on their priority. For a connection with a large volume of messages in flight, this lowers the latency for higher-priority messages.

Assigning a high priority level to an address does not guarantee that messages sent to the address will be delivered before messages sent to lower-priority addresses. However, higher-priority messages will travel more quickly through the router network than they otherwise would.

Note

You can also control the priority level of individual messages by setting the priority level in the message header. However, the address priority takes precedence: if you send a prioritized message to an address with a different priority level, the router will use the address priority level.

Procedure

  • In the router’s configuration file, add or edit an address and assign a priority level.

    This example adds an address with the highest priority level. The router will attempt to deliver messages sent to this address before messages with lower priority levels. 

    address {
    prefix: my-high-priority-address
    priority: 9
    ...
}
    priority
    The priority level to assign to all messages sent to this address. The range of valid priority levels is 0-9, in which the higher the number, the higher the priority. The default is 4.

Additional resources

5.2.6. Routing Messages Between Clients

You can route messages between clients without using a broker. In a brokerless scenario (sometimes called direct-routed messaging), AMQ Interconnect routes messages between clients directly.

To route messages between clients, you configure an address with a routing distribution pattern. When a router receives a message with this address, the message is routed to its destination or destinations based on the address’s routing distribution pattern.

Procedure

  1. In the router’s configuration file, add an address section: 

    address {
    prefix: ADDRESS_PREFIX
    distribution: balanced|closest|multicast
    ...
}
    prefix | pattern

    The address or group of addresses to which the address settings should be applied. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards.

    A prefix matches either an exact address or the beginning segment within an address that is delimited by either a . or / character. For example, the prefix my_address would match the address my_address as well as my_address.1 and my_address/1. However, it would not match my_address1.

    A pattern matches an address that corresponds to a pattern. A pattern is a sequence of words delimited by either a . or / character. You can use wildcard characters to represent a word. The * character matches exactly one word, and the # character matches any sequence of zero or more words.

    The * and # characters are reserved as wildcards. Therefore, you should not use them in the message address.

    For more information about creating address patterns, see Section 5.3.6, “Pattern Matching for Addresses”.

    Note

    You can convert a prefix value to a pattern by appending /# to it. For example, the prefix a/b/c is equivalent to the pattern a/b/c/#.

    distribution

    The message distribution pattern. The default is balanced, but you can specify any of the following options:

    • balanced - Messages sent to the address will be routed to one of the receivers, and the routing network will attempt to balance the traffic load based on the rate of settlement.
    • closest - Messages sent to the address are sent on the shortest path to reach the destination. It means that if there are multiple receivers for the same address, only the closest one will receive the message.

    • multicast - Messages are sent to all receivers that are attached to the address in a publish/subscribe model.

      For more information about message distribution patterns, see Routing Patterns.

    For information about additional attributes, see address in the qdrouterd.conf man page.

  2. Add the same address section to any other routers that need to use the address.

    The address that you added to this router configuration file only controls how this router distributes messages sent to the address. If you have additional routers in your router network that should distribute messages for this address, then you must add the same address section to each of their configuration files.

5.2.7. Routing Messages Through a Broker Queue

You can route messages to and from a broker queue to provide clients with access to the queue through a router. In this scenario, clients connect to a router to send and receive messages, and the router routes the messages to or from the broker queue.

You can route messages to a queue hosted on a single broker, or route messages to a sharded queue distributed across multiple brokers.

Figure 5.6. Brokered Messaging

Brokered Messaging

In this diagram, the sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to the broker, and then sends the messages to my_queue. Later, the receiver connects to the router and requests messages from my_queue. The router attaches an incoming link to the broker to receive the messages from my_queue, and then delivers them to the receiver.

You can also route messages to a sharded queue, which is a single, logical queue comprised of multiple, underlying physical queues. Using queue sharding, it is possible to distribute a single queue over multiple brokers. Clients can connect to any of the brokers that hold a shard to send and receive messages.

Figure 5.7. Brokered Messaging with Sharded Queue

Brokered Messaging with Sharded Queue

In this diagram, a sharded queue (my_queue) is distributed across two brokers. The router is connected to the clients and to both brokers. The sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to each broker, and then sends messages to each shard (by default, the routing distribution is balanced). Later, the receiver connects to the router and requests all of the messages from my_queue. The router attaches an incoming link to one of the brokers to receive the messages from my_queue, and then delivers them to the receiver.

Procedure

  1. Add a waypoint address.

    This address identifies the queue to which you want to route messages.

  2. Add autolinks to connect the router to the broker.

    Autolinks connect the router to the broker queue identified by the waypoint address.

  3. If the queue is sharded, add autolinks for each additional broker that hosts a shard.

5.2.7.1. Configuring Waypoint Addresses

A waypoint address identifies a queue on a broker to which you want to route messages. You need to configure the waypoint address on each router that needs to use the address. For example, if a client is connected to Router A to send messages to the broker queue, and another client is connected to Router B to receive those messages, then you would need to configure the waypoint address on both Router A and Router B.

Prerequisites

An incoming connection (listener) to which the clients can connect should be configured. This connection defines how the producers and consumers connect to the router to send and receive messages. For more information, see Section 4.2.2, “Listening for client connections”.

Procedure

  • Create waypoint addresses on each router that needs to use the address: 

    address {
    prefix: ADDRESS_PREFIX
    waypoint: yes
}
    prefix | pattern

    The address prefix or pattern that matches the broker queue to which you want to send messages. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards.

    A prefix matches either an exact address or the beginning segment within an address that is delimited by either a . or / character. For example, the prefix my_address would match the address my_address as well as my_address.1 and my_address/1. However, it would not match my_address1.

    A pattern matches an address that corresponds to a pattern. A pattern is a sequence of words delimited by either a . or / character. You can use wildcard characters to represent a word. The * character matches exactly one word, and the # character matches any sequence of zero or more words.

    The * and # characters are reserved as wildcards. Therefore, you should not use them in the message address.

    For more information about creating address patterns, see Section 5.3.6, “Pattern Matching for Addresses”.

    Note

    You can convert a prefix value to a pattern by appending /# to it. For example, the prefix a/b/c is equivalent to the pattern a/b/c/#.

    waypoint
    Set this attribute to yes so that the router handles messages sent to this address as a waypoint.

5.2.7.2. Connecting a Router to the Broker

After you add waypoint addresses to identify the broker queue, you must connect a router to the broker using autolinks.

With autolinks, client traffic is handled on the router, not the broker. Clients attach their links to the router, and then the router uses internal autolinks to connect to the queue on the broker. Therefore, the queue will always have a single producer and a single consumer regardless of how many clients are attached to the router.

Note

If the connection to the broker fails, AMQ Interconnect automatically attempts to reestablish the connection and reroute message deliveries to any available alternate destinations. However, some deliveries could be returned to the sender with a RELEASED or MODIFIED disposition. Therefore, you should ensure that your clients can handle these deliveries appropriately (generally by resending them).

  1. If this router is different than the router that is connected to the clients, then add the waypoint address.

  2. Add an outgoing connection to the broker: 

    connector {
    name: NAME
    host: HOST_NAME/ADDRESS
    port: PORT_NUMBER/NAME
    role: route-container
    ...
}
    name
    The name of the connector. Specify a name that describes the broker.
    host
    Either an IP address (IPv4 or IPv6) or hostname on which the router should connect to the broker.
    port
    The port number or symbolic service name on which the router should connect to the broker.
    role
    Specify route-container to indicate that this connection is for an external container (broker).

    For information about additional attributes, see connector in the qdrouterd.conf man page.

  3. If you want to send messages to the broker queue, create an outgoing autolink to the broker queue: 

    autoLink {
    address: ADDRESS
    connection: CONNECTOR_NAME
    direction: out
    ...
}
    address
    The address of the broker queue. When the autolink is created, it will be attached to this address.
    externalAddress
    An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the sender uses. In this scenario, senders send messages to the addr address, and then the router routes them to the broker queue represented by the externalAddress address.
    connection | containerID
    How the router should connect to the broker. You can specify either an outgoing connection (connection) or the container ID of the broker (containerID).
    direction
    Set this attribute to out to specify that this autolink can send messages from the router to the broker.

    For information about additional attributes, see autoLink in the qdrouterd.conf man page.

  4. If you want to receive messages from the broker queue, create an incoming autolink from the broker queue: 

    autoLink {
    address: ADDRESS
    connection: CONNECTOR_NAME
    direction: in
    ...
}
    address
    The address of the broker queue. When the autolink is created, it will be attached to this address.
    externalAddress
    An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the receiver uses. In this scenario, receivers receive messages from the addr address, and the router retrieves them from the broker queue represented by the externalAddress address.
    connection | containerID
    How the router should connect to the broker. You can specify either an outgoing connection (connection) or the container ID of the broker (containerID).
    direction
    Set this attribute to in to specify that this autolink can receive messages from the broker to the router.

    For information about additional attributes, see autoLink in the qdrouterd.conf man page.

5.2.8. Handling Undeliverable Messages for an Address

You handle undeliverable messages for an address by configuring autolinks that point to fallback destinations. A fallback destination (such as a queue on a broker) stores messages that are not directly routable to any consumers.

During normal message delivery, AMQ Interconnect delivers messages to the consumers that are attached to the router network. However, if no consumers are reachable, the messages are diverted to any fallback destinations that were configured for the address (if the autolinks that point to the fallback destinations are active). When a consumer reconnects and becomes reachable again, it receives the messages stored at the fallback destination.

Note

AMQ Interconnect preserves the original delivery order for messages stored at a fallback destination. However, when a consumer reconnects, any new messages produced while the queue is draining will be interleaved with the messages stored at the fallback destination.

Prerequisites

Procedure

This procedure enables fallback for an address and configures autolinks to connect to the broker queue that provides the fallback destination for the address.

  1. Enable fallback destinations for the address. 

    address {
    prefix: my-address
    enableFallback: yes
}

  2. Add an outgoing autolink to a queue on the broker.

    For the address for which you enabled fallback, if messages are not routable to any consumers, the router will use this autolink to send the messages to a queue on the broker. 

    autoLink {
    address: my-address.2
    direction: out
    connection: my-broker
    fallback: yes
}

  3. If you want the router to send queued messages to attached consumers as soon as they connect to the router network, add an incoming autolink.

    As soon as a consumer attaches to the router, it will receive the messages stored in the broker queue, along with any new messages sent by the producer. The original delivery order of the queued messages is preserved; however, the queued messages will be interleaved with the new messages.

    If you do not add the incoming autolink, the messages will be stored on the broker, but will not be sent to consumers when they attach to the router. 

    autoLink {
    address: my-address.2
    direction: in
    connection: my-broker
    fallback: yes
}

5.2.9. Example: Routing Messages Through Broker Queues

This example shows how waypoints and autolinks can route messages through a pair of queues on a broker.

5.2.9.1. Router Configuration

connector {  1
    name: broker
    role: route-container
    host: 198.51.100.1
    port: 61617
    saslMechanisms: ANONYMOUS
}

address {  2
    prefix: queue
    waypoint: yes
}

autoLink {  3
    address: queue.first
    direction: in
    connection: broker
}

autoLink {  4
    address: queue.first
    direction: out
    connection: broker
}

autoLink {  5
    address: queue.second
    direction: in
    connection: broker
}

autoLink {  6
    address: queue.second
    direction: out
    connection: broker
}
1
The outgoing connection from the router to the broker. The route-container role enables the router to connect to an external AMQP container (in this case, a broker).
2
The namespace queue on the broker to which the router should route messages. All addresses that start with queue will be routed to a queue on the broker.
3
The incoming autolink from queue.first on the broker to the router.
4
The outgoing autolink from the router to queue.first on the broker.
5
The incoming autolink from queue.second on the broker to the router.
6
The outgoing autolink from the router to queue.second on the broker.

5.2.9.2. How the Messages are Routed

Initially, when the broker is offline, the autolinks are inactive. 

$ qdstat --autolinks
AutoLinks
  addr          dir  phs  extAddr  link  status    lastErr
  ========================================================
  queue.first   in   1                   inactive
  queue.first   out  0                   inactive
  queue.second  in   1                   inactive
  queue.second  out  0                   inactive

Once the broker is online, the autolinks attempt to activate. In this case, the broker starts with the queue.first queue only, and the queue.first autolinks become active. The queue.second autolinks are in a failed state, because the queue.second queue does not exist on the broker. 

$ qdstat --autolinks
AutoLinks
  addr          dir  phs  extAddr  link  status  lastErr
  ===========================================================================
  queue.first   in   1             6     active
  queue.first   out  0             7     active
  queue.second  in   1                   failed  Node not found: queue.second
  queue.second  out  0                   failed  Node not found: queue.second

The producer now connects to the router and sends three messages to queue.first. 

$ python simple_send.py -a 127.0.0.1/queue.first -m3
all messages confirmed

The router’s address statistics show that the messages were delivered to the queue. 

$ qdstat -a
Router Addresses
  class   addr           phs  distrib   in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  ========================================================================================================
  mobile  queue.first    1    balanced  0        0      0       0      0   0    0     0        0
  mobile  queue.first    0    balanced  0        1      0       0      3   3    0     0        0

The queue.first address appears twice in the output: once for each phase of the address. Phase 0 is for routing messages from producers to the outgoing autolink. Phase 1 is for routing messages from the incoming autolink to the subscribed consumers. In this case, Phase 0 of the address has counted three messages in the in column (the messages that arrived on the router from the producer), and three messages in the out column (the messages that were sent from the router to the broker queue).

The consumer now connects to the router and receives the three messages from queue.first. 

$ python simple_recv.py -a 127.0.0.1:5672/queue.first -m3
{u'sequence': int32(1)}
{u'sequence': int32(2)}
{u'sequence': int32(3)}

The router’s address statistics now show that all three messages were received by the consumer from the broker queue. 

$ qdstat -a
Router Addresses
  class   addr           phs  distrib   in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  ========================================================================================================
  mobile  queue.first    1    balanced  0        0      0       0      3   3    0     0        0
  mobile  queue.first    0    balanced  0        1      0       0      3   3    0     0        0

The command output shows that Phase 1 of the address was used to deliver all three messages from the queue to the consumer.

Note

Even in a multi-router network, and with multiple producers and consumers for queue.first, all deliveries are routed through the queue on the connected broker.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.