Chapter 33. Configure JGroups

33.1. About JGroups

JGroups is the underlying group communication library used to connect Red Hat JBoss Data Grid instances. For a full list of JGroups protocols supported in JBoss Data Grid, see Supported JGroups Protocols.

33.2. Configure Red Hat JBoss Data Grid Interface Binding (Remote Client-Server Mode)

33.2.1. Interfaces

Red Hat JBoss Data Grid allows users to specify an interface type rather than a specific (unknown) IP address.

Note

If you use JBoss Data Grid as a containerized image with OpenShift, you must set environment variables to change the XML configuration during pod creation. See the Data Grid for OpenShift Guide.

link-local: Uses a 169.x.x.x or 254.x.x.x address. This suits the traffic within one box.

<interfaces>
    <interface name="link-local">
        <link-local-address/>
    </interface>
    <!-- Additional configuration elements here -->
</interfaces>

site-local: Uses a private IP address, for example 192.168.[replaceable]x.[replaceable]x````. This prevents extra bandwidth charged from GoGrid, and similar providers.
```
<interfaces>
    <interface name="site-local">
        <site-local-address/>
    </interface>
    
</interfaces>
```

global: Picks a public IP address. This should be avoided for replication traffic.

<interfaces>
    <interface name="global">
        <any-address/>
    </interface>
    <!-- Additional configuration elements here -->
</interfaces>

non-loopback: Uses the first address found on an active interface that is not a 127.x.x.x address.

<interfaces>
    <interface name="non-loopback">
        <not>
	    <loopback />
	</not>
    </interface>
</interfaces>

33.2.2. Binding Sockets

33.2.2.1. Binding Sockets

Socket bindings provide a method of associating a name with networking details, such as an interface, a port, a multicast-address, or other details. Sockets may be bound to the interface either individually or using a socket binding group.

33.2.2.2. Binding a Single Socket Example

The following is an example depicting the use of JGroups interface socket binding to bind an individual socket using the socket-binding element.

Socket Binding

<socket-binding name="jgroups-udp" <!-- Additional configuration elements here --> interface="site-local"/>

33.2.2.3. Binding a Group of Sockets Example

The following is an example depicting the use of Groups interface socket bindings to bind a group, using the socket-binding-group element:

Bind a Group

<socket-binding-group name="ha-sockets" default-interface="global">
	<!-- Additional configuration elements here -->
	<socket-binding name="jgroups-tcp" port="7600"/>
	<socket-binding name="jgroups-tcp-fd" port="57600"/>
	<!-- Additional configuration elements here -->
</socket-binding-group>

The two sample socket bindings in the example are bound to the same default-interface (global), therefore the interface attribute does not need to be specified.

33.2.3. Configure JGroups Socket Binding

Each JGroups stack, configured in the JGroups subsystem, uses a specific socket binding. Set up the socket binding as follows:

JGroups UDP Socket Binding Configuration

The following example utilizes UDP automatically form the cluster. In this example the jgroups-udp socket binding is defined for the transport:

<subsystem xmlns="urn:jboss:domain:jgroups:3.0" default-stack="udp">
    <stack name="udp">
        <transport type="UDP" socket-binding="jgroups-udp">
            <!-- Additional configuration elements here -->
        </transport>
        <protocol type="PING"/>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
        <protocol type="FD_ALL"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2"/>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="UFC"/>
        <protocol type="MFC"/>
        <protocol type="FRAG3"/>
    </stack>
</subsystem>

JGroups TCP Socket Binding Configuration

The following example uses TCP to establish direct communication between two clusters nodes. In the example below node1 is located at 192.168.1.2:7600, and node2 is located at 192.168.1.3:7600. The port in use will be defined by the jgroups-tcp property in the socket-binding section.

<subsystem xmlns="urn:infinispan:server:jgroups:8.0" default-stack="tcp">
    <stack name="tcp">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="TCPPING">
            <property name="initial_hosts">192.168.1.2[7600],192.168.1.3[7600]</property>
            <property name="num_initial_members">2</property>
            <property name="port_range">0</property>
            <property name="timeout">2000</property>
        </protocol>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
        <protocol type="FD_ALL"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2">
            <property name="use_mcast_xmit">false</property>
        </protocol>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="MFC"/>
        <protocol type="FRAG3"/>
    </stack>
</subsystem>

The decision of UDP vs TCP must be made in each environment. By default JGroups uses UDP, as it allows for dynamic detection of clustered members and scales better in larger clusters due to a smaller network footprint. In addition, when using UDP only one packet per cluster is required, as multicast packets are received by all subscribers to the multicast address; however, in environments where multicast traffic is prohibited, or if UDP traffic can not reach the remote cluster nodes, such as when cluster members are on separate VLANs, TCP traffic can be used to create a cluster.

Important

When using UDP as the JGroups transport, the socket binding has to specify the regular (unicast) port, multicast address, and multicast port.

33.3. Configure JGroups (Library Mode)

33.3.1. Configure JGroups for Clustered Modes

Red Hat JBoss Data Grid must have an appropriate JGroups configuration in order to operate in clustered mode.

JGroups XML Configuration

<infinispan xmlns="urn:infinispan:config:8.5">
    <jgroups>
        <stack-file name="jgroupsStack" path="/path/to/jgroups/xml/jgroups.xml}"/>
    </jgroups>
    <cache-container name="default" default-cache="default">
        <transport stack="jgroupsStack" lock-timeout="600000" cluster="default" />
    </cache-container>
</infinispan>

JBoss Data Grid will first search for jgroups.xml in the classpath; if no instances are found in the classpath it will then search for an absolute path name.

33.3.2. JGroups Transport Protocols

33.3.2.1. JGroups Transport Protocols

A transport protocol is the protocol at the bottom of a protocol stack. Transport Protocols are responsible for sending and receiving messages from the network.

Red Hat JBoss Data Grid ships with both UDP and TCP transport protocols.

33.3.2.2. The UDP Transport Protocol

UDP is a transport protocol that uses:

IP multicasting to send messages to all members of a cluster.
UDP datagrams for unicast messages, which are sent to a single member.

When the UDP transport is started, it opens a unicast socket and a multicast socket. The unicast socket is used to send and receive unicast messages, the multicast socket sends and receives multicast sockets. The physical address of the channel will be the same as the address and port number of the unicast socket.

UDP is the default transport protocol used in JBoss Data Grid as it allows for dynamic discovery of additional cluster nodes, and scales well with cluster sizes.

33.3.2.3. The TCP Transport Protocol

TCP/IP is a replacement transport for UDP in situations where IP multicast cannot be used, such as operations over a WAN where routers may discard IP multicast packets.

TCP is a transport protocol used to send unicast and multicast messages.

When sending multicast messages, TCP sends multiple unicast messages.

As IP multicasting cannot be used to discover initial members, another mechanism must be used to find initial membership.

33.3.2.4. Using the TCPPing Protocol

Some networks only allow TCP to be used. The pre-configured default-configs/default-jgroups-tcp.xml includes the MPING protocol, which uses UDP multicast for discovery. When UDP multicast is not available, the MPING protocol, has to be replaced by a different mechanism. The recommended alternative is the TCPPING protocol. The TCPPING configuration contains a static list of IP addresses which are contacted for node discovery.

Configure the JGroups Subsystem to Use TCPPING

<TCP bind_port="7800" />
<TCPPING initial_hosts="${jgroups.tcpping.initial_hosts:HostA[7800],HostB[7801]}"
         port_range="1" />

33.3.3. Pre-Configured JGroups Files

33.3.3.1. Pre-Configured JGroups Files

Red Hat JBoss Data Grid ships with pre-configured JGroups files in infinispan-embedded.jar, available on the classpath by default.

These JGroups configuration files contain default settings that you should adjust and tune to achieve optimal network performance.

To use a JGroups configuration file, replace jgroups.xml with one of the following:

default-configs/default-jgroups-udp.xml
default-configs/default-jgroups-tcp.xml
default-configs/default-jgroups-ec2.xml
default-configs/default-jgroups-google.xml
default-configs/default-jgroups-kubernetes.xml