Appendix A. Recommended JGroups Values for JBoss Data Grid

TCP

TCP/IP is a replacement transport for UDP in situations where IP multicast cannot be used, such as operations over a WAN where routers may discard IP multicast packets.

TCP is a transport protocol used to send unicast and multicast messages.

As IP multicasting cannot be used to discover initial members, another mechanism must be used to find initial membership.

Red Hat JBoss Data Grid’s Hot Rod is a custom TCP client/server protocol.

UDP

UDP is a transport protocol that uses:

When the UDP transport is started, it opens a unicast socket and a multicast socket. The unicast socket is used to send and receive unicast messages, the multicast socket sends and receives multicast sockets. The physical address of the channel with be the same as the address and port number of the unicast socket.

PING

The PING protocol is used for the initial discovery of members. It is used to detect the coordinator, which is the oldest member, by multicasting PING requests to an IP multicast address.

Each member responds to the ping with a packet containing the coordinator’s and their own address. After a specified number of milliseconds (N) or replies (M), the joiner determines the coordinator from the responses and sends it a JOIN request (handled by GMS). If there is no response, the joiner is considered the first member of the group.

PING differs from TCPPING because it used dynamic discovery, which means that a member does not need to know in advance where the other cluster members are. PING uses the transport’s IP multicasting abilities to send a discovery request to the cluster. As a result, PING requires UDP as transport.

TCPPING

The TCCPING protocol uses a set of known members and pings them for discovery. This protocol has a static configuration.

MPING

The MPING (Multicast PING) protocol uses IP multicast to discover the initial membership. It can be used with all transports, but is usually used in combination with TCP.

S3_PING

S3_PING is a discovery protocol that is ideal for use with Amazon’s Elastic Compute Cloud (EC2) because EC2 does not allow multicast and therefore MPING is not allowed.

Each EC2 instance adds a small file to an S3 data container, known as a bucket. Each instance then reads the files in the bucket to discover the other members of the cluster.

JDBC_PING

JDBC_PING is a discovery protocol that utilizes a shared database to store information regarding nodes in the cluster.

TCPGOSSIP

TCPGOSSIP is a discovery protocol that uses one or more configured GossipRouter processes to store information about the nodes in the cluster.

MERGE3

The MERGE3 protocol is available in JGroups 3.1 onwards. Unlike MERGE2, in MERGE3, all members periodically send an INFO message with their address (UUID), logical name, physical address and View ID. Periodically, each coordinator reviews the INFO details to ensure that there are no inconsistencies.

FD_ALL

Used for failure detection, FD_ALL uses a simple heartbeat protocol. Each member maintains a table of all other members (except itself) and periodically multicasts a heartbeat. For example, when data or a heartbeat from P is received, the timestamp for P is set to the current time. Periodically, expired members are identified using the timestamp values.

FD_SOCK

FD_SOCK is a failure detection protocol based on a ring of TCP sockets created between cluster members. Each cluster member connects to its neighbor (the last member connects to the first member), which forms a ring. Member B is suspected when its neighbor A detects an abnormal closing of its TCP socket (usually due to node B crashing). However, if member B is leaving gracefully, it informs member A and does not become suspected when it does exit.

FD_HOST

FD_HOST is a failure detection protocol that detects the crashing or hanging of entire hosts and suspects all cluster members of that host through ICMP ping messages or custom commands. FD_HOST does not detect the crashing or hanging of single members on the local hosts, but only checks whether all the other hosts in the cluster are live and available. It is therefore used in conjunction with other failure detection protocols such as FD_ALL and FD_SOCK. This protocol is typically used when multiple cluster members are running on the same physical box.

The FD_HOST protocol is supported on Windows for JBoss Data Grid. The cmd parameter must be set to ping.exe and the ping count must be specified.

VERIFY_SUSPECT

The VERIFY_SUSPECT protocol verifies whether a suspected member is dead by pinging the member before excluding it. If the member responds, the suspect message is discarded.

NAKACK2

The NAKACK2 protocol is a successor to the NAKACK protocol and was introduced in JGroups 3.1.

The NACKACK2 protocol is used for multicast messages and uses NAK. Each message is tagged with a sequence number. The receiver tracks the sequence numbers and delivers the messages in order. When a gap in the sequence numbers is detected, the receiver asks the sender to retransmit the missing message.

UNICAST3

The UNICAST3 protocol provides reliable delivery (no message sent by a sender is lost because they are sent in a numbered sequence) and uses the FIFO (First In First Out) properties for point to point messages between a sender and a receiver.

UNICAST3 uses positive acks for retransmission. For example, sender A keeps sending message M until receiver B receives message M and returns an ack to indicate a successful delivery. Sender A keeps resending message M until it receives an ack from B, until B leaves the cluster, or A crashes.

STABLE

The STABLE protocol is a garbage collector for messages that have been viewed by all members in a cluster. Each member stores all messages because retransmission may be required. A message can only be removed from the retransmission buffers when all members have seen the message. The STABLE protocol periodically gossips its highest and lowest messages seen. The lowest value is used to compute the min (all lowest sequence numbers for all members) and messages with a sequence number below the min value can be discarded

GMS

The GMS protocol is the group membership protocol. This protocol handles joins/leaves/crashes (suspicions) and emits new views accordingly.

MFC

MFC is the Multicast version of the flow control protocol.

UFC

UFC is the Unicast version of the flow control protocol.

FRAG3

The FRAG3 protocol fragments large messages into smaller ones and then sends the smaller messages. At the receiver side, the smaller fragments are reassembled into larger, complete messages and delivered to the application. FRAG3 is used for both multicast and unicast messages.

SYM_ENCRYPT

JGroups includes the SYM_ENCRYPT protocol to provide encryption for cluster traffic. By default, encryption only encrypts the message body; it does not encrypt message headers. To encrypt the entire message, including all headers, as well as destination and source addresses, the property encrypt_entire_message must be true. When defining this protocol it should be placed directly under NAKACK2.

The SYM_ENCRYPT layer is used to encrypt and decrypt communication in JGroups by defining a secret key in a keystore.

Each message is identified as encrypted with a specific encryption header identifying the encrypt header and an MD5 digest identifying the version of the key being used to encrypt and decrypt messages.

ASYM_ENCRYPT

JGroups includes the ASYM_ENCRYPT protocol to provide encryption for cluster traffic. By default, encryption only encrypts the message body; it does not encrypt message headers. To encrypt the entire message, including all headers, as well as destination and source addresses, the property encrypt_entire_message must be true. When defining this protocol it should be placed directly under NAKACK2.

The ASYM_ENCRYPT layer is used to encrypt and decrypt communication in JGroups by having a coordinator generate a secret key using defined algorithms and key sizes.

Each message is identified as encrypted with a specific encryption header identifying the encrypt header and an MD5 digest identifying the version of the key being used to encrypt and decrypt messages.

SASL

The SASL (Simple Authentication and Security Layer) protocol is a framework that provides authentication and data security services in connection-oriented protocols using replaceable mechanisms. Additionally, SASL provides a structured interface between protocols and mechanisms.

RELAY2

The RELAY protocol bridges two remote clusters by creating a connection between one node in each site. This allows multicast messages sent out in one site to be relayed to the other and vice versa.

JGroups includes the RELAY2 protocol, which is used for communication between sites in Red Hat JBoss Data Grid’s Cross-Site Replication.

The RELAY2 protocol works similarly to RELAY but with slight differences. Unlike RELAY, the RELAY2 protocol:

bind_addr

Any non-loopback

Set address on specific interface

bind_port

Any free port

Set specific port

loopback

true

Same as default

port_range

50

Set based on desired range of ports

recv_buf_size

150,000

Same as default

send_buf_size

150,000

640,000

sock_conn_timeout

2,000

300

bundler_type

transfer-queue

no-bundler

max_bundle_size

64,000

64,000

enable_diagnostics

true

false

thread_pool.enabled

true

Same as default

thread_pool.min_threads

2

This should equal the number of nodes

thread_pool.max_threads

30

This should be higher than thread_pool.min_threads. For example, for a smaller grid (2-10 nodes), set this value to twice the number of nodes, but for a larger grid (20 or more nodes), the ratio should be lower. As an example, if a grid contains 20 nodes, set this value to 25 and if the grid contains 100 nodes, set the value to 110.

thread_pool.keep_alive_time

30,000

60,000

bind_addr

Any non-loopback

Set address on specific interface

break_on_coord_rsp

true

Same as default

mcast_addr

230.5.6.7

228.2.4.6

mcast_port

7555

43366

ip_ttl

8

2

min_interval

1,000

10,000

max_interval

10,000

30,000

client_bind_por

0 (randomly selects a port and uses it)

Same as default

get_cache_timeout

1000 milliseconds

Same as default

keep_alive

true

Same as default

num_tries

3

Same as default

start_port

0 (randomly selects a port and uses it)

Same as default

suspect_msg_interval

5000 milliseconds.

Same as default

timeout

40,000

60,000. The FD_ALL timeout value is set to two times the longest possible stop the world garbage collection pause in the CMS garbage collector. In a well-tuned JVM, the longest pause is proportional to heap size and should not exceed 1 second per GB of heap. For example, an 8GB heap should not have a pause longer than 8 seconds, so the FD_ALL timeout value must be set to 16 seconds. If longer garbage collection pauses are used, then this timeout value should be increased to avoid false failure detection on a node.

interval

8,000

15,000. The FD_ALL interval value must be at least four times smaller than the value set for FD_ALL’s timeout value.

timeout_check_interval

2,000

5,000

check_timeout

3,000

5,000

cmd

InetAddress.isReachable() (ICMP ping)

-

interval

20,000

15,000. The interval value for FD_HOST must be four times smaller than FD_HOST’s timeout value.

timeout

60,000

60,000.

timeout

2,000

5,000

use_mcast_xmit

true

false

xmit_interval

1,000

100

xmit_table_num_rows

50

Same as default

xmit_table_msgs_per_row

10,000

1,024

xmit_table_max_compaction_time

10,000

30,000

max_msg_batch_size

100

Same as default

resend_last_seqno

false

true

xmit_interval

500

100

xmit_table_num_rows

100

50

xmit_table_msgs_per_row

1,0000

1,024

xmit_table_max_compaction_time

600,000

30,000

max_msg_batch_size

500

100

conn_close_timeout

60,000

No recommended value.

conn_expiry_timeout

120,000

0

stability_delay

6,000

500

desired_avg_gossip

20,000

5,000

max_bytes

2,000,000

1,000,000

print_local_addr

true

false

join_timeout

5,000

Same as default

view_bundling

true

Same as default

max_credits

500,000

2,000,000

min_threshold

0.40

Same as default

frag_size

60,000

Same as default

sym_algorithm

AES

-

sym_keylength

128

-

provider

Bouncy Castle Provider

-

asym_algorithm

RSA

-

asym_keylength

512

-

sym_provider

Bouncy Castle Provider

-

change_key_on_leave

false

-

bind_addr

Any non-loopback

Set address on specific interface

bind_port

Any free port

Set specific port

loopback

true

true

port_range

50

Set based on desired range of ports

mcast_addr

228.8.8.8

228.6.7.8

mcast_port

7600

46655

tos

8

Same as default

ucast_recv_buf_size

64,000

5,000,000

ucast_send_buf_size

100,000

2,000,000

mcast_recv_buf_size

500,000

5,000,000

mcast_send_buf_size

100,000

1,000,000

ip_ttl

8

2

thread_naming_pattern

cl

pl

bundler_type

transfer-queue

no-bundler

max_bundle_size

64,000

8700

enable_diagnostics

true

false

thread_pool.enabled

true

Same as default

thread_pool.min_threads

2

This should equal the number of nodes.

thread_pool.max_threads

30

This should be higher than thread_pool.min_threads. For example, for a smaller grid (2-10 nodes), set this value to twice the number of nodes, but for a larger grid (20 or more nodes), the ratio should be lower. As an example, if a grid contains 20 nodes, set this value to 25 and if the grid contains 100 nodes, set the value to 110.

thread_pool.keep_alive_time

30,000

60,000

min_interval

1,000

10,000

max_interval

10,000

30,000

client_bind_por

0 (randomly selects a port and uses it)

Same as default

get_cache_timeout

1000 milliseconds

Same as default

keep_alive

true

Same as default

num_tries

3

Same as default

start_port

0 (randomly selects a port and uses it)

Same as default

suspect_msg_interval

5000 milliseconds.

Same as default

timeout

40,000

60,000. The FD_ALL timeout value is set to two times the longest possible stop the world garbage collection pause in the CMS garbage collector. In a well-tuned JVM, the longest pause is proportional to heap size and should not exceed 1 second per GB of heap. For example, an 8GB heap should not have a pause longer than 8 seconds, so the FD_ALL timeout value must be set to 16 seconds. If longer garbage collection pauses are used, then this timeout value should be increased to avoid false failure detection on a node.

interval

8,000

15,000. The FD_ALL interval value must be at least four times smaller than the value set for FD_ALL’s timeout value.

timeout_check_interval

2,000

5,000

check_timeout

3,000

5,000

cmd

InetAddress.isReachable() (ICMP ping)

-

interval

20,000

15,000. The interval value for FD_HOST must be four times smaller than FD_HOST’s timeout value.

timeout

-

60,000.

timeout

2,000

5,000

use_mcast_xmit

true

Same as default

xmit_interval

1,000

100

xmit_table_num_rows

50

Same as default

xmit_table_msgs_per_row

10,000

1,024

xmit_table_max_compaction_time

10,000

30,000

max_msg_batch_size

100

Same as default

resend_last_seqno

false

true

xmit_interval

500

100

xmit_table_num_rows

100

50

xmit_table_msgs_per_row

1,0000

1,024

xmit_table_max_compaction_time

600,000

30,000

max_msg_batch_size

500

100

conn_close_timeout

60,000

No recommended value

conn_expiry_timeout

120,000

0

stability_delay

6,000

500

desired_avg_gossip

20,000

5,000

max_bytes

2,000,000

1,000,000

print_local_addr

true

false

join_timeout

5,000

Same as default

view_bundling

true

Same as default

max_credits

500,000

2,000,000

min_threshold

0.40

Same as default

max_credits

500,000

2,000,000

min_threshold

0.40

Same as default

frag_size

60,000

Same as default

sym_algorithm

AES

-

sym_keylength

128

-

provider

Bouncy Castle Provider

-

asym_algorithm

RSA

-

asym_keylength

512

-

provider

Bouncy Castle Provider

-

change_key_on_leave

false

-