Chapter 7. Setting Up Cluster Transport

7.1. Getting Started with Default Stacks

Use default JGroups stacks with recommended settings as a starting point for your cluster transport layer.

Declaratively

Specify default JGroups stacks with the stack attribute.

<infinispan>
  <cache-container default-cache="replicatedCache">
    <transport stack="udp" /> 1
    ...
  </cache-container>
</infinispan>

1: uses the default-jgroups-udp.xml stack for cluster transport.

Programmatically

Specify default JGroups stacks with the addProperty() method.

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder().transport()
        .defaultTransport()
        .clusterName("qa-cluster")
        .addProperty("configurationFile", "default-jgroups-udp.xml") 1
        .build();

1: uses the default-jgroups-udp.xml stack for cluster transport.

Tip

Use the cluster-stack argument with the Data Grid server startup script.

$ bin/server.sh --cluster-stack=tcp

7.1.1. Default JGroups Stacks

Default JGroups stacks are included in infinispan-core.jar and on the classpath. You can locate the default JGroups stacks in the default-configs directory.

File name	Stack name	Description
`default-jgroups-udp.xml`	`udp`	Uses UDP for transport and UDP multicast for discovery. Suitable for larger clusters (over 100 nodes) or if you are using replicated caches or invalidation mode. Minimizes the number of open sockets.
`default-jgroups-tcp.xml`	`tcp`	Uses TCP for transport and the `MPING` protocol for discovery, which uses `UDP` multicast. Suitable for smaller clusters (under 100 nodes) only if you are using distributed caches because TCP is more efficient than UDP as a point-to-point protocol.
`default-jgroups-ec2.xml`	`ec2`	Uses TCP for transport and `S3_PING` for discovery. Suitable for Amazon EC2 nodes where UDP multicast is not available.
`default-jgroups-kubernetes.xml`	`kubernetes`	Uses TCP for transport and `DNS_PING` for discovery. Suitable for Kubernetes and Red Hat OpenShift nodes where UDP multicast is not always available.
`default-jgroups-google.xml`	`google`	Uses TCP for transport and `GOOGLE_PING2` for discovery. Suitable for Google Cloud Platform nodes where UDP multicast is not available.
`default-jgroups-azure.xml`	`azure`	Uses TCP for transport and `AZURE_PING` for discovery. Suitable for Microsoft Azure nodes where UDP multicast is not available.

Reference

7.2. Using Inline JGroups Stacks

Custom JGroups stacks can help you optimize network performance for Data Grid clusters compared to using the default stacks.

Procedure

Embed your custom JGroups stack definitions in infinispan.xml as in the following example:

<infinispan> 1
  <jgroups> 2
    <stack name="prod"> 3
      <TCP bind_port="7800" port_range="30" recv_buf_size="20000000" send_buf_size="640000"/>
      <MPING bind_addr="127.0.0.1" break_on_coord_rsp="true"
             mcast_addr="${jgroups.mping.mcast_addr:228.2.4.6}"
             mcast_port="${jgroups.mping.mcast_port:43366}"
             num_discovery_runs="3"
             ip_ttl="${jgroups.udp.ip_ttl:2}"/>
      <MERGE3 />
      <FD_SOCK />
      <FD_ALL timeout="3000" interval="1000" timeout_check_interval="1000" />
      <VERIFY_SUSPECT timeout="1000" />
      <pbcast.NAKACK2 use_mcast_xmit="false" xmit_interval="100" xmit_table_num_rows="50"
                      xmit_table_msgs_per_row="1024" xmit_table_max_compaction_time="30000" />
      <UNICAST3 xmit_interval="100" xmit_table_num_rows="50" xmit_table_msgs_per_row="1024"
                xmit_table_max_compaction_time="30000" />
      <pbcast.STABLE stability_delay="200" desired_avg_gossip="2000" max_bytes="1M" />
      <pbcast.GMS print_local_addr="false" join_timeout="${jgroups.join_timeout:2000}" />
      <UFC max_credits="4m" min_threshold="0.40" />
      <MFC max_credits="4m" min_threshold="0.40" />
      <FRAG3 />
    </stack>
  </jgroups>
  <cache-container default-cache="replicatedCache"> 4
    <transport stack="prod" /> 5
    ...
  </cache-container>
</infinispan>

1: root element of infinispan.xml.
2: contains JGroups stack definitions.
3: defines a JGroups stack named "prod".
4: configures a Data Grid Cache Manager and names the "replicatedCache" cache definition as the default.
5: uses the "prod" JGroups stack for cluster transport.

Tip

Use inheritance with inline JGroups stacks to tune and customize specific transport properties.

Reference

7.3. Adjusting and Tuning JGroups Stacks

Use inheritance to combine, extend, remove, and replace specific properties in the default JGroups stacks or custom configurations.

Procedure

Add a new JGroups stack declaration.
Name a parent stack with the extends attribute.
Tip
Use the default JGroups stacks as parents. For example, extends="tcp" to tunes the default TCP stack.
Modify transport properties with the stack.combine attribute.

7.3.1. Stack Combine Attribute

stack.combine modifies inherited JGroups properties.

stack.position identifies protocols to modify. If you do not specify stack.position, Data Grid defaults to the same protocol as the inherited configuration, which resets all non-specified attributes to the default values.

Value	Description
`COMBINE`	Overrides existing protocol attributes.
`REPLACE`	Replaces existing protocols.
`INSERT_AFTER`	Inserts protocols into the JGroups stack.
`REMOVE`	Removes protocols from the inherited configuration.

For example, evaluate a Gossip router for cluster discovery using a TCP stack configuration named "prod":

<jgroups>
  ...
  <stack name="gossip-prod" extends="prod"> 1
    <TCPGOSSIP initial_hosts="${jgroups.tunnel.gossip_router_hosts:localhost[12001]}"
             stack.combine="REPLACE" stack.position="MPING" /> 2
    <FD_SOCK stack.combine="REMOVE"/> 3
    <VERIFY_SUSPECT timeout="2000"/> 4
    <SYM_ENCRYPT sym_algorithm="AES"
                 key_store_name="defaultStore.keystore"
                 store_password="changeit"
                 alias="myKey" stack.combine="INSERT_AFTER"
                 stack.position="pbcast.NAKACK2" /> 5
  </stack>
  ...
</jgroups>

1: creates a new stack named "gossip-prod" that inherits from "prod".
2: uses TCPGOSSIP discovery instead of MPING.
3: removes FD_SOCK from the "gossip-prod" stack.
4: increases the VERIFY_SUSPECT timeout.
5: adds SYM_ENCRYPT for cluster security to the stack after the pbcast.NAKACK2 protocol.

7.4. Using JGroups Stacks in External Files

You can use custom JGroups transport configuration from external files.

Data Grid looks for JGroups configuration files on your classpath first and then for absolute path names.

Declaratively

Specify your JGroups transport configuration with the stack-file element.

<infinispan>
  <jgroups>
     <stack-file name="prod-tcp" path="prod-jgroups-tcp.xml"/> 1
  </jgroups>
  <cache-container default-cache="replicatedCache">
    <transport stack="prod-tcp" /> 2
    <replicated-cache name="replicatedCache"/>
  </cache-container>
  ...
</infinispan>

1: adds the "prod-jgroups-tcp.xml" stack definition.
2: configures Data Grid cluster to use the stack.

Programmatically

Specify your JGroups transport configuration with the addProperty() method.

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder().transport()
        .defaultTransport()
        .clusterName("prod-cluster")
        .addProperty("configurationFile", "prod-jgroups-tcp.xml") 1
        .build();

1: adds the "prod-jgroups-tcp.xml" stack definition as the default for the cluster transport.

Reference

7.5. Tuning JGroups Stacks with System Properties

Pass system properties to the JVM at startup to tune JGroups stacks.

For example, to change the TCP port and IP address do the following:

$ java -cp ... -Djgroups.tcp.port=1234 -Djgroups.tcp.address=192.0.2.0

7.5.1. System Properties for Default JGroups Stacks

Use system properties with default JGroups stacks.

Common Properties

The following system properties apply to all JGroups stacks, including default-jgroups-udp.xml and default-jgroups-tcp.xml:

System Property	Description	Default Value	Required/Optional
`jgroups.bind.address`	Bind address for cluster transport.	`SITE_LOCAL`	Optional
`jgroups.bind.port`	Bind port for the socket.	`7800`	Optional
`jgroups.mcast_addr`	IP address for multicast, both discovery and inter-cluster communication. The IP address must be a valid "class D" address that is suitable for IP multicast.	`228.6.7.8`	Optional
`jgroups.mcast_port`	Port for the multicast socket.	`46655`	Optional
`jgroups.ip_ttl`	Time-to-live (TTL) for IP multicast packets. The value defines the number of network hops a packet can make before it is dropped.	2	Optional
`jgroups.thread_pool.min_threads`	Minimum number of threads for the thread pool.	0	Optional
`jgroups.thread_pool.max_threads`	Maximum number of threads for the thread pool.	200	Optional
`jgroups.join_timeout`	Maximum number of milliseconds to wait for join requests to succeed.	2000	Optional

Amazon EC3

The following system properties apply to default-jgroups-ec2.xml:

System Property	Description	Default Value	Required/Optional
`jgroups.s3.access_key`	Amazon S3 access key for an S3 bucket.	No default value.	Optional
`jgroups.s3.secret_access_key`	Amazon S3 secret key used for an S3 bucket.	No default value.	Optional
`jgroups.s3.bucket`	Name of the Amazon S3 bucket. The name must exist and be unique.	No default value.	Optional

Kubernetes

The following system properties apply to default-jgroups-kubernetes.xml:

System Property	Description	Default Value	Required/Optional
`jgroups.dns.query`	Sets the DNS record that returns cluster members.	N/A	Required if you do not set the `dns_query` parameter.

Google Cloud Platform

The following system properties apply to default-jgroups-google.xml:

System Property	Description	Default Value	Required/Optional
`jgroups.google.bucket_name`	Name of the Google Compute Engine bucket. The name must exist and be unique.	N/A	Required if you do not set the `dns_query` parameter.

Reference

7.1. Getting Started with Default Stacks

7.1.1. Default JGroups Stacks

7.2. Using Inline JGroups Stacks

7.3. Adjusting and Tuning JGroups Stacks

7.3.1. Stack Combine Attribute

7.4. Using JGroups Stacks in External Files

7.5. Tuning JGroups Stacks with System Properties

7.5.1. System Properties for Default JGroups Stacks

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links