Chapter 6. Clustering in Web Applications

JBoss EAP 7 introduces a refined public clustering API for use by applications. The new services are designed to be lightweight, easily injectable, with no external dependencies.

A clustered singleton service, also known as a high-availability (HA) singleton, is a service deployed on multiple nodes in a cluster. The service is provided on only one of the nodes. The node running the singleton service is usually called the master node.

When the master node either fails or shuts down, another master is selected from the remaining nodes and the service is restarted on the new master. Other than a brief interval when one master has stopped and another has yet to take over, the service is provided by one, and only one, node.

HA Singleton ServiceBuilder API

JBoss EAP 7 introduces a new public API for building singleton services that simplifies the process significantly.

The SingletonServiceBuilder implementation installs its services so they will start asynchronously, preventing deadlocking of the Modular Service Container (MSC).

HA Singleton Service Election Policies

If there is a preference for which node should start the HA singleton, you can set the election policy in the ServiceActivator class.

JBoss EAP provides two election policies:

HA Singleton Service Preferences

An HA singleton service election policy may optionally specify one or more preferred servers. This preferred server, when available, will be the master for all singleton applications under that policy.

You can define the preferences either through the node name or through the outbound socket binding name.

By default, JBoss EAP high availability configurations provide a simple election policy named default with no preferred server. You can set the preference by creating a custom policy and defining the preferred server.

Quorum

A potential issue with a singleton service arises when there is a network partition. In this situation, also known as the split-brain scenario, subsets of nodes cannot communicate with each other. Each set of servers consider all servers from the other set failed and continue to work as the surviving cluster. This might result in data consistency issues.

JBoss EAP allows you to specify a quorum in the election policy to prevent the split-brain scenario. The quorum specifies a minimum number of nodes to be present before a singleton provider election can take place.

A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any active singleton services.

Create an HA Singleton Service Application

The following is an abbreviated example of the steps required to create and deploy an application as a cluster-wide singleton service. This example demonstrates a querying service that regularly queries a singleton service to get the name of the node on which it is running.

To see the singleton behavior, you must deploy the application to at least two servers. It is transparent whether the singleton service is running on the same node or whether the value is obtained remotely.

See the ha-singleton-service quickstart that ships with JBoss EAP for the complete working example. This quickstart also provides a second example that demonstrates a singleton service that is installed with a backup service. The backup service is running on all nodes that are not elected to be running the singleton service. Finally, this quickstart also demonstrates how to configure a few different election policies.

JBoss EAP 7 adds the ability to deploy a given application as a singleton deployment.

When deployed to a group of clustered servers, a singleton deployment will only deploy on a single node at any given time. If the node on which the deployment is active stops or fails, the deployment will automatically start on another node.

The policies for controlling HA singleton behavior are managed by a new singleton subsystem. A deployment can either specify a specific singleton policy or use the default subsystem policy. A deployment identifies itself as a singleton deployment by using a META-INF/singleton-deployment.xml deployment descriptor, which is most easily applied to an existing deployment as a deployment overlay. Alternatively, the requisite singleton configuration can be embedded within an existing jboss-all.xml file.

Defining or Choosing a Singleton Deployment

Creating a Singleton Deployment

JBoss EAP provides two election policies:

Preferences

Additionally, any singleton election policy can indicate a preference for one or more members of a cluster. Preferences can be defined either by using the node name or by using the outbound socket binding name. Node preferences always take precedent over the results of an election policy.

/subsystem=singleton/singleton-policy=foo/election-policy=simple:list-add(name=name-preferences, value=nodeA)

/subsystem=singleton/singleton-policy=bar/election-policy=random:list-add(name=socket-binding-preferences, value=binding1)

batch
/subsystem=singleton/singleton-policy=my-new-policy:add(cache-container=server)
/subsystem=singleton/singleton-policy=my-new-policy/election-policy=simple:add(name-preferences=[node1, node2, node3, node4])
run-batch

/subsystem=singleton:write-attribute(name=default, value=my-new-policy)

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="my-other-new-policy">
      <singleton-policy name="my-other-new-policy" cache-container="server">
         <random-election-policy>
            <socket-binding-preferences>binding1 binding2 binding3 binding4</socket-binding-preferences>
         </random-election-policy>
      </singleton-policy>
   </singleton-policies>
</subsystem>

Define a Quorum

Network partitions are particularly problematic for singleton deployments, since they can trigger multiple singleton providers for the same deployment to run at the same time. To defend against this scenario, a singleton policy can define a quorum that requires a minimum number of nodes to be present before a singleton provider election can take place. A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any singleton deployments using the respective singleton policy.

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="default">
      <singleton-policy name="default" cache-container="server" quorum="4">
         <simple-election-policy/>
      </singleton-policy>
   </singleton-policies>
</subsystem>

/subsystem=singleton/singleton-policy=foo:write-attribute(name=quorum, value=3)

See the ha-singleton-deployment quickstart that ships with JBoss EAP for a complete working example of a service packaged in an application as a cluster-wide singleton using singleton deployments.