Chapter 6. Clustering in Web Applications

JBoss EAP 7 introduced a refined public clustering API for use by applications. The new services are designed to be lightweight, easily injectable, with no external dependencies.

A clustered singleton service, also known as a high-availability (HA) singleton, is a service deployed on multiple nodes in a cluster. The service is provided on only one of the nodes. The node running the singleton service is usually called the master node.

When the master node either fails or shuts down, another master is selected from the remaining nodes and the service is restarted on the new master. Other than a brief interval when one master has stopped and another has yet to take over, the service is provided by one, and only one, node.

HA Singleton ServiceBuilder API

JBoss EAP 7 introduced a new public API for building singleton services that simplifies the process significantly.

The SingletonServiceConfigurator implementation installs its services so they will start asynchronously, preventing deadlocking of the Modular Service Container (MSC).

HA Singleton Service Election Policies

If there is a preference for which node should start the HA singleton, you can set the election policy in the ServiceActivator class.

JBoss EAP provides two election policies:

HA Singleton Service Preferences

An HA singleton service election policy may optionally specify one or more preferred servers. This preferred server, when available, will be the master for all singleton applications under that policy.

You can define the preferences either through the node name or through the outbound socket binding name.

By default, JBoss EAP high availability configurations provide a simple election policy named default with no preferred server. You can set the preference by creating a custom policy and defining the preferred server.

Quorum

A potential issue with a singleton service arises when there is a network partition. In this situation, also known as the split-brain scenario, subsets of nodes cannot communicate with each other. Each set of servers consider all servers from the other set failed and continue to work as the surviving cluster. This might result in data consistency issues.

JBoss EAP allows you to specify a quorum in the election policy to prevent the split-brain scenario. The quorum specifies a minimum number of nodes to be present before a singleton provider election can take place.

A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any active singleton services.

HA Singleton Service Election Listener

After electing a new primary singleton service provider, any registered SingletonElectionListener is triggered, notifying every member of the cluster about the new primary provider. The following example illustrates the usage of SingletonElectionListener:

public class MySingletonElectionListener implements SingletonElectionListener {
    @Override
    public void elected(List<Node> candidates, Node primary) {
        // ...
    }
}

public class MyServiceActivator implements ServiceActivator {
    @Override
    public void activate(ServiceActivatorContext context) {
        String containerName = "foo";
        SingletonElectionPolicy policy = new MySingletonElectionPolicy();
        SingletonElectionListener listener = new MySingletonElectionListener();
        int quorum = 3;
        ServiceName name = ServiceName.parse("my.service.name");
        // Use a SingletonServiceConfiguratorFactory backed by default cache of "foo" container
        Supplier<SingletonServiceConfiguratorFactory> factory = new ActiveServiceSupplier<>(context.getServiceTarget(), ServiceName.parse(SingletonDefaultCacheRequirement.SINGLETON_SERVICE_CONFIGURATOR_FACTORY.resolve(containerName).getName()));
        ServiceBuilder<?> builder = factory.get().createSingletonServiceConfigurator(name)
                .electionListener(listener)
                .electionPolicy(policy)
                .requireQuorum(quorum)
                .build(context.getServiceTarget());
        Service service = new MyService();
        builder.setInstance(service).install();
    }
}

public class MySingletonElectionListener implements SingletonElectionListener {
    @Override
    public void elected(List<Node> candidates, Node primary) {
        // ...
    }
}

public class MyServiceActivator implements ServiceActivator {
    @Override
    public void activate(ServiceActivatorContext context) {
        String containerName = "foo";
        SingletonElectionPolicy policy = new MySingletonElectionPolicy();
        SingletonElectionListener listener = new MySingletonElectionListener();
        int quorum = 3;
        ServiceName name = ServiceName.parse("my.service.name");
        // Use a SingletonServiceConfiguratorFactory backed by default cache of "foo" container
        Supplier<SingletonServiceConfiguratorFactory> factory = new ActiveServiceSupplier<>(context.getServiceTarget(), ServiceName.parse(SingletonDefaultCacheRequirement.SINGLETON_SERVICE_CONFIGURATOR_FACTORY.resolve(containerName).getName()));
        ServiceBuilder<?> builder = factory.get().createSingletonServiceConfigurator(name)
                .electionListener(listener)
                .electionPolicy(policy)
                .requireQuorum(quorum)
                .build(context.getServiceTarget());
        Service service = new MyService();
        builder.setInstance(service).install();
    }
}

Create an HA Singleton Service Application

The following is an abbreviated example of the steps required to create and deploy an application as a cluster-wide singleton service. This example demonstrates a querying service that regularly queries a singleton service to get the name of the node on which it is running.

To see the singleton behavior, you must deploy the application to at least two servers. It is transparent whether the singleton service is running on the same node or whether the value is obtained remotely.

See the ha-singleton-service quickstart that ships with JBoss EAP for the complete working example. This quickstart also provides a second example that demonstrates a singleton service that is installed with a backup service. The backup service is running on all nodes that are not elected to be running the singleton service. Finally, this quickstart also demonstrates how to configure a few different election policies.

JBoss EAP 7 adds the ability to deploy a given application as a singleton deployment.

When deployed to a group of clustered servers, a singleton deployment will only deploy on a single node at any given time. If the node on which the deployment is active stops or fails, the deployment will automatically start on another node.

The policies for controlling HA singleton behavior are managed by a new singleton subsystem. A deployment can either specify a specific singleton policy or use the default subsystem policy. A deployment identifies itself as a singleton deployment by using a META-INF/singleton-deployment.xml deployment descriptor, which is most easily applied to an existing deployment as a deployment overlay. Alternatively, the requisite singleton configuration can be embedded within an existing jboss-all.xml file.

Defining or Choosing a Singleton Deployment

Creating a Singleton Deployment

JBoss EAP provides two election policies:

Preferences

Additionally, any singleton election policy can indicate a preference for one or more members of a cluster. Preferences can be defined either by using the node name or by using the outbound socket binding name. Node preferences always take precedent over the results of an election policy.

/subsystem=singleton/singleton-policy=foo/election-policy=simple:list-add(name=name-preferences, value=nodeA)

/subsystem=singleton/singleton-policy=bar/election-policy=random:list-add(name=socket-binding-preferences, value=binding1)

/subsystem=singleton/singleton-policy=foo/election-policy=simple:list-add(name=name-preferences, value=nodeA)

/subsystem=singleton/singleton-policy=bar/election-policy=random:list-add(name=socket-binding-preferences, value=binding1)

batch
/subsystem=singleton/singleton-policy=my-new-policy:add(cache-container=server)
/subsystem=singleton/singleton-policy=my-new-policy/election-policy=simple:add(name-preferences=[node1, node2, node3, node4])
run-batch

batch
/subsystem=singleton/singleton-policy=my-new-policy:add(cache-container=server)
/subsystem=singleton/singleton-policy=my-new-policy/election-policy=simple:add(name-preferences=[node1, node2, node3, node4])
run-batch

/subsystem=singleton:write-attribute(name=default, value=my-new-policy)

/subsystem=singleton:write-attribute(name=default, value=my-new-policy)

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="my-other-new-policy">
      <singleton-policy name="my-other-new-policy" cache-container="server">
         <random-election-policy>
            <socket-binding-preferences>binding1 binding2 binding3 binding4</socket-binding-preferences>
         </random-election-policy>
      </singleton-policy>
   </singleton-policies>
</subsystem>

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="my-other-new-policy">
      <singleton-policy name="my-other-new-policy" cache-container="server">
         <random-election-policy>
            <socket-binding-preferences>binding1 binding2 binding3 binding4</socket-binding-preferences>
         </random-election-policy>
      </singleton-policy>
   </singleton-policies>
</subsystem>

Define a Quorum

Network partitions are particularly problematic for singleton deployments, since they can trigger multiple singleton providers for the same deployment to run at the same time. To defend against this scenario, a singleton policy can define a quorum that requires a minimum number of nodes to be present before a singleton provider election can take place. A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any singleton deployments using the respective singleton policy.

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="default">
      <singleton-policy name="default" cache-container="server" quorum="4">
         <simple-election-policy/>
      </singleton-policy>
   </singleton-policies>
</subsystem>

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="default">
      <singleton-policy name="default" cache-container="server" quorum="4">
         <simple-election-policy/>
      </singleton-policy>
   </singleton-policies>
</subsystem>

/subsystem=singleton/singleton-policy=foo:write-attribute(name=quorum, value=3)

/subsystem=singleton/singleton-policy=foo:write-attribute(name=quorum, value=3)

See the ha-singleton-deployment quickstart that ships with JBoss EAP for a complete working example of a service packaged in an application as a cluster-wide singleton using singleton deployments.

Determine the Primary Singleton Service Provider Using the CLI

The singleton subsystem exposes a runtime resource for each singleton deployment or service created from a particular singleton policy. This helps you determine the primary singleton provider using the CLI.

You can view the name of the cluster member currently acting as the singleton provider. For example:

/subsystem=singleton/singleton-policy=default/deployment=singleton.jar:read-attribute(name=primary-provider)
{
    "outcome" => "success",
    "result" => "node1"
}

/subsystem=singleton/singleton-policy=default/deployment=singleton.jar:read-attribute(name=primary-provider)
{
    "outcome" => "success",
    "result" => "node1"
}

You can also view the names of the nodes on which the singleton deployment or service is installed. For example:

/subsystem=singleton/singleton-policy=default/deployment=singleton.jar:read-attribute(name=providers)
{
    "outcome" => "success",
    "result" => [
        "node1",
        "node2"
    ]
}

/subsystem=singleton/singleton-policy=default/deployment=singleton.jar:read-attribute(name=providers)
{
    "outcome" => "success",
    "result" => [
        "node1",
        "node2"
    ]
}

The distributable-web subsystem facilitates flexible and distributable web session configurations. The subsystem defines a set of distributable web session management profiles. One of these profiles is designated as the default profile. It defines the default behavior of a distributable web application. For example:

[standalone@embedded /] /subsystem=distributable-web:read-attribute(name=default-session-management)
{
    "outcome" => "success",
    "result" => "default"
}

[standalone@embedded /] /subsystem=distributable-web:read-attribute(name=default-session-management)
{
    "outcome" => "success",
    "result" => "default"
}

The default session management stores web session data within an Infinispan cache as the following example illustrates:

[standalone@embedded /] /subsystem=distributable-web/infinispan-session-management=default:read-resource
{
    "outcome" => "success",
    "result" => {
        "cache" => undefined,
        "cache-container" => "web",
        "granularity" => "SESSION",
        "affinity" => {"primary-owner" => undefined}
    }
}

[standalone@embedded /] /subsystem=distributable-web/infinispan-session-management=default:read-resource
{
    "outcome" => "success",
    "result" => {
        "cache" => undefined,
        "cache-container" => "web",
        "granularity" => "SESSION",
        "affinity" => {"primary-owner" => undefined}
    }
}

The attributes used in this example and the possible values are:

You must enable ranked session affinity in your load balancer to have session affinity with multiple, ordered routes. For more information, see Enabling Ranked Session Affinity in Your Load Balancer in the Configuration Guide for JBoss EAP.

You can override the default distributable session management behavior by referencing a session management profile by name or by providing a deployment-specific session management configuration. For more information, see Overide Default Distributable Session Management Behavior.

[standalone@embedded /]/subsystem=distributable-web/hotrod-session-management=ExampleRemoteSessionStore:add(remote-cache-container=datagrid, cache-configuration=__REMOTE_CACHE_CONFIG_NAME__, granularity=ATTRIBUTE)
{
    "outcome" => "success"
}

[standalone@embedded /]/subsystem=distributable-web/hotrod-session-management=ExampleRemoteSessionStore:add(remote-cache-container=datagrid, cache-configuration=__REMOTE_CACHE_CONFIG_NAME__, granularity=ATTRIBUTE)
{
    "outcome" => "success"
}