Chapter 1. Cross-site replication

Data Grid can back up data between clusters running in geographically dispersed data centers and across different cloud providers. Cross-site replication provides Data Grid with a global cluster view and:

Figure 1.1. Cross-site replication

Relay nodes are the nodes in Data Grid clusters that are responsible for sending and receiving requests from backup locations. If a node is not a relay node, it must forward backup requests to a local relay node.

For optimal performance, you should configure all nodes as relay nodes. This increases the speed of backup requests because each node in the cluster can backup to remote clusters directly without having to forward backup requests to local relay nodes.

Data Grid caches include a backups configuration that let you name remote sites as backup locations.

For example, the following diagram shows three caches, "customers", "eu-orders", and "us-orders":

Data Grid replicates data between clusters at the same time that writes to caches occur. For example, if a client writes "k1" to LON, Data Grid backs up "k1" to NYC at the same time.

To back up data to a different cluster, Data Grid can use either a synchronous or asynchronous strategy.

Synchronous strategy

When Data Grid replicates data to backup locations, it writes to the cache on the local cluster and the cache on the remote cluster concurrently. With the synchronous strategy, Data Grid waits for both write operations to complete before returning.

You can control how Data Grid handles writes to the local cluster’s cache if backup operations fail. Data Grid can take the following actions:

Synchronous backups also support two-phase commits for caches that participate in optimistic transactions. The first phase acquires the locks, and the second phase commits the modifications.

Asynchronous strategy

When Data Grid replicates data to backup locations, it does not wait for the write operation to complete before writing to the local cache.

Asynchronous backup write operations to the local cache are independent of each other. If a backup write operation fails, the local cache continue and no exceptions are thrown. When this happens, Data Grid also retries the failed backup operation until the remote cluster disconnects from the cross-site view.

Synchronous vs asynchronous backups

Synchronous backups offer the strongest guarantee of data consistency across clusters. If strategy=sync, when cache.put() calls return you know the value is up to date in the local cache and in the backup locations.

The trade-off for this consistency is performance. Synchronous backups have much greater latency in comparison to asynchronous backups.

Asynchronous backups, on the other hand, do not add latency to client requests, so they have no performance impact. However, if strategy=async, when cache.put() calls return you cannot be sure of that the value in the backup location is the same as in the local cache.

To avoid wasting resources Data Grid can take backup locations offline when they stop accepting requests after a specific period of time.

Data Grid takes remote clusters offline based on the number of failed sequential requests and/or the time interval since the first failure. Requests fail when a response is not received or a connection between cluster is down.

Backup timeouts

Backup configurations include timeout values for operations to replicate data between clusters. If operations do not complete before the timeout expires, Data Grid records them as failures.

In the following example, operations to replicate data to NYC are recorded as failures if they do not complete after 10 seconds:

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s"/>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s"/>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000"
            }
          }
        }
      }
    }
  }
}

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000"
            }
          }
        }
      }
    }
  }
}

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"

Number of failures

You can specify the number of consecutive failures that can occur before backup locations go offline.

In the following example, if a cluster attempts to replicate data to NYC and five consecutive operations fail, NYC automatically goes offline:

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s">
               <take-offline after-failures="5"/>
            </backup>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s">
               <take-offline after-failures="5"/>
            </backup>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000",
              "take-offline": {
                "after-failures": "5"
              }
            }
          }
        }
      }
    }
  }
}

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000",
              "take-offline": {
                "after-failures": "5"
              }
            }
          }
        }
      }
    }
  }
}

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"
            takeOffline:
              afterFailures: "5"

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"
            takeOffline:
              afterFailures: "5"

Time to wait

You can also specify how long to wait before taking backup cluster offline following a failure. If a backup request succeeds before the wait time elapses, Data Grid does not take the backup location offline. The elapsed time is checked only after a write operation. Therefore, if the cluster has no load, the backup location will not transition to the offline state.

One to two minutes is generally a suitable wait time before automatically taking backup locations offline. If the wait period is too short, backup locations might go offline too soon. You then need to bring the clusters back online and perform state transfer operations to ensure data synchronization.

In the following example, if a cluster attempts to replicate data to NYC and there are more than five consecutive failures and 15 seconds elapse after the initial failure, NYC automatically goes offline:

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s">
               <take-offline after-failures="5"
                             min-wait="15000"/>
            </backup>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

<infinispan>
   <cache-container>
      <distributed-cache name="customers">
         <backups>
            <backup site="NYC"
                    strategy="ASYNC"
                    timeout="10s">
               <take-offline after-failures="5"
                             min-wait="15000"/>
            </backup>
         </backups>
      </distributed-cache>
   </cache-container>
</infinispan>

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000",
              "take-offline": {
                "after-failures": "5",
                "min-wait": "15000"
              }
            }
          }
        }
      }
    }
  }
}

{
  "infinispan": {
    "cache-container": {
      "distributed-cache": {
        "name": "customers",
        "backups": {
          "NYC": {
            "backup": {
              "strategy": "ASYNC",
              "timeout": "10000",
              "take-offline": {
                "after-failures": "5",
                "min-wait": "15000"
              }
            }
          }
        }
      }
    }
  }
}

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"
            takeOffline:
              afterFailures: "5"
              minWait: "15000"

infinispan:
  cacheContainer:
    distributedCache:
      name: "customers"
      backups:
        NYC:
          backup:
            strategy: "ASYNC"
            timeout: "10000"
            takeOffline:
              afterFailures: "5"
              minWait: "15000"

State transfer is an administrative operation that synchronizes data between clusters.

For example, LON goes offline and NYC starts handling client requests. When you bring LON back online, the Data Grid cluster in LON does not have the same data as the cluster in NYC.

To ensure the data is consistent between LON and NYC, you can push state from NYC to LON.

Automatic state transfer

By default, you must manually perform cross-site state transfer operations with the CLI or via JMX or REST.

However, when using the asynchronous backup strategy, Data Grid can automatically perform cross-site state transfer operations.

When a backup location comes back online, and the network connection is stable, Data Grid initiates bidirectional state transfer between backup locations. For example, Data Grid simultaneously transfers state from LON to NYC and NYC to LON.

The automatic state transfer is also triggered when a cache starts.

In the scenario where LON is starting up, after a cache starts, it sends a notification to NYC. Following this, NYC starts a unidirectional state transfer to LON.

Clients can write to Data Grid clusters using either an Active/Passive or Active/Active configuration. Data Grid does not provide specific configuration for each mode. To achieve Active/Passive, the client, application, or load balancer must be configured to select a primary cluster.

Active/Passive

The following diagram illustrates Active/Passive where Data Grid handles client requests from one cluster only:

In the preceding image:

With Active/Passive, NYC provides data redundancy. If the Data Grid cluster at LON goes offline for any reason, clients can start sending requests to NYC. When you bring LON back online you can synchronize data with NYC and then switch clients back to LON.

Active/Active

The following diagram illustrates Active/Active where Data Grid handles client requests at two sites:

In the preceding image:

With Active/Active both NYC and LON replicate data to remote caches while handling client requests. If either NYC or LON go offline, clients can start sending requests to the online site. You can then bring offline sites back online, push state to synchronize data, and switch clients as required.

Backup strategies and client connections

If multiple clients attempt to write to the same entry concurrently, and the backup strategy is synchronous (strategy=sync), then deadlocks occur. However, you can use the synchronous backup strategy with an Active/Passive configuration if both sites access different data sets, in which case there is no risk of deadlocks from concurrent writes.

            LON       NYC

k1=(n/a)    0,0       0,0

k1=2        1,0  -->  1,0   k1=2

k1=3        1,1  <--  1,1   k1=3

k1=5        2,1       1,2   k1=8

                 -->  2,1 (conflict)
(conflict)  1,2  <--

            LON       NYC

k1=(n/a)    0,0       0,0

k1=2        1,0  -->  1,0   k1=2

k1=3        1,1  <--  1,1   k1=3

k1=5        2,1       1,2   k1=8

                 -->  2,1 (conflict)
(conflict)  1,2  <--

Expiration removes cache entries based on time. Data Grid provides two ways to configure expiration for entries: