Chapter 10. Setting up cross-site replication
Ensure service availability with Data Grid Operator by configuring cross-site replication to back up data between Data Grid clusters.
10.1. Using Data Grid Operator to manage cross-site connections
Data Grid Operator in one data center can discover a Data Grid cluster that Data Grid Operator manages in another data center. This discovery allows Data Grid to automatically form cross-site views and create global clusters.
The following illustration provides an example in which Data Grid Operator manages a Data Grid cluster at a data center in New York City, NYC. At another data center in London, LON, Data Grid Operator also manages a Data Grid cluster.
Data Grid Operator uses the Kubernetes API to establish a secure connection between the OpenShift Container Platform clusters in NYC and LON. Data Grid Operator then creates a cross-site replication service so Data Grid clusters can back up data across locations.
Data Grid Operator in each OpenShift cluster must have network access to the remote Kubernetes API.
When you configure automatic connections, Data Grid clusters do not start running until Data Grid Operator discovers all backup locations in the configuration.
Each Data Grid cluster has one site master node that coordinates all backup requests. Data Grid Operator identifies the site master node so that all traffic through the cross-site replication service goes to the site master.
If the current site master node goes offline then a new node becomes site master. Data Grid Operator automatically finds the new site master node and updates the cross-site replication service to forward backup requests to it.
10.1.1. Creating service account tokens
Generate service account tokens on each OpenShift cluster that acts as a backup location. Clusters use these tokens to authenticate with each other so Data Grid Operator can create a cross-site replication service.
Procedure
- Log in to an OpenShift cluster.
Create a service account.
For example, create a service account at LON:
$ oc create sa lon serviceaccount/lon created
Add the view role to the service account with the following command:
$ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
If you use a node port service to expose Data Grid clusters on the network, you must also add the
cluster-reader
role to the service account:$ oc adm policy add-cluster-role-to-user cluster-reader -z <service-account-name> -n <namespace>
- Repeat the preceding steps on your other OpenShift clusters.
Additional resources
10.1.2. Exchanging service account tokens
After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.
Prerequisites
Get tokens from each service account.
Use the following command or get the token from the OpenShift Web Console:
$ oc sa get-token lon eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
Procedure
- Log in to an OpenShift cluster.
Add the service account token for a backup location with the following command:
$ oc create secret generic <token-name> --from-literal=token=<token>
For example, log in to the OpenShift cluster at NYC and create a
lon-token
secret as follows:$ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
- Repeat the preceding steps on your other OpenShift clusters.
10.1.3. Configuring Data Grid Operator to handle cross-site connections
Configure Data Grid Operator to establish cross-site views with Data Grid clusters.
Prerequisites
- Create secrets that contain service account tokens for each backup location.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set the value of the
spec.service.sites.local.expose.type
field to eitherNodePort
orLoadBalancer
. Optionally configure ports with the following fields:
-
spec.service.sites.local.expose.nodePort
if you useNodePort
. -
spec.service.sites.local.expose.port
if you useLoadBalancer
.
-
-
Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
. If Data Grid cluster names or namespaces at the remote site do not match the local site, specify those values with the
clusterName
andnamespace
fields.The following are example
Infinispan
CR definitions for LON and NYC:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443 secretName: nyc-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: nyc-cluster spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 locations: - name: LON clusterName: example-infinispan namespace: rhdg-namespace url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443 secretName: lon-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
10.1.4. Resources for managed cross-site connections
This topic describes resources for cross-site connections that Data Grid Operator manages.
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.site-b.devcluster.openshift.com:6443 secretName: nyc-token
Field | Description |
---|---|
| Data Grid supports cross-site replication with Data Grid service clusters only. |
| Names the local site where a Data Grid cluster runs. |
|
Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to |
|
Specifies a static port within the default range of |
|
Specifies the network port for the service if you expose Data Grid through a |
| Provides connection information for all backup locations. |
|
Specifies a backup location that matches |
| Specifies the URL of the Kubernetes API for the backup location. |
| Specifies the secret that contains the service account token for the backup site. |
| Specifies the cluster name at the backup location if it is different to the cluster name at the local site. |
| Specifies the namespace of the Data Grid cluster at the backup location if it does not match the namespace at the local site. |
10.2. Manually connecting Data Grid clusters
You can specify static network connection details to perform cross-site replication with Data Grid clusters running outside OpenShift. Manual cross-site connections are necessary in any scenario where access to the Kubernetes API is not available outside the OpenShift cluster where Data Grid runs.
You can use both automatic and manual connections for Data Grid clusters in the same Infinispan
CR. However, you must ensure that Data Grid clusters establish connections in the same way at each site.
Prerequisites
Manually connecting Data Grid clusters to form cross-site views requires predictable network locations for Data Grid services.
You need to know the network locations before they are created, which requires you to:
- Have the host names and ports for each Data Grid cluster that you plan to configure as a backup location.
-
Have the host name of the
<cluster-name>-site
service for any remote Data Grid cluster that is running on OpenShift.
You must use the<cluster-name>-site
service to form a cross-site view between a cluster that Data Grid Operator manages and any other cluster.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set the value of the
spec.service.sites.local.expose.type
field to eitherNodePort
orLoadBalancer
. Optionally configure ports with the following fields:
-
spec.service.sites.local.expose.nodePort
if you useNodePort
. -
spec.service.sites.local.expose.port
if you useLoadBalancer
.
-
Provide the name and static URL for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
, for example:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900 logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-infinispan spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 locations: - name: LON url: infinispan+xsite://infinispan-lon.myhost.com logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
10.2.1. Resources for manual cross-site connections
This topic describes resources for cross-site connections that you maintain manually.
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900
Field | Description |
---|---|
| Data Grid supports cross-site replication with Data Grid service clusters only. |
| Names the local site where a Data Grid cluster runs. |
|
Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to |
|
Specifies a static port within the default range of |
|
Specifies the network port for the service if you expose Data Grid through a |
| Provides connection information for all backup locations. |
|
Specifies a backup location that matches |
|
Specifies the static URL for the backup location in the format of |
10.3. Configuring sites in the same OpenShift cluster
For evaluation and demonstration purposes, you can configure Data Grid to back up between nodes in the same OpenShift cluster.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set
ClusterIP
as the value of thespec.service.sites.local.expose.type
field. -
Provide the name of the Data Grid cluster that acts as a backup location with
spec.service.sites.locations.clusterName
. If both Data Grid clusters have the same name, specify the namespace of the backup location with
spec.service.sites.locations.namespace
.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-clustera spec: replicas: 1 expose: type: LoadBalancer service: type: DataGrid sites: local: name: SiteA expose: type: ClusterIP locations: - name: SiteB clusterName: example-clusterb namespace: cluster-namespace
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.$ oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.