Chapter 13. Setting up cross-site replication
Ensure availability with Data Grid Operator by configuring geographically distributed clusters as a unified service.
You can configure clusters to perform cross-site replication with:
- Connections that Data Grid Operator manages.
- Connections that you configure and manage.
You can use both managed and manual connections for Data Grid clusters in the same Infinispan
CR. You must ensure that Data Grid clusters establish connections in the same way at each site.
13.1. Cross-site replication expose types
You can use a NodePort
service, a LoadBalancer
service, or an OpenShift Route
to handle network traffic for backup operations between Data Grid clusters. Before you start setting up cross-site replication you should determine what expose type is available for your Red Hat OpenShift cluster. In some cases you may require an administrator to provision services before you can configure an expose type.
NodePort
A NodePort
is a service that accepts network traffic at a static port, in the 30000
to 32767
range, on an IP address that is available externally to the OpenShift cluster.
To use a NodePort
as the expose type for cross-site replication, an administrator must provision external IP addresses for each OpenShift node. In most cases, an administrator must also configure DNS routing for those external IP addresses.
LoadBalancer
A LoadBalancer
is a service that directs network traffic to the correct node in the OpenShift cluster.
Whether you can use a LoadBalancer
as the expose type for cross-site replication depends on the host platform. AWS supports network load balancers (NLB) while some other cloud platforms do not. To use a LoadBalancer
service, an administrator must first create an ingress controller backed by an NLB.
Route
An OpenShift Route
allows Data Grid clusters to connect with each other through a public secure URL.
Data Grid uses TLS with the SNI header to send backup requests between clusters through an OpenShift Route
. To do this you must add a keystore with TLS certificates so that Data Grid can encrypt network traffic for cross-site replication.
When you specify Route
as the expose type for cross-site replication, Data Grid Operator creates a route with TLS passthrough encryption for each Data Grid cluster that it manages. You can specify a hostname for the Route
but you cannot specify a Route
that you have already created.
Additional resources
13.2. Managed cross-site replication
Data Grid Operator can discover Data Grid clusters running in different data centers to form global clusters.
When you configure managed cross-site connections, Data Grid Operator creates router pods in each Data Grid cluster. Data Grid pods use the <cluster_name>-site
service to connect to these router pods and send backup requests.
Router pods maintain a record of all pod IP addresses and parse RELAY message headers to forward backup requests to the correct Data Grid cluster. If a router pod crashes then all Data Grid pods start using any other available router pod until OpenShift restores it.
To manage cross-site connections, Data Grid Operator uses the Kubernetes API. Each OpenShift cluster must have network access to the remote Kubernetes API and a service account token for each backup cluster.
Data Grid clusters do not start running until Data Grid Operator discovers all backup locations that you configure.
13.2.1. Creating service account tokens for managed cross-site connections
Generate service account tokens on OpenShift clusters that allow Data Grid Operator to automatically discover Data Grid clusters and manage cross-site connections.
Prerequisites
Ensure all OpenShift clusters have access to the Kubernetes API.
Data Grid Operator uses this API to manage cross-site connections.NoteData Grid Operator does not modify remote Data Grid clusters. The service account tokens provide read only access through the Kubernetes API.
Procedure
- Log in to an OpenShift cluster.
Create a service account.
For example, create a service account at LON:
oc create sa lon serviceaccount/lon created
Add the view role to the service account with the following command:
oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
If you use a
NodePort
service to expose Data Grid clusters on the network, you must also add thecluster-reader
role to the service account:oc adm policy add-cluster-role-to-user cluster-reader -z <service-account-name> -n <namespace>
- Repeat the preceding steps on your other OpenShift clusters.
- Exchange service account tokens on each OpenShift cluster.
Additional resources
13.2.2. Exchanging service account tokens
After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.
Prerequisites
Get tokens from each service account.
Use the following command or get the token from the OpenShift Web Console:
oc sa get-token lon eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
Procedure
- Log in to an OpenShift cluster.
Add the service account token for a backup location with the following command:
oc create secret generic <token-name> --from-literal=token=<token>
For example, log in to the OpenShift cluster at NYC and create a
lon-token
secret as follows:oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
- Repeat the preceding steps on your other OpenShift clusters.
13.2.3. Configuring managed cross-site connections
Configure Data Grid Operator to establish cross-site views with Data Grid clusters.
Prerequisites
-
Determine a suitable expose type for cross-site replication.
If you use an OpenShiftRoute
you must add a keystore with TLS certificates and secure cross-site connections. - Create and exchange Red Hat OpenShift service account tokens for each Data Grid cluster.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. Configure the expose type for cross-site replication.
Set the value of the
spec.service.sites.local.expose.type
field to one of the following:-
NodePort
-
LoadBalancer
-
Route
-
Optionally specify a port or custom hostname with the following fields:
-
spec.service.sites.local.expose.nodePort
if you use aNodePort
service. -
spec.service.sites.local.expose.port
if you use aLoadBalancer
service. -
spec.service.sites.local.expose.routeHostName
if you use an OpenShiftRoute
.
-
Specify the number of pods that can send RELAY messages with the
service.sites.local.maxRelayNodes
field.TipConfigure all pods in your cluster to send
RELAY
messages for better performance. If all pods send backup requests directly, then no pods need to forward backup requests.-
Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
. If Data Grid cluster names or namespaces at the remote site do not match the local site, specify those values with the
clusterName
andnamespace
fields.The following are example
Infinispan
CR definitions for LON and NYC:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 maxRelayNodes: 1 locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443 secretName: nyc-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: nyc-cluster spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 maxRelayNodes: 1 locations: - name: LON clusterName: infinispan namespace: rhdg-namespace url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443 secretName: lon-token logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
Additional resources
13.3. Manually configuring cross-site connections
You can specify static network connection details to perform cross-site replication with Data Grid clusters running outside OpenShift. Manual cross-site connections are necessary in any scenario where access to the Kubernetes API is not available outside the OpenShift cluster where Data Grid runs.
Prerequisites
-
Determine a suitable expose type for cross-site replication.
If you use an OpenShiftRoute
you must add a keystore with TLS certificates and secure cross-site connections. Ensure you have the correct host names and ports for each Data Grid cluster and each
<cluster-name>-site
service.Manually connecting Data Grid clusters to form cross-site views requires predictable network locations for Data Grid services, which means you need to know the network locations before they are created.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. Configure the expose type for cross-site replication.
Set the value of the
spec.service.sites.local.expose.type
field to one of the following:-
NodePort
-
LoadBalancer
-
Route
-
Optionally specify a port or custom hostname with the following fields:
-
spec.service.sites.local.expose.nodePort
if you use aNodePort
service. -
spec.service.sites.local.expose.port
if you use aLoadBalancer
service. -
spec.service.sites.local.expose.routeHostName
if you use an OpenShiftRoute
.
-
Provide the name and static URL for each Data Grid cluster that acts as a backup location with
spec.service.sites.locations
, for example:LON
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan spec: replicas: 3 service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 maxRelayNodes: 1 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900 logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
NYC
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan spec: replicas: 2 service: type: DataGrid sites: local: name: NYC expose: type: LoadBalancer port: 65535 maxRelayNodes: 1 locations: - name: LON url: infinispan+xsite://infinispan-lon.myhost.com logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
ImportantBe sure to adjust logging categories in your
Infinispan
CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.spec: logging: categories: org.jgroups.protocols.TCP: error org.jgroups.protocols.relay.RELAY2: error
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.
Next steps
If your clusters have formed a cross-site view, you can start adding backup locations to caches.
Additional resources
13.4. Resources for configuring cross-site replication
The following tables provides fields and descriptions for cross-site resources.
Field | Description |
---|---|
| Data Grid supports cross-site replication with Data Grid service clusters only. |
Field | Description |
---|---|
| Names the local site where a Data Grid cluster runs. |
|
Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to |
|
Specifies a static port within the default range of |
|
Specifies the network port for the service if you expose Data Grid through a |
|
Specifies a custom hostname if you expose Data Grid through an OpenShift |
|
Specifies the maximum number of pods that can send RELAY messages for cross-site replication. The default value is |
Field | Description |
---|---|
| Provides connection information for all backup locations. |
|
Specifies a backup location that matches |
| Specifies the URL of the Kubernetes API for managed connections or a static URL for manual connections.
Use
Note that the
Use the |
| Specifies the secret that contains the service account token for the backup site. |
| Specifies the cluster name at the backup location if it is different to the cluster name at the local site. |
| Specifies the namespace of the Data Grid cluster at the backup location if it does not match the namespace at the local site. |
Managed cross-site connections
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer maxRelayNodes: 1 locations: - name: NYC clusterName: <nyc_cluster_name> namespace: <nyc_cluster_namespace> url: openshift://api.site-b.devcluster.openshift.com:6443 secretName: nyc-token
Manual cross-site connections
spec: service: type: DataGrid sites: local: name: LON expose: type: LoadBalancer port: 65535 maxRelayNodes: 1 locations: - name: NYC url: infinispan+xsite://infinispan-nyc.myhost.com:7900
13.5. Securing cross-site connections
Add keystores and trust stores so that Data Grid clusters can secure cross-site replication traffic.
You must add a keystore to use an OpenShift Route
as the expose type for cross-site replication. Securing cross-site connections is optional if you use a NodePort
or LoadBalancer
as the expose type.
Prerequisites
Have a PKCS12 keystore that Data Grid can use to encrypt and decrypt RELAY messages.
You must provide a keystore for relay pods and router pods to secure cross-site connections.
The keystore can be the same for relay pods and router pods or you can provide separate keystores for each.
You can also use the same keystore for each Data Grid cluster or a unique keystore for each cluster.Optionally have a trust store that contains part of the certificate chain or root CA certificate that verifies public certificates for Data Grid relay pods and router pods.
By default, Data Grid uses the Java trust store to verify public certificates.
Procedure
Create cross-site encryption secrets.
- Create keystore secrets.
- Create trust store secrets if you do not want to use the default Java trust store.
-
Modify the
Infinispan
CR for each Data Grid cluster to specify the secret name for theencryption.transportKeyStore.secretName
andencryption.routerKeyStore.secretName
fields. Configure any other fields to encrypt RELAY messages as required and then apply the changes.
apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan spec: replicas: 2 expose: type: LoadBalancer service: type: DataGrid sites: local: name: SiteA # ... encryption: protocol: TLSv1.3 transportKeyStore: secretName: transport-tls-secret alias: transport filename: keystore.p12 routerKeyStore: secretName: router-tls-secret alias: router filename: keystore.p12 trustStore: secretName: truststore-tls-secret filename: truststore.p12 locations: # ...
13.5.1. Resources for configuring cross-site encryption
The following tables provides fields and descriptions for encrypting cross-site connections.
Field | Description |
---|---|
|
Specifies the TLS protocol to use for cross-site connections. The default value is |
| Configures a keystore secret for relay pods. |
| Configures a keystore secret for router pods. |
| Configures an optional trust store secret for relay pods and router pods. |
Field | Description |
---|---|
| Specifies the secret that contains a keystore that relay pods can use to encrypt and decrypt RELAY messages. This field is required. |
|
Optionally specifies the alias of the certificate in the keystore. The default value is |
|
Optionally specifies the filename of the keystore. The default value is |
Field | Description |
---|---|
| Specifies the secret that contains a keystore that router pods can use to encrypt and decrypt RELAY messages. This field is required. |
|
Optionally specifies the alias of the certificate in the keystore. The default value is |
|
Optionally specifies the filename of the keystore. The default value is |
Field | Description |
---|---|
|
Optionally specifies the secret that contains a trust store to verify public certificates for relay pods and router pods. The default value is |
|
Optionally specifies the filename of the trust store. The default value is |
13.5.2. Cross-site encryption secrets
Cross-site replication encryption secrets add keystores and optional trust stores for securing cross-site connections.
Cross-site encryption secrets
apiVersion: v1 kind: Secret metadata: name: tls-secret type: Opaque stringData: password: changeme type: pkcs12 data: <file-name>: "MIIKDgIBAzCCCdQGCSqGSIb3DQEHA..."
Field | Description |
---|---|
| Specifies the password for the keystore or trust store. |
|
Optionally specifies the keystore or trust store type. The default value is |
| Adds a base64-encoded keystore or trust store. |
13.6. Configuring sites in the same OpenShift cluster
For evaluation and demonstration purposes, you can configure Data Grid to back up between pods in the same OpenShift cluster.
Using ClusterIP
as the expose type for cross-site replication is intended for demonstration purposes only. It would be appropriate to use this expose type only to perform a temporary proof-of-concept deployment on a laptop or something of that nature.
Procedure
-
Create an
Infinispan
CR for each Data Grid cluster. -
Specify the name of the local site with
spec.service.sites.local.name
. -
Set
ClusterIP
as the value of thespec.service.sites.local.expose.type
field. -
Provide the name of the Data Grid cluster that acts as a backup location with
spec.service.sites.locations.clusterName
. If both Data Grid clusters have the same name, specify the namespace of the backup location with
spec.service.sites.locations.namespace
.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: example-clustera spec: replicas: 1 expose: type: LoadBalancer service: type: DataGrid sites: local: name: SiteA expose: type: ClusterIP maxRelayNodes: 1 locations: - name: SiteB clusterName: example-clusterb namespace: cluster-namespace
-
Configure your
Infinispan
CRs with any other Data Grid service resources and then apply the changes. Verify that Data Grid clusters form a cross-site view.
Retrieve the
Infinispan
CR.oc get infinispan -o yaml
-
Check for the
type: CrossSiteViewFormed
condition.