Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 6. Deploy Data Grid for HA with the Data Grid Operator

This chapter describes the procedures required to deploy Data Grid in a multiple-cluster environment (cross-site). For simplicity, this topic uses the minimum configuration possible that allows Red Hat build of Keycloak to be used with an external Data Grid.

This chapter assumes two OpenShift clusters named Site-A and Site-B.

This is a building block following the concepts described in the Concepts for active-passive deployments chapter. See the Multi-site deployments chapter for an overview.

6.1. Architecture
Copy link

This setup deploys two synchronously replicating Data Grid clusters in two sites with a low-latency network connection. An example of this scenario could be two availability zones in one AWS region.

Red Hat build of Keycloak, loadbalancer and database have been removed from the following diagram for simplicity.

6.2. Prerequisites
Copy link

6.3. Procedure
Copy link

  1. Install the Data Grid Operator

  2. Configure the credential to access the Data Grid cluster.

    Red Hat build of Keycloak needs this credential to be able to authenticate with the Data Grid cluster. The following identities.yaml file sets the username and password with admin permissions 

    credentials:
  - username: developer
    password: strong-password
    roles:
      - admin
    Copy to Clipboard Toggle word wrap

    The identities.yaml could be set in a secret as one of the following:

    • As a Kubernetes Resource:

      Credential Secret

      apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: connect-secret
  namespace: keycloak
data:
  identities.yaml: Y3JlZGVudGlhbHM6CiAgLSB1c2VybmFtZTogZGV2ZWxvcGVyCiAgICBwYXNzd29yZDogc3Ryb25nLXBhc3N3b3JkCiAgICByb2xlczoKICAgICAgLSBhZG1pbgo=
      1
      Copy to Clipboard Toggle word wrap

      1
      The identities.yaml from the previous example base64 encoded.

    • Using the CLI 

      oc create secret generic connect-secret --from-file=identities.yaml
      Copy to Clipboard Toggle word wrap

      Check the Configuring Authentication documentation for more details.

      These commands must be executed on both OpenShift clusters.

  3. Create a service account.

    A service account is required to establish a connection between clusters. The Data Grid Operator uses it to inspect the network configuration from the remote site and to configure the local Data Grid cluster accordingly.

    For more details, see the Managing Cross-Site Connections documentation.

    1. Create a service-account-token secret type as follows. The same YAML file can be used in both OpenShift clusters.

      xsite-sa-secret-token.yaml

      apiVersion: v1
kind: Secret
metadata:
  name: ispn-xsite-sa-token
      1 
      
  annotations:
    kubernetes.io/service-account.name: "xsite-sa"
      2 
      
type: kubernetes.io/service-account-token
      Copy to Clipboard Toggle word wrap

      1
      The secret name.
      2
      The service account name.

    2. Create the service account and generate an access token in both OpenShift clusters.

      Create the service account in Site-A

      oc create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
oc create -f xsite-sa-secret-token.yaml
oc get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-A-token.txt
      Copy to Clipboard Toggle word wrap

      Create the service account in Site-B

      oc create sa -n keycloak xsite-sa
oc policy add-role-to-user view -n keycloak -z xsite-sa
oc create -f xsite-sa-secret-token.yaml
oc get secrets ispn-xsite-sa-token -o jsonpath="{.data.token}" | base64 -d > Site-B-token.txt
      Copy to Clipboard Toggle word wrap

    3. The next step is to deploy the token from Site-A into Site-B and the reverse:

      Deploy Site-B token into Site-A

      oc create secret generic -n keycloak xsite-token-secret \
  --from-literal=token="$(cat Site-B-token.txt)"
      Copy to Clipboard Toggle word wrap

      Deploy Site-A token into Site-B

      oc create secret generic -n keycloak xsite-token-secret \
  --from-literal=token="$(cat Site-A-token.txt)"
      Copy to Clipboard Toggle word wrap

  4. Create TLS secrets

    In this chapter, Data Grid uses an OpenShift Route for the cross-site communication. It uses the SNI extension of TLS to direct the traffic to the correct Pods. To achieve that, JGroups use TLS sockets, which require a Keystore and Truststore with the correct certificates.

    For more information, see the Securing Cross Site Connections documentation or this Red Hat Developer Guide.

    Upload the Keystore and the Truststore in an OpenShift Secret. The secret contains the file content, the password to access it, and the type of the store. Instructions for creating the certificates and the stores are beyond the scope of this guide.

    To upload the Keystore as a Secret, use the following command:

    Deploy a Keystore

    oc -n keycloak create secret generic xsite-keystore-secret \
  --from-file=keystore.p12="./certs/keystore.p12" \
    1 
    
  --from-literal=password=secret \
    2 
    
  --from-literal=type=pkcs12
    3
    Copy to Clipboard Toggle word wrap

    1
    The filename and the path to the Keystore.
    2
    The password to access the Keystore.
    3
    The Keystore type.

    To upload the Truststore as a Secret, use the following command:

    Deploy a Truststore

    oc -n keycloak create secret generic xsite-truststore-secret \
        --from-file=truststore.p12="./certs/truststore.p12" \
    1 
    
        --from-literal=password=caSecret \
    2 
    
        --from-literal=type=pkcs12
    3
    Copy to Clipboard Toggle word wrap

    1
    The filename and the path to the Truststore.
    2
    The password to access the Truststore.
    3
    The Truststore type.
    Note

    Keystore and Truststore must be uploaded in both OpenShift clusters.

  5. Create a Cluster for Data Grid with Cross-Site enabled

    The Setting Up Cross-Site documentation provides all the information on how to create and configure your Data Grid cluster with cross-site enabled, including the previous steps.

    A basic example is provided in this chapter using the credentials, tokens, and TLS Keystore/Truststore created by the commands from the previous steps.

    The Infinispan CR for Site-A

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
    1 
    
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true'
    2 
    
spec:
  replicas: 3
  security:
    endpointSecretName: connect-secret
    3 
    
  service:
    type: DataGrid
    sites:
      local:
        name: site-a
    4 
    
        expose:
          type: Route
    5 
    
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret
    6 
    
            alias: xsite
    7 
    
            filename: keystore.p12
    8 
    
          routerKeyStore:
            secretName: xsite-keystore-secret
    9 
    
            alias: xsite
    10 
    
            filename: keystore.p12
    11 
    
          trustStore:
            secretName: xsite-truststore-secret
    12 
    
            filename: truststore.p12
    13 
    
      locations:
        - name: site-b
    14 
    
          clusterName: infinispan
          namespace: keycloak
    15 
    
          url: openshift://api.site-b
    16 
    
          secretName: xsite-token-secret
    17
    Copy to Clipboard Toggle word wrap

    1
    The cluster name
    2
    Allows the cluster to be monitored by Prometheus.
    3
    If using a custom credential, configure here the secret name.
    4
    The name of the local site, in this case Site-A.
    5
    Exposing the cross-site connection using OpenShift Route.
    6 9
    The secret name where the Keystore exists as defined in the previous step.
    7 10
    The alias of the certificate inside the Keystore.
    8 11
    The secret key (filename) of the Keystore as defined in the previous step.
    12
    The secret name where the Truststore exists as defined in the previous step.
    13
    The Truststore key (filename) of the Keystore as defined in the previous step.
    14
    The remote site’s name, in this case Site-B.
    15
    The namespace of the Data Grid cluster from the remote site.
    16
    The OpenShift API URL for the remote site.
    17
    The secret with the access toke to authenticate into the remote site.

    For Site-B, the Infinispan CR looks similar to the above. Note the differences in point 4, 11 and 13.

    The Infinispan CR for Site-B

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
    1 
    
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true'
    2 
    
spec:
  replicas: 3
  security:
    endpointSecretName: connect-secret
    3 
    
  service:
    type: DataGrid
    sites:
      local:
        name: site-b
    4 
    
        expose:
          type: Route
    5 
    
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret
    6 
    
            alias: xsite
    7 
    
            filename: keystore.p12
    8 
    
          routerKeyStore:
            secretName: xsite-keystore-secret
    9 
    
            alias: xsite
    10 
    
            filename: keystore.p12
    11 
    
          trustStore:
            secretName: xsite-truststore-secret
    12 
    
            filename: truststore.p12
    13 
    
      locations:
        - name: site-a
    14 
    
          clusterName: infinispan
          namespace: keycloak
    15 
    
          url: openshift://api.site-a
    16 
    
          secretName: xsite-token-secret
    17
    Copy to Clipboard Toggle word wrap

  6. Creating the caches for Red Hat build of Keycloak.

    Red Hat build of Keycloak requires the following caches to be present: sessions, actionTokens, authenticationSessions, offlineSessions, clientSessions, offlineClientSessions, loginFailures, and work.

    The Data Grid Cache CR allows deploying the caches in the Data Grid cluster. Cross-site needs to be enabled per cache as documented by Cross Site Documentation. The documentation contains more details about the options used by this chapter. The following example shows the Cache CR for Site-A.

    sessions in Site-A

    apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: sessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: sessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: 14000
      stateTransfer:
        chunkSize: 16
      backups:
        mergePolicy: ALWAYS_REMOVE
    1 
    
        site-b:
    2 
    
          backup:
            strategy: "SYNC"
    3 
    
            timeout: 13000
            stateTransfer:
              chunkSize: 16
    Copy to Clipboard Toggle word wrap

    1 1
    The cross-site merge policy, invoked when there is a write-write conflict. Set this for the caches sessions, authenticationSessions, offlineSessions, clientSessions and offlineClientSessions, and do not set it for all other caches.
    2 2
    The remote site name.
    3 3
    The cross-site communication, in this case, SYNC.

    For Site-B, the Cache CR is similar except in point 2.

    session in Site-B

    apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: sessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: sessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: 14000
      stateTransfer:
        chunkSize: 16
      backups:
        mergePolicy: ALWAYS_REMOVE
    1 
    
        site-a:
    2 
    
          backup:
            strategy: "SYNC"
    3 
    
            timeout: 13000
            stateTransfer:
              chunkSize: 16
    Copy to Clipboard Toggle word wrap

6.4. Verifying the deployment
Copy link

Confirm that the Data Grid cluster is formed, and the cross-site connection is established between the OpenShift clusters.

Wait until the Data Grid cluster is formed

oc wait --for condition=WellFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan
Copy to Clipboard Toggle word wrap

Wait until the Data Grid cross-site connection is established

oc wait --for condition=CrossSiteViewFormed --timeout=300s infinispans.infinispan.org -n keycloak infinispan
Copy to Clipboard Toggle word wrap

6.5. Next steps
Copy link

After Data Grid is deployed and running, use the procedure in the Connect Red Hat build of Keycloak with an external Data Grid chapter to connect your Red Hat build of Keycloak cluster with the Data Grid cluster.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat