Chapter 5. Deploy Data Grid for HA with the Data Grid Operator

Configure the credential to access the Data Grid cluster.

Red Hat build of Keycloak needs this credential to be able to authenticate with the Data Grid cluster. The following identities.yaml file sets the username and password with admin permissions

credentials:
  - username: developer
    password: strong-password
    roles:
      - admin

The identities.yaml could be set in a secret as one of the following:

Create a service account.

A service account is required to establish a connection between clusters. The Data Grid Operator uses it to inspect the network configuration from the remote site and to configure the local Data Grid cluster accordingly.

For more details, see the Managing Cross-Site Connections documentation.

Create TLS secrets

In this chapter, Data Grid uses an OpenShift Route for the cross-site communication. It uses the SNI extension of TLS to direct the traffic to the correct Pods. To achieve that, JGroups use TLS sockets, which require a Keystore and Truststore with the correct certificates.

For more information, see the Securing Cross Site Connections documentation or this Red Hat Developer Guide.

Upload the Keystore and the Truststore in an OpenShift Secret. The secret contains the file content, the password to access it, and the type of the store. Instructions for creating the certificates and the stores are beyond the scope of this chapter.

To upload the Keystore as a Secret, use the following command:

oc -n keycloak create secret generic xsite-keystore-secret \
  --from-file=keystore.p12="./certs/keystore.p12" \ 1
  --from-literal=password=secret \ 2
  --from-literal=type=pkcs12 3

To upload the Truststore as a Secret, use the following command:

oc -n keycloak create secret generic xsite-truststore-secret \
        --from-file=truststore.p12="./certs/truststore.p12" \  1
        --from-literal=password=caSecret \  2
        --from-literal=type=pkcs12  3

Create a Cluster for Data Grid with Cross-Site enabled

The Setting Up Cross-Site documentation provides all the information on how to create and configure your Data Grid cluster with cross-site enabled, including the previous steps.

A basic example is provided in this chapter using the credentials, tokens, and TLS Keystore/Truststore created by the commands from the previous steps.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan 1
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true' 2
spec:
  replicas: 3
  jmx:
    enabled: true
  security:
    endpointSecretName: connect-secret 3
  service:
    type: DataGrid
    sites:
      local:
        name: site-a 4
        expose:
          type: Route 5
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret 6
            alias: xsite 7
            filename: keystore.p12 8
          routerKeyStore:
            secretName: xsite-keystore-secret 9
            alias: xsite 10
            filename: keystore.p12 11
          trustStore:
            secretName: xsite-truststore-secret 12
            filename: truststore.p12 13
      locations:
        - name: site-b 14
          clusterName: infinispan
          namespace: keycloak 15
          url: openshift://api.site-b 16
          secretName: xsite-token-secret 17

For Site-B, the Infinispan CR looks similar to the above. Note the differences in point 4, 11 and 13.

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan 1
  namespace: keycloak
  annotations:
    infinispan.org/monitoring: 'true' 2
spec:
  replicas: 3
  jmx:
    enabled: true
  security:
    endpointSecretName: connect-secret 3
  service:
    type: DataGrid
    sites:
      local:
        name: site-b 4
        expose:
          type: Route 5
        maxRelayNodes: 128
        encryption:
          transportKeyStore:
            secretName: xsite-keystore-secret 6
            alias: xsite 7
            filename: keystore.p12 8
          routerKeyStore:
            secretName: xsite-keystore-secret 9
            alias: xsite 10
            filename: keystore.p12 11
          trustStore:
            secretName: xsite-truststore-secret 12
            filename: truststore.p12 13
      locations:
        - name: site-a 14
          clusterName: infinispan
          namespace: keycloak 15
          url: openshift://api.site-a 16
          secretName: xsite-token-secret 17

Creating the caches for Red Hat build of Keycloak.

Red Hat build of Keycloak requires the following caches to be present: actionTokens, authenticationSessions, loginFailures, and work.

The Data Grid Cache CR allows deploying the caches in the Data Grid cluster. Cross-site needs to be enabled per cache as documented by Cross Site Documentation. The documentation contains more details about the options used by this chapter. The following example shows the Cache CR for Site-A.

apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: authenticationsessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: authenticationSessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: "5000"
      encoding:
        media-type: "application/x-protostream"
      locking:
        acquireTimeout: "4000"
      transaction:
        mode: "NON_XA" 1
        locking: "PESSIMISTIC" 2
      stateTransfer:
        chunkSize: "16"
      backups:
        site-b: 3
          backup:
            strategy: "SYNC" 4
            timeout: "4500" 5
            failurePolicy: "FAIL" 6
            stateTransfer:
              chunkSize: "16"

The example above is the recommended configuration to achieve the best data consistency.

For Site-B, the Cache CR is similar, except for the backups.<name> outlined in point 3 of the above diagram.

apiVersion: infinispan.org/v2alpha1
kind: Cache
metadata:
  name: authenticationsessions
  namespace: keycloak
spec:
  clusterName: infinispan
  name: authenticationSessions
  template: |-
    distributedCache:
      mode: "SYNC"
      owners: "2"
      statistics: "true"
      remoteTimeout: "5000"
      encoding:
        media-type: "application/x-protostream"
      locking:
        acquireTimeout: "4000"
      transaction:
        mode: "NON_XA" 1
        locking: "PESSIMISTIC" 2
      stateTransfer:
        chunkSize: "16"
      backups:
        site-a: 3
          backup:
            strategy: "SYNC" 4
            timeout: "4500" 5
            failurePolicy: "FAIL" 6
            stateTransfer:
              chunkSize: "16"

Create a Secret with the username and password to connect to the external Data Grid deployment:

apiVersion: v1
kind: Secret
metadata:
  name: remote-store-secret
  namespace: keycloak
type: Opaque
data:
  username: ZGV2ZWxvcGVy # base64 encoding for 'developer'
  password: c2VjdXJlX3Bhc3N3b3Jk # base64 encoding for 'secure_password'

Extend the Red Hat build of Keycloak Custom Resource with additionalOptions as shown below.

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  labels:
    app: keycloak
  name: keycloak
  namespace: keycloak
spec:
  additionalOptions:
    - name: cache-remote-host 1
      value: "infinispan.keycloak.svc"
    - name: cache-remote-port 2
      value: "11222"
    - name: cache-remote-username 3
      secret:
        name: remote-store-secret
        key: username
    - name: cache-remote-password 4
      secret:
        name: remote-store-secret
        key: password
    - name: spi-connections-infinispan-quarkus-site-name 5
      value: keycloak