Chapter 10. Configuring distributed caches

The following data is kept local to each node in the cluster using local caches:

Local caches for realms, users, and authorization are configured to hold up to 10,000 entries per default. The local key cache can hold up to 1,000 entries per default and defaults to expire every one hour. Therefore, keys are forced to be periodically downloaded from external clients or identity providers.

In order to achieve an optimal runtime and avoid additional round-trips to the database you should consider looking at the configuration for each cache to make sure the maximum number of entries is aligned with the size of your database. More entries you can cache, less often the server needs to fetch data from the database. You should evaluate the trade-offs between memory utilization and performance.

When one Red Hat build of Keycloak node updates data in the shared database, all other nodes need to be aware of it, so they invalidate that data from their caches.

The work cache is a replicated cache and used for sending these invalidation messages. The entries/messages in this cache are very short-lived, and you should not expect this cache growing in size over time.

The authenticationSessions distributed cache is used to store authentication sessions and any other data associated with it during the authentication process.

By relying on a distributable cache, authentication sessions are available to any node in the cluster so that users can be redirected to any node without losing their authentication state. However, production-ready deployments should always consider session affinity and favor redirecting users to the node where their sessions were initially created. By doing that, you are going to avoid unnecessary state transfer between nodes and improve CPU, memory, and network utilization.

User and client sessions are automatically destroyed whenever the user performs a logout, the client performs a token revocation, or due to reaching their expiration time.

The session data are stored in the database by default and loaded on-demand to the following caches:

By relying on a distributable cache, cached user and client sessions are available to any node in the cluster so that users can be redirected to any node without the need to load session data from the database. However, production-ready deployments should always consider session affinity and favor redirecting users to the node where their sessions were initially created. By doing that, you are going to avoid unnecessary state transfer between nodes and improve CPU, memory, and network utilization.

These in-memory caches for user sessions and client sessions are limited to, by default, 10000 entries per node which reduces the overall memory usage of Red Hat build of Keycloak for larger installations. The internal caches will run with only a single owner for each cache entry. Consider trade-off between memory consumption and the database utilization and set different sizes for the caches, edit Red Hat build of Keycloak’s cache config file (conf/cache-ispn.xml) to set a <memory max-count="..."/> for those caches.

Since all the sessions in this setup are stored in-memory, there are two side effects related to this:

Follow these steps to enable this setup:

The following caches are used to store offline sessions:

Similarly to regular user and client sessions caches, also the offline user and client session caches are limited to 10000 entries per node by default. Items which are evicted from the memory will be loaded on-demand from the database when needed. Consider trade-off between memory consumption and the database utilization and set different sizes for the caches, edit Red Hat build of Keycloak’s cache config file (conf/cache-ispn.xml) to set a <memory max-count="..."/> for those caches.

In order to reduce memory usage, it’s possible to place an upper bound on the number of entries which are stored in a given cache. To specify an upper bound of on a cache, you must provide the following command line argument --cache-embedded-${CACHE_NAME}-max-count=, with ${CACHE_NAME} replaced with the name of the cache you would like to apply the upper bound to. For example, to apply an upper-bound of 1000 to the offlineSessions cache you would configure --cache-embedded-offline-sessions-max-count=1000. An upper bound can not be defined on the following caches: actionToken, authenticationSessions, loginFailures, work.

Distributed caches replicate cache entries on a subset of nodes in a cluster and assigns entries to fixed owner nodes.

Each distributed cache, that is a primary source of truth of the data (authenticationSessions, loginFailures and actionTokens) has two owners per default, which means that two nodes have a copy of the specific cache entries. Non-owner nodes query the owners of a specific cache to obtain data. When both owner nodes are offline, all data is lost.

The default number of owners is enough to survive 1 node (owner) failure in a cluster setup with at least three nodes. You are free to change the number of owners accordingly to better fit into your availability requirements. To change the number of owners, open conf/cache-ispn.xml and change the value for owners=<value> for the distributed caches to your desired value.

To specify your own cache configuration file, enter this command:

bin/kc.[sh|bat] start --cache-config-file=my-cache-file.xml

bin/kc.[sh|bat] start --cache-config-file=my-cache-file.xml

The configuration file is relative to the conf/ directory.

For configuration of Red Hat build of Keycloak server for high availability and multi-node clustered setup there was introduced following CLI options cache-remote-host, cache-remote-port, cache-remote-username and cache-remote-password simplifying configuration within the XML file. Once any of the declared CLI parameters are present, it is expected there is no configuration related to remote store present in the XML file.

The following table shows transport stacks that are available without any further configuration than using the --cache-stack runtime option:

The following table shows transport stacks that are available using the --cache-stack runtime option and a minimum configuration:

When using the tcp, udp or jdbc-ping-udp stack, each cluster must use a different multicast address and/or port so that their nodes form distinct clusters. By default, Red Hat build of Keycloak uses 239.6.7.8 as multicast address for jgroups.mcast_addr and 46655 for the multicast port jgroups.mcast_port.

When using a service mesh like Istio, you might need to allow a direct mTLS communication between the Red Hat build of Keycloak Pods to allow for the mutual authentication to work. Otherwise, you might see error messages like JGRP000006: failed accepting connection from peer SSLSocket that indicate that a wrong certificate was presented, and the cluster will not form correctly.

You then have the option to allow direct mTLS communication between the Red Hat build of Keycloak Pods, or rely on the service mesh transport security to encrypt the communication and to authenticate peers.

To allow direct mTLS communication for Red Hat build of Keycloak when using Istio:

As an alternative, to disable the mTLS communication, and rely on the service mesh to encrypt the traffic:

Although not recommended for standard setups, if it is essential in a specific setup, you can configure the keystore with the certificate for the transport stack manually. cache-embedded-mtls-key-store-file sets the path to the keystore, and cache-embedded-mtls-key-store-password sets the password to decrypt it. The truststore contains the valid certificates to accept connection from, and it can be configured with cache-embedded-mtls-trust-store-file (path to the truststore), and cache-embedded-mtls-trust-store-password (password to decrypt it). To restrict unauthorized access, always use a self-signed certificate for each Red Hat build of Keycloak deployment.