Red Hat SummitChapter 23. Checking if rolling updates are possible
Execute the update compatibility command to check if Red Hat build of Keycloak supports a rolling update for a change in your deployment.
Use the update compatibility command to determine if you can update your deployment with a rolling update strategy when enabling or disabling features or changing the Red Hat build of Keycloak version, configurations or providers and themes. The outcome shows whether a rolling update is possible or if a recreate update is required.
In its current version, it shows that a rolling update is possible when the Red Hat build of Keycloak version is the same for the old and the new version. Future versions of Red Hat build of Keycloak might change that behavior to use additional information from the configuration, the image and the version to determine if a rolling update is possible.
In the next iteration of this feature, it is possible to use rolling update strategy also when updating to the following patch release of Red Hat build of Keycloak. Refer to Section 23.4, “Rolling updates for patch releases” section for more details.
This is fully scriptable, so your update procedure can use that information to perform a rolling or recreate strategy depending on the change performed. It is also GitOps friendly, as it allows storing the metadata of the previous configuration in a file. Use this file in a CI/CD pipeline with the new configuration to determine if a rolling update is possible or if a recreate update is needed.
If you are using the Red Hat build of Keycloak Operator, continue to the Avoiding downtime with rolling updates chapter and the Auto strategy for more information.
23.1. Supported update strategies
- Rolling Update
- In this guide, a rolling update is an update that can be performed with zero downtime for your deployment, which consists of at least two nodes. Update your Red Hat build of Keycloak one by one; shut down one of your old deployment nodes and start a new deployment node. Wait until the new node’s start-up probe returns success before proceeding to the next Red Hat build of Keycloak node. See chapter Tracking instance status with health checks for details on how to enable and use the start-up probe.
- Recreate Update
- A recreate update is not compatible with zero-downtime and requires downtime to be applied. Shut down all nodes of the cluster running the old version before starting the nodes with the new version.
23.2. Determining the update strategy for an updated configuration
To determine if a rolling update is possible:
- Run the update compatibility command to generate the required metadata with the old configuration.
- Check the metadata with the new configuration to determine the update strategy.
If you do not use --optimized keep in mind that an update command may implicitly create or update an optimized build for you - if you are running the command from the same machine as a server instance, this may impact the next start of your server.
Consumers of these commands should not rely on the internal behavior or the structure of the metadata file. Instead, they should rely only on the exit code of the check command to benefit from future enhancements on the internal logic to determine when a rolling update is possible.
23.2.1. Generating the Metadata
To generate the metadata, execute the following command using the same Red Hat build of Keycloak version and configuration options:
Generate and save the metadata from the current deployment.
bin/kc.[sh|bat] update-compatibility metadata --file=/path/to/file.json
bin/kc.[sh|bat] update-compatibility metadata --file=/path/to/file.json
This command accepts all options used by the start command. The command displays the metadata, in JSON format, in the console for debugging purposes. The --file parameter allows you to save the metadata to a file. Use this file with the subsequent check command.
Ensure that all configuration options, whether set via environment variables or CLI arguments, are included when running the above command.
Omitting any configuration options results in incomplete metadata, and could lead to a wrong reported result in the next step.
23.2.2. Checking the Metadata
This command checks the metadata generated by the previous command and compares it with the current configuration and Red Hat build of Keycloak version. If you are updating to a new Red Hat build of Keycloak version, this command must be executed with the new version.
Check the metadata from a previous deployment.
bin/kc.[sh|bat] update-compatibility check --file=/path/to/file.json
bin/kc.[sh|bat] update-compatibility check --file=/path/to/file.json- Ensure that all configuration options, whether set via environment variables or CLI arguments, are included when running this command.
- Verify that the correct Red Hat build of Keycloak version is used.
Failure to meet these requirements results in an incorrect outcome.
The command prints the result to the console. For example, if a rolling update is possible, it displays:
Rolling Update possible message
[OK] Rolling Update is available.
[OK] Rolling Update is available.If no rolling update is possible, the command provides details about the incompatibility:
Rolling Update not possible message
[keycloak] Rolling Update is not available. 'keycloak.version' is incompatible: 26.2.0 -> 26.2.1
[keycloak] Rolling Update is not available. 'keycloak.version' is incompatible: 26.2.0 -> 26.2.1 - 1
- In this example, the Keycloak version
26.2.0is not compatible with version26.2.1and a rolling update is not possible.
In the next iteration of this feature, it is possible to use rolling update strategy also when updating to the following patch release of Red Hat build of Keycloak. Refer to Section 23.4, “Rolling updates for patch releases” section for more details.
Command exit code
Use the command’s exit code to determine the update type in your automation pipeline:
| Exit Code | Description |
|---|---|
|
| Rolling Update is possible. |
|
| Unexpected error occurred (such as the metadata file is missing or corrupted). |
|
| Invalid CLI option. |
|
| Rolling Update is not possible. The deployment must be shut down before applying the new configuration. |
|
|
Rolling Update is not possible. The feature |
23.3. Rolling incompatible changes
The following configuration changes return a "Rolling Update is not possible" result code.
23.3.1. Features
23.3.1.1. Recreate always
The enabling or disabling of the following features requires a recreate update:
| Feature | Description |
|---|---|
| multi-site:v1 | Multi-site support |
| persistent-user-sessions:v1 | Persistent online user sessions across restarts and upgrades |
23.3.1.2. Recreate on feature version change
Changing the following features versions triggers a recreate update:
| Feature | Description |
|---|---|
| login:v1 | Legacy Login Theme |
| login:v2 | New Login Theme |
| passkeys-conditional-ui-authenticator:v1 | Passkeys conditional UI authenticator |
23.3.2. Configuration options
Changing the value of one of the following CLI options triggers a recreate update:
| Option | Rationale |
|---|---|
|
|
The |
|
| Changing the configuration file could result in incompatible cache or transport configurations, resulting in clusters not forming as expected. |
|
| Changing stack will result in the cluster not forming during rolling update and will lead to data loss. |
|
| Enabling/Disabling TLS will result in the cluster not forming during rolling update and will lead to data loss. |
|
| Connecting to a new remote cache will cause previously cached data to be lost. |
|
| Connecting to a new remote cache will cause previously cached data to be lost. |
Red Hat build of Keycloak does not verify changes to the content of the cache configuration file provided via --cache-config-file. If you change this file, you need to review and test your changes to ensure that nodes using the new configuration can form a cluster with the nodes running the old configuration. If a cluster cannot be formed, you should shut down Red Hat build of Keycloak running the old configuration first before migrating to the new configuration.
| Option | Rationale |
|---|---|
|
| Migration to a new database vendor should be applied to all cluster members to ensure data consistency. |
|
| Migration to a new database schema should be applied to all cluster members to ensure data consistency. |
|
| Migration to a new database name should be applied to all cluster members to ensure data consistency. |
|
| All cluster members should be connecting to the same database to ensure data consistency. |
|
| All cluster members should be connecting to the same database to ensure data consistency. |
Red Hat build of Keycloak allows changes to the --db-url option to be rolled out in order to facilitate changes to JDBC properties. Great care should be taken when updating this value as changes to the host, port or database name could lead to distinct cluster members connecting to a different database, resulting in data consistency issues.
23.4. Rolling updates for patch releases
This behavior is currently in preview mode, and it is not recommended for use in production.
It is possible to configure the Red Hat build of Keycloak compatibility command to allow rolling updates when upgrading to a newer patch version in the same major.minor release stream.
To enable this behavior for compatibility check command enable feature rolling-updates:v2 as shown in the following example.
bin/kc.[sh|bat] update-compatibility check --file=/path/to/file.json --features=rolling-updates:v2
bin/kc.[sh|bat] update-compatibility check --file=/path/to/file.json --features=rolling-updates:v2
Note there is no change needed when generating metadata using metadata command.
Recommended Configuration:
- Enable sticky sessions in your loadbalancer to avoid users bouncing between different versions of Red Hat build of Keycloak as this could result in users needing to refresh their Account Console and Admin UI multiple times while the upgrade is progressing.
Supported functionality during rolling updates:
- Users can log in and log out for OpenID Connect clients.
- OpenID Connect clients can perform all operations, for example, refreshing tokens and querying the user info endpoint.
Known limitations:
- If there have been changes to the Account Console or Admin UI in the patch release, and the user opened the Account Console or Admin UI before or during the upgrade, the user might see an error message and be asked to reload the application while navigating in browser during or after the upgrade.
- If the two patch releases of Red Hat build of Keycloak use different versions of the embedded Infinispan, no rolling update of Red Hat build of Keycloak be performed.
23.5. Further reading
The Red Hat build of Keycloak Operator uses the functionality described above to determine if a rolling update is possible. See the Avoiding downtime with rolling updates chapter and the Auto strategy for more information.
23.6. Relevant options
| Value | |
|---|---|
| 🛠
|
|
| 🛠
|
|
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}