Chapter 7. User Storage SPI

One important method of UserModel is the getId() method. When implementing UserModel developers must be aware of the user id format. The format must be:

"f:" + component id + ":" + external id

The Red Hat build of Keycloak runtime often has to look up users by their user id. The user id contains enough information so that the runtime does not have to query every single UserStorageProvider in the system to find the user.

The component id is the id returned from ComponentModel.getId(). The ComponentModel is passed in as a parameter when creating the provider class so you can get it from there. The external id is information your provider class needs to find the user in the external store. This is often a username or a uid. For example, it might look something like this:

f:332a234e31234:wburke

When the runtime does a lookup by id, the id is parsed to obtain the component id. The component id is used to locate the UserStorageProvider that was originally used to load the user. That provider is then passed the id. The provider again parses the id to obtain the external id and it will use to locate the user in external user storage.

This format has the drawback that it can generate long IDs for the external storage users. This is specially important when combined with the WebAuthn authentication, which limits the user handle ID to 64 bytes. For that reason, if the storage users are going to use WebAuthn authentication, it is important to limit the full storage ID to 64 characters. The method validateConfiguration can be used to assign a short ID for the provider component on creation, giving some space to the user IDs within the 64 byte limitation.

    @Override
    void validateConfiguration(KeycloakSession session, RealmModel realm, ComponentModel model)
            throws ComponentValidationException
    {
        // ...
        if (model.getId() == null) {
            // On creation use short UUID of 22 chars, 40 chars left for the user ID
            model.setId(KeycloakModelUtils.generateShortId());
        }
    }

The first thing we will walk through is the UserStorageProvider class.

public class PropertyFileUserStorageProvider implements
        UserStorageProvider,
        UserLookupProvider,
        CredentialInputValidator,
        CredentialInputUpdater
{
...
}

Our provider class, PropertyFileUserStorageProvider, implements many interfaces. It implements the UserStorageProvider as that is a base requirement of the SPI. It implements the UserLookupProvider interface because we want to be able to log in with users stored by this provider. It implements the CredentialInputValidator interface because we want to be able to validate passwords entered in using the login screen. Our property file is read-only. We implement the CredentialInputUpdater because we want to post an error condition when the user attempts to update his password.

    protected KeycloakSession session;
    protected Properties properties;
    protected ComponentModel model;
    // map of loaded users in this transaction
    protected Map<String, UserModel> loadedUsers = new HashMap<>();

    public PropertyFileUserStorageProvider(KeycloakSession session, ComponentModel model, Properties properties) {
        this.session = session;
        this.model = model;
        this.properties = properties;
    }

The constructor for this provider class is going to store the reference to the KeycloakSession, ComponentModel, and property file. We’ll use all of these later. Also notice that there is a map of loaded users. Whenever we find a user we will store it in this map so that we avoid re-creating it again within the same transaction. This is a good practice to follow as many providers will need to do this (that is, any provider that integrates with JPA). Remember also that provider class instances are created once per transaction and are closed after the transaction completes.

    @Override
    public UserModel getUserByUsername(RealmModel realm, String username) {
        UserModel adapter = loadedUsers.get(username);
        if (adapter == null) {
            String password = properties.getProperty(username);
            if (password != null) {
                adapter = createAdapter(realm, username);
                loadedUsers.put(username, adapter);
            }
        }
        return adapter;
    }

    protected UserModel createAdapter(RealmModel realm, String username) {
        return new AbstractUserAdapter(session, realm, model) {
            @Override
            public String getUsername() {
                return username;
            }
        };
    }

    @Override
    public UserModel getUserById(RealmModel realm, String id) {
        StorageId storageId = new StorageId(id);
        String username = storageId.getExternalId();
        return getUserByUsername(realm, username);
    }

    @Override
    public UserModel getUserByEmail(RealmModel realm, String email) {
        return null;
    }

"f:" + component id + ":" + username

    @Override
    public boolean isConfiguredFor(RealmModel realm, UserModel user, String credentialType) {
        String password = properties.getProperty(user.getUsername());
        return credentialType.equals(PasswordCredentialModel.TYPE) && password != null;
    }

    @Override
    public boolean supportsCredentialType(String credentialType) {
        return credentialType.equals(PasswordCredentialModel.TYPE);
    }

    @Override
    public boolean isValid(RealmModel realm, UserModel user, CredentialInput input) {
        if (!supportsCredentialType(input.getType())) return false;

        String password = properties.getProperty(user.getUsername());
        if (password == null) return false;
        return password.equals(input.getChallengeResponse());
    }

    @Override
    public boolean updateCredential(RealmModel realm, UserModel user, CredentialInput input) {
        if (input.getType().equals(PasswordCredentialModel.TYPE)) throw new ReadOnlyException("user is read only for this update");

        return false;
    }

    @Override
    public void disableCredentialType(RealmModel realm, UserModel user, String credentialType) {

    }

    @Override
    public Stream<String> getDisableableCredentialTypesStream(RealmModel realm, UserModel user) {
        return Stream.empty();
    }

Now that the provider class is complete, we now turn our attention to the provider factory class.

public class PropertyFileUserStorageProviderFactory
                 implements UserStorageProviderFactory<PropertyFileUserStorageProvider> {

    public static final String PROVIDER_NAME = "readonly-property-file";

    @Override
    public String getId() {
        return PROVIDER_NAME;
    }

First thing to notice is that when implementing the UserStorageProviderFactory class, you must pass in the concrete provider class implementation as a template parameter. Here we specify the provider class we defined before: PropertyFileUserStorageProvider.

The getId() method identifies the factory in the runtime and will also be the string shown in the admin console when you want to enable a user storage provider for the realm.

    private static final Logger logger = Logger.getLogger(PropertyFileUserStorageProviderFactory.class);
    protected Properties properties = new Properties();

    @Override
    public void init(Config.Scope config) {
        InputStream is = getClass().getClassLoader().getResourceAsStream("/users.properties");

        if (is == null) {
            logger.warn("Could not find users.properties in classpath");
        } else {
            try {
                properties.load(is);
            } catch (IOException ex) {
                logger.error("Failed to load users.properties file", ex);
            }
        }
    }

    @Override
    public PropertyFileUserStorageProvider create(KeycloakSession session, ComponentModel model) {
        return new PropertyFileUserStorageProvider(session, model, properties);
    }

kc.[sh|bat] start --spi-storage-readonly-property-file-path=/other-users.properties

public void init(Config.Scope config) {
    String path = config.get("path");
    InputStream is = getClass().getClassLoader().getResourceAsStream(path);

    ...
}

    @Override
    public PropertyFileUserStorageProvider create(KeycloakSession session, ComponentModel model) {
        return new PropertyFileUserStorageProvider(session, model, properties);
    }

The class files for our provider implementation should be placed in a jar. You also have to declare the provider factory class within the META-INF/services/org.keycloak.storage.UserStorageProviderFactory file.

org.keycloak.examples.federation.properties.FilePropertiesStorageFactory

To deploy this jar, copy it to the providers/ directory, then run bin/kc.[sh|bat] build.

You enable user storage providers per realm within the User Federation page in the Admin Console.

You will now be able to log in with a user declared in the users.properties file. This user will only be able to view the account page after logging in.

Let’s expand our PropertyFileUserStorageProviderFactory example to allow you to point a provider instance to a specific file on disk.

public class PropertyFileUserStorageProviderFactory
                  implements UserStorageProviderFactory<PropertyFileUserStorageProvider> {

    protected static final List<ProviderConfigProperty> configMetadata;

    static {
        configMetadata = ProviderConfigurationBuilder.create()
                .property().name("path")
                .type(ProviderConfigProperty.STRING_TYPE)
                .label("Path")
                .defaultValue("${jboss.server.config.dir}/example-users.properties")
                .helpText("File path to properties file")
                .add().build();
    }

    @Override
    public List<ProviderConfigProperty> getConfigProperties() {
        return configMetadata;
    }

The ProviderConfigurationBuilder class is a great helper class to create a list of configuration properties. Here we specify a variable named path that is a String type. On the Admin Console configuration page for this provider, this configuration variable is labeled as Path and has a default value of ${jboss.server.config.dir}/example-users.properties. When you hover over the tooltip of this configuration option, it displays the help text, File path to properties file.

The next thing we want to do is to verify that this file exists on disk. We do not want to enable an instance of this provider in the realm unless it points to a valid user property file. To do this, we implement the validateConfiguration() method.

    @Override
    public void validateConfiguration(KeycloakSession session, RealmModel realm, ComponentModel config)
                   throws ComponentValidationException {
        String fp = config.getConfig().getFirst("path");
        if (fp == null) throw new ComponentValidationException("user property file does not exist");
        fp = EnvUtil.replace(fp);
        File file = new File(fp);
        if (!file.exists()) {
            throw new ComponentValidationException("user property file does not exist");
        }
    }

The validateConfiguration() method provides the configuration variable from the ComponentModel to verify if that file exists on disk. Notice that the use of the org.keycloak.common.util.EnvUtil.replace() method. With this method any string that includes ${} will replace that value with a system property value. The ${jboss.server.config.dir} string corresponds to the conf/ directory of our server and is really useful for this example.

Next thing we have to do is remove the old init() method. We do this because user property files are going to be unique per provider instance. We move this logic to the create() method.

    @Override
    public PropertyFileUserStorageProvider create(KeycloakSession session, ComponentModel model) {
        String path = model.getConfig().getFirst("path");

        Properties props = new Properties();
        try {
            InputStream is = new FileInputStream(path);
            props.load(is);
            is.close();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

        return new PropertyFileUserStorageProvider(session, model, props);
    }

This logic is, of course, inefficient as every transaction reads the entire user property file from disk, but hopefully this illustrates, in a simple way, how to hook in configuration variables.

Now that the configuration is enabled, you can set the path variable when you configure the provider in the Admin Console.

Use this procedure to implement adding and removing users from the particular store, we first have to be able to save our properties file to disk.

    public void save() {
        String path = model.getConfig().getFirst("path");
        path = EnvUtil.replace(path);
        try {
            FileOutputStream fos = new FileOutputStream(path);
            properties.store(fos, "");
            fos.close();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

Then, the implementation of the addUser() and removeUser() methods becomes simple.

    public static final String UNSET_PASSWORD="#$!-UNSET-PASSWORD";

    @Override
    public UserModel addUser(RealmModel realm, String username) {
        synchronized (properties) {
            properties.setProperty(username, UNSET_PASSWORD);
            save();
        }
        return createAdapter(realm, username);
    }

    @Override
    public boolean removeUser(RealmModel realm, UserModel user) {
        synchronized (properties) {
            if (properties.remove(user.getUsername()) == null) return false;
            save();
            return true;
        }
    }

Notice that when adding a user we set the password value of the property map to be UNSET_PASSWORD. We do this as we can’t have null values for a property in the property value. We also have to modify the CredentialInputValidator methods to reflect this.

The addUser() method will be called if the provider implements the UserRegistrationProvider interface. If your provider has a configuration switch to turn off adding a user, returning null from this method will skip the provider and call the next one.

    @Override
    public boolean isValid(RealmModel realm, UserModel user, CredentialInput input) {
        if (!supportsCredentialType(input.getType()) || !(input instanceof UserCredentialModel)) return false;

        UserCredentialModel cred = (UserCredentialModel)input;
        String password = properties.getProperty(user.getUsername());
        if (password == null || UNSET_PASSWORD.equals(password)) return false;
        return password.equals(cred.getValue());
    }

Since we can now save our property file, it also makes sense to allow password updates.

    @Override
    public boolean updateCredential(RealmModel realm, UserModel user, CredentialInput input) {
        if (!(input instanceof UserCredentialModel)) return false;
        if (!input.getType().equals(PasswordCredentialModel.TYPE)) return false;
        UserCredentialModel cred = (UserCredentialModel)input;
        synchronized (properties) {
            properties.setProperty(user.getUsername(), cred.getValue());
            save();
        }
        return true;
    }

We can now also implement disabling a password.

    @Override
    public void disableCredentialType(RealmModel realm, UserModel user, String credentialType) {
        if (!credentialType.equals(PasswordCredentialModel.TYPE)) return;
        synchronized (properties) {
            properties.setProperty(user.getUsername(), UNSET_PASSWORD);
            save();
        }

    }

    private static final Set<String> disableableTypes = new HashSet<>();

    static {
        disableableTypes.add(PasswordCredentialModel.TYPE);
    }

    @Override
    public Stream<String> getDisableableCredentialTypes(RealmModel realm, UserModel user) {

        return disableableTypes.stream();
    }

With these methods implemented, you’ll now be able to change and disable the password for the user in the Admin Console.

UserQueryProvider is combination of UserQueryMethodsProvider and UserCountMethodsProvider. Without implementing UserQueryMethodsProvider the Admin Console would not be able to view and manage users that were loaded by our example provider. Let’s look at implementing this interface.

    @Override
    public int getUsersCount(RealmModel realm) {
        return properties.size();
    }

    @Override
    public Stream<UserModel> searchForUserStream(RealmModel realm, String search, Integer firstResult, Integer maxResults) {
        Predicate<String> predicate = "*".equals(search) ? username -> true : username -> username.contains(search);
        return properties.keySet().stream()
                .map(String.class::cast)
                .filter(predicate)
                .skip(firstResult)
                .map(username -> getUserByUsername(realm, username))
                .limit(maxResults);
    }

The first declaration of searchForUserStream() takes a String parameter. In this example, the parameter represents a username that you want to search by. This string can be a substring, which explains the choice of the String.contains() method when doing the search. Notice the use of * to indicate to request a list of all users. The method iterates over the key set of the property file, delegating to getUserByUsername() to load a user. Notice that we are indexing this call based on the firstResult and maxResults parameter. If your external store does not support pagination, you will have to do similar logic.

    @Override
    public Stream<UserModel> searchForUserStream(RealmModel realm, Map<String, String> params, Integer firstResult, Integer maxResults) {
        // only support searching by username
        String usernameSearchString = params.get("username");
        if (usernameSearchString != null)
            return searchForUserStream(realm, usernameSearchString, firstResult, maxResults);

        // if we are not searching by username, return all users
        return searchForUserStream(realm, "*", firstResult, maxResults);
    }

The searchForUserStream() method that takes a Map parameter can search for a user based on first, last name, username, and email. Only usernames are stored, so the search is based only on usernames except when the Map parameter does not contain the username attribute. In this case, all users are returned. In that situation, the searchForUserStream(realm, search, firstResult, maxResults) is used.

    @Override
    public Stream<UserModel> getGroupMembersStream(RealmModel realm, GroupModel group, Integer firstResult, Integer maxResults) {
        return Stream.empty();
    }

    @Override
    public Stream<UserModel> searchForUserByUserAttributeStream(RealmModel realm, String attrName, String attrValue) {
        return Stream.empty();
    }

Groups or attributes are not stored, so the other methods return an empty stream.

In our PropertyFileUserStorageProvider example, we just need a simple change to our provider to use the AbstractUserAdapterFederatedStorage.

    protected UserModel createAdapter(RealmModel realm, String username) {
        return new AbstractUserAdapterFederatedStorage(session, realm, model) {
            @Override
            public String getUsername() {
                return username;
            }

            @Override
            public void setUsername(String username) {
                String pw = (String)properties.remove(username);
                if (pw != null) {
                    properties.put(username, pw);
                    save();
                }
            }
        };
    }

We instead define an anonymous class implementation of AbstractUserAdapterFederatedStorage. The setUsername() method makes changes to the properties file and saves it.

If you remember earlier in this chapter, we discussed how querying for a user worked. Local storage is queried first, if the user is found there, then the query ends. This is a problem for our above implementation as we want to proxy the local UserModel so that we can keep usernames in sync. The User Storage SPI has a callback for whenever a linked local user is loaded from the local database.

package org.keycloak.storage.user;
public interface ImportedUserValidation {
    /**
     * If this method returns null, then the user in local storage will be removed
     *
     * @param realm
     * @param user
     * @return null if user no longer valid
     */
    UserModel validate(RealmModel realm, UserModel user);
}

Whenever a linked local user is loaded, if the user storage provider class implements this interface, then the validate() method is called. Here you can proxy the local user passed in as a parameter and return it. That new UserModel will be used. You can also optionally do a check to see if the user still exists in the external store. If validate() returns null, then the local user will be removed from the database.

With the import strategy you can see that it is possible for the local user copy to get out of sync with external storage. For example, maybe a user has been removed from the external store. The User Storage SPI has an additional interface you can implement to deal with this, org.keycloak.storage.user.ImportSynchronization:

package org.keycloak.storage.user;

public interface ImportSynchronization {
    SynchronizationResult sync(KeycloakSessionFactory sessionFactory, String realmId, UserStorageProviderModel model);
    SynchronizationResult syncSince(Date lastSync, KeycloakSessionFactory sessionFactory, String realmId, UserStorageProviderModel model);
}

This interface is implemented by the provider factory. Once this interface is implemented by the provider factory, the administration console management page for the provider shows additional options. You can manually force a synchronization by clicking a button. This invokes the ImportSynchronization.sync() method. Also, additional configuration options are displayed that allow you to automatically schedule a synchronization. Automatic synchronizations invoke the syncSince() method.

You can access the user cache by calling KeycloakSession.getProvider(UserCache.class).

/**
 * All these methods effect an entire cluster of Keycloak instances.
 *
 * @author <a href="mailto:bill@burkecentral.com">Bill Burke</a>
 * @version $Revision: 1 $
 */
public interface UserCache extends UserProvider {
    /**
     * Evict user from cache.
     *
     * @param user
     */
    void evict(RealmModel realm, UserModel user);

    /**
     * Evict users of a specific realm
     *
     * @param realm
     */
    void evict(RealmModel realm);

    /**
     * Clear cache entirely.
     *
     */
    void clear();
}

There are methods for evicting specific users, users contained in a specific realm, or the entire cache.

You might want to cache additional information that is specific to your provider implementation. The User Storage SPI has a callback whenever a user is cached: org.keycloak.models.cache.OnUserCache.

public interface OnUserCache {
    void onCache(RealmModel realm, CachedUserModel user, UserModel delegate);
}

Your provider class should implement this interface if it wants this callback. The UserModel delegate parameter is the UserModel instance returned by your provider. The CachedUserModel is an expanded UserModel interface. This is the instance that is cached locally in local storage.

public interface CachedUserModel extends UserModel {

    /**
     * Invalidates the cache for this user and returns a delegate that represents the actual data provider
     *
     * @return
     */
    UserModel getDelegateForUpdate();

    boolean isMarkedForEviction();

    /**
     * Invalidate the cache for this model
     *
     */
    void invalidate();

    /**
     * When was the model was loaded from database.
     *
     * @return
     */
    long getCacheTimestamp();

    /**
     * Returns a map that contains custom things that are cached along with this model.  You can write to this map.
     *
     * @return
     */
    ConcurrentHashMap getCachedWith();
}

This CachedUserModel interface allows you to evict the user from the cache and get the provider UserModel instance. The getCachedWith() method returns a map that allows you to cache additional information pertaining to the user. For example, credentials are not part of the UserModel interface. If you wanted to cache credentials in memory, you would implement OnUserCache and cache your user’s credentials using the getCachedWith() method.

On the administration console management page for your user storage provider, you can specify a unique cache policy.

The earlier User Federation SPI required you to create a local copy of a user in the Red Hat build of Keycloak’s database and import information from your external store to the local copy. However, this is no longer a requirement. You can still port your earlier provider as-is, but you should consider whether a non-import strategy might be a better approach.

Advantages of the import strategy:

There are some obvious disadvantages though to using an import strategy:

The first thing to notice is that UserFederationProvider was a complete interface. You implemented every method in this interface. However, UserStorageProvider has instead broken up this interface into multiple capability interfaces that you implement as needed.

UserFederationProvider.getUserByUsername() and getUserByEmail() have exact equivalents in the new SPI. The difference between the two is how you import. If you are going to continue with an import strategy, you no longer call KeycloakSession.userStorage().addUser() to create the user locally. Instead you call KeycloakSession.userLocalStorage().addUser(). The userStorage() method no longer exists.

The UserFederationProvider.validateAndProxy() method has been moved to an optional capability interface, ImportedUserValidation. You want to implement this interface if you are porting your earlier provider as-is. Also note that in the earlier SPI, this method was called every time the user was accessed, even if the local user is in the cache. In the later SPI, this method is only called when the local user is loaded from local storage. If the local user is cached, then the ImportedUserValidation.validate() method is not called at all.

The UserFederationProvider.isValid() method no longer exists in the later SPI.

The UserFederationProvider methods synchronizeRegistrations(), registerUser(), and removeUser() have been moved to the UserRegistrationProvider capability interface. This new interface is optional to implement so if your provider does not support creating and removing users, you don’t have to implement it. If your earlier provider had switch to toggle support for registering new users, this is supported in the new SPI, returning null from UserRegistrationProvider.addUser() if the provider doesn’t support adding users.

The earlier UserFederationProvider methods centered around credentials are now encapsulated in the CredentialInputValidator and CredentialInputUpdater interfaces, which are also optional to implement depending on if you support validating or updating credentials. Credential management used to exist in UserModel methods. These also have been moved to the CredentialInputValidator and CredentialInputUpdater interfaces. One thing to note that if you do not implement the CredentialInputUpdater interface, then any credentials provided by your provider can be overridden locally in Red Hat build of Keycloak storage. So if you want your credentials to be read-only, implement the CredentialInputUpdater.updateCredential() method and return a ReadOnlyException.

The UserFederationProvider query methods such as searchByAttributes() and getGroupMembers() are now encapsulated in an optional interface UserQueryProvider. If you do not implement this interface, then users will not be viewable in the admin console. You’ll still be able to log in though.

The synchronization methods in the earlier SPI are now encapsulated within an optional ImportSynchronization interface. If you have implemented synchronization logic, then have your new UserStorageProviderFactory implement the ImportSynchronization interface.

The User Storage SPI instances are stored in a different set of relational tables. Red Hat build of Keycloak automatically runs a migration script. If any earlier User Federation providers are deployed for a realm, they are converted to the later storage model as is, including the id of the data. This migration will only happen if a User Storage provider exists with the same provider ID (i.e., "ldap", "kerberos") as the earlier User Federation provider.

So, knowing this there are different approaches you can take.

If you have decided to get rid of the import strategy and rewrite your User Storage provider, we suggest that you remove the earlier provider before upgrading Red Hat build of Keycloak. This will remove linked local imported copies of any user you imported.