Chapter 9. Tuning SSSD performance for large IdM-AD trust deployments
Retrieving user and group information is a very data-intensive operation for the System Security Services Daemon (SSSD), especially in an IdM deployment with a trust to a large Active Directory (AD) domain. You can improve this performance by adjusting which information SSSD retrieves from identity providers and for how long.
9.1. Tuning SSSD in IdM servers for large IdM-AD trust deployments
This procedure applies tuning options to the configuration of the SSSD service in an IdM server to improve its response time when retrieving information from a large AD environment.
Prerequisites
-
You need
root
permissions to edit the/etc/sssd/sssd.conf
configuration file.
Procedure
-
Open the
/etc/sssd/sssd.conf
configuration file in a text editor. Add the following options to the
[domain]
section for your Identity Management (IdM) domain:[domain/idm.example.com] ignore_group_members = true subdomain_inherit = ignore_group_members ...
NoteSettings listed in the
subdomain_inherit
options apply to both the main (IdM) domain as well as the trusted AD domain(s).-
Save and close the
/etc/sssd/sssd.conf
file on the server. Restart the SSSD service to load the configuration changes.
[root@client ~]# systemctl restart sssd
9.2. Tuning the config timeout for the ipa-extdom plugin on IdM servers
IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom
plugin to receive information about AD users and groups, and that information is forwarded to the requesting client.
The ipa-extdom
plug-in sends a request to SSSD for the data about AD users. If the information is not in the SSSD cache, SSSD requests the data from an AD domain controller (DC). You can adjust the config timeout value, which defines how long the ipa-extdom
plug-in waits for a reply from SSSD before the plug-in cancels the connection and returns a timeout error to the caller. The default value is 10000 milliseconds (10 seconds).
The following example adjusts the config timeout to 20 seconds (20000 milliseconds).
Exercise caution when adjusting the config timeout:
- If you set a value that is too small, such as 500 milliseconds, SSSD might not have enough time to reply and requests will always return a timeout.
- If you set a value that is too large, such as 30000 milliseconds (30 seconds), a single request might block the connection to SSSD for this amount of time. Because only one thread can connect to SSSD at a time, all other requests from the plug-in have to wait.
- If there are many requests sent by IdM clients, they can block all available workers configured for the Directory Server on the IdM server. As a consequence, the server might not be able to reply to any kind of request for some time.
Only change the config timeout in the following situations:
- If IdM clients frequently receive timeout errors before their own search timeout is reached when requesting information about AD users and groups, the config timeout value is too small.
-
If the Directory Server on the IdM server is often locked and the
pstack
utility reports that many or all worker threads are handlingipa-extdom
requests at this time, the value is too large.
Prerequisites
- The LDAP Directory Manager password
Procedure
Use the following command to adjust the config timeout to 20000 milliseconds:
# ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxNssTimeout ipaExtdomMaxNssTimeout: 20000
9.3. Tuning the maximum buffer size for the ipa-extdom plugin on IdM servers
IdM clients cannot receive information about users and groups from Active Directory (AD) directly, so IdM servers use the ipa-extdom
plugin to receive information about AD users and groups, and that information is forwarded to the requesting client.
You can tune the maximum buffer size for the ipa-extdom
plugin, which adjusts the size of the buffer where SSSD can store the data it receives. If the buffer is too small, SSSD returns an ERANGE
error and the plug-in retries the request with a larger buffer. The default buffer size is 134217728 bytes (128 MB).
The following example adjusts the maximum buffer size to 256 MB (268435456 bytes).
Prerequisites
- The LDAP Directory Manager password
Procedure
Use the following command to set the maximum buffer size to 268435456 bytes:
# ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxNssBufSize ipaExtdomMaxNssBufSize: 268435456
9.4. Tuning the maximum number of instances for the ipa-extdom plugin on IdM servers
As IdM clients cannot receive information about users and groups from Active Directory (AD) directly, IdM servers use the ipa-extdom
plugin to receive information about AD users and groups and then forward this information to the requesting client.
By default, the ipa-extdom
plugin is configured to use up to 80% of the LDAP worker threads to handle requests from IdM clients. If the SSSD service on an IdM client has requested a large amount of information about AD trust users and groups, this operation can halt the LDAP service if it uses most of the LDAP threads. If you experience these issues, you might see similar errors in the SSSD log file for your AD domain, /var/log/sssd/sssd__your-ad-domain-name.com_.log
:
(2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(2022-05-22 5:00:13): [be[ad.example.com]] [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Server is busy(51), Too many extdom instances running.
You can adjust the maximum number of ipa-extdom
instances by setting the value for the ipaExtdomMaxInstances
option, which must be an integer larger than 0 and less than the total number of worker threads.
Prerequisites
- The LDAP Directory Manager password
Procedure
Retrieve the total number of worker threads:
# ldapsearch -xLLLD cn=directory\ manager -W -b cn=config -s base nsslapd-threadnumber Enter LDAP Password: dn: cn=config nsslapd-threadnumber: 16
This means that the current value for
ipaExtdomMaxInstances
is 13.Adjust the maximum number of instances. This example changes the value to 14:
# ldapmodify -D "cn=directory manager" -W dn: cn=ipa_extdom_extop,cn=plugins,cn=config changetype: modify replace: ipaExtdomMaxInstances ipaExtdomMaxInstances: 14
Retrieve the current value of
ipaExtdomMaxInstances
:# ldapsearch -xLLLD "cn=directory manager" -W -b "cn=ipa_extdom_extop,cn=plugins,cn=config" |grep ipaextdommaxinstances Enter LDAP Password: ipaextdommaxinstances: 14
-
Monitor the IdM directory server’s performance and if it does not improve, repeat this procedure and adjust the value of the
ipaExtdomMaxInstances
variable.
9.5. Tuning SSSD in IdM clients for large IdM-AD trust deployments
This procedure applies tuning options to SSSD service configuration in an IdM client to improve its response time when retrieving information from a large AD environment.
Prerequisites
-
You need
root
permissions to edit the/etc/sssd/sssd.conf
configuration file.
Procedure
Determine the number of seconds a single un-cached login takes.
Clear the SSSD cache on the IdM client
client.example.com
.[root@client ~]# sss_cache -E
Measure how long it takes to log in as an AD user with the
time
command. In this example, from the IdM clientclient.example.com
, log into the same host as the userad-user
from thead.example.com
AD domain.[root@client ~]# time ssh ad-user@ad.example.com@client.example.com
Type in the password as soon as possible.
Password: Last login: Sat Jan 23 06:29:54 2021 from 10.0.2.15 [ad-user@ad.example.com@client ~]$
Log out as soon as possible to display elapsed time. In this example, a single un-cached login takes about
9
seconds.[ad-user@ad.example.com@client /]$ exit logout Connection to client.example.com closed. real 0m8.755s user 0m0.017s sys 0m0.013s
-
Open the
/etc/sssd/sssd.conf
configuration file in a text editor. Add the following options to the
[domain]
section for your Active Directory domain. Set thepam_id_timeout
andkrb5_auth_timeout
options to the number of seconds an un-cached login takes. If you do not already have a domain section for your AD domain, create one.[domain/example.com/ad.example.com] krb5_auth_timeout = 9 ldap_deref_threshold = 0 ...
Add the following option to the
[pam]
section:[pam] pam_id_timeout = 9
-
Save and close the
/etc/sssd/sssd.conf
file on the server. Restart the SSSD service to load the configuration changes.
[root@client ~]# systemctl restart sssd
9.6. Mounting the SSSD cache in tmpfs
The System Security Services Daemon (SSSD) constantly writes LDAP objects to its cache. These internal SSSD transactions write data to disk, which is much slower than reading and writing from Random-Access Memory (RAM).
To improve this performance, mount the SSSD cache in RAM.
Considerations
- Cached information does not persist after a reboot if the SSSD cache is in RAM.
- It is safe to perform this change on IdM servers, as the SSSD instance on an IdM server cannot lose connectivity with the Directory Server on the same host.
- If you perform this adjustment on an IdM client and it loses connectivity to IdM servers, users will not be able to authenticate after a reboot until you reestablish connectivity.
Prerequisites
-
You need
root
permissions to edit the/etc/fstab
configuration file.
Procedure
Create a
tmpfs
temporary filesystem:On RHEL 8.6 and later, confirm that the SSSD user owns the
config.ldb
file:# ls -al /var/lib/sss/db/config.ldb -rw-------. 1 sssd sssd 1286144 Jun 8 16:41 /var/lib/sss/db/config.ldb
Add the following entry to the
/etc/fstab
file as a single line:tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,uid=sssd,gid=sssd,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0
On RHEL 8 versions lower than 8.6, the
config.ldb
file is owned by theroot
user:# ls -al /var/lib/sss/db/config.ldb -rw-------. 1 root root 1286144 Jun 8 14:15 /var/lib/sss/db/config.ldb
Add the following entry to the
/etc/fstab
file as a single line:tmpfs /var/lib/sss/db/ tmpfs size=300M,mode=0700,rootcontext=system_u:object_r:sssd_var_lib_t:s0 0 0
This example creates a 300MB cache. Tune the
size
parameter according to your IdM and AD directory size, estimating 100 MBs per 10,000 LDAP entries.
Mount the new SSSD cache directory.
[root@host ~]# mount /var/lib/sss/db/
Restart SSSD to reflect this configuration change.
[root@host ~]# systemctl restart sssd
9.7. Options in sssd.conf
for tuning IdM servers and clients for large IdM-AD trust deployments
You can use the following options in the /etc/sssd/sssd.conf
configuration file to tune the performance of SSSD in IdM servers and clients when you have a large IdM-AD trust deployment.
9.7.1. Tuning options for IdM servers
- ignore_group_members
Knowing which groups a user belongs to, as opposed to all the users that belong to a group, is important when authenticating and authorizing a user. When
ignore_group_members
is set totrue
, SSSD only retrieves information about the group objects themselves and not their members, providing a significant performance boost.NoteThe
id user@ad-domain.com
command still returns the correct list of groups, butgetent group ad-group@ad-domain.com
returns an empty list.Default value
false
Recommended value
true
NoteYou should not set this option to
true
when the deployment involves an IdM server with the compat tree.- subdomain_inherit
With the
subdomain_inherit
option, you can apply theignore_group_members
setting to the trusted AD domains’ configuration. Settings listed in thesubdomain_inherit
options apply to both the main (IdM) domain as well as the AD subdomain.Default value
none
Recommended value
subdomain_inherit = ignore_group_members
9.7.2. Tuning options for IdM clients
- pam_id_timeout
This parameter controls how long results from a PAM session are cached, to avoid excessive round-trips to the identity provider during an identity lookup. The default value of
5
seconds might not be enough in environments where complex group memberships are populated on the IdM Server and IdM client side. Red Hat recommends settingpam_id_timeout
to the number of seconds a single un-cached login takes.Default value
5
Recommended value
the number of seconds a single un-cached login takes
- krb5_auth_timeout
Increasing
krb5_auth_timeout
allows more time to process complex group information in environments where users are members of a large number of groups. Red Hat recommends setting this value to the number of seconds a single un-cached login takes.Default value
6
Recommended value
the number of seconds a single un-cached login takes
- ldap_deref_threshold
A dereference lookup is a means of fetching all group members in a single LDAP call. The
ldap_deref_threshold
value specifies the number of group members that must be missing from the internal cache to trigger a dereference lookup. If less members are missing, they are looked up individually. Dereference lookups may take a long time in large environments and decrease performance. To disable dereference lookups, set this option to0
.Default value
10
Recommended value
0