Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 37. Performance Tuning for Bulk Provisioning of Entries
Adding a large number of entries using the usual workflow, such as Chapter 11, Managing User Accounts for adding users, can be very slow. This chapter describes how to tune the process to ensure the provisioning is completed as quickly as possible.
As part of the procedure:
- Identity Management (IdM) reads entries to be provisioned from an LDIF file and then imports them to the target IdM LDAP instance.
- The administrator sets custom values for certain attributes, such as cache sizes, and disables the MemberOf and Schema Compatibility plug-ins. The procedure includes running the
fixup-memberof.pl
plug-in on the provisioned entries to compensate for disabling MemberOf.
This procedure has been designed and tested to provision the following entry types: user, user group, host, host group, sudo rules, and host-based access control (HBAC) rules.
Recommendations and Prerequisites for Bulk Provisioning
Recommendations:
- When provisioning a large number of entries (10,000 or more), do not allow any LDAP client to access the server on which the entries are provisioned or to rely on the information from the server. For example, you can achieve this by disabling ports 389 and 636 on the server and using LDAPI to work over Unix sockets.Reason: The MemberOf plug-in is disabled on the server, which means that membership information on the server is not valid.
- Stop applications that are not required to be running during the provisioning.Reason: This helps free as much memory on the machine as possible. The free memory will be used by the file system cache, thus improving the performance of the provisioning.Note that the procedure below already includes steps to stop the IdM services, and restart only the Directory Server (DS) instance. IdM services, especially
tomcat
, consume a lot of memory, but are not used during the provisioning. - Run the procedure on a fresh IdM deployment with only one server. Create replicas only after the provisioning has been completed.Reason: The provisioning throughput is much faster than replication. In a deployment with more that one server, information on the replicas would become significantly outdated.
Prerequisites:
- Generate an LDIF file containing the entries you want to provision. For example, if you are migrating an existing IdM deployment, create the LDIF file by exporting all the entries using the
ldapsearch
utility.For details on the LDIF format, see About the LDIF File Format in the Red Hat Directory Server 10 Administration Guide.
Backing up the Current DS Tuning Parameter Values
- Retrieve the current values for the DS tuning parameters:
- the database cache size and database locks:
# ldapsearch -D "cn=directory manager" -w secret -b "cn=config,cn=ldbm database,cn=plugins,cn=config" nsslapd-dbcachesize nsslapd-db-locks ... nsslapd-dbcachesize: 10000000 nsslapd-db-locks: 50000 ...
- the entry cache size and DN cache size:
# ldapsearch -D "cn=directory manager" -w secret -b "cn=userRoot,cn=ldbm database,cn=plugins,cn=config" nsslapd-cachememsize nsslapd-dncachememsize ... nsslapd-cachememsize: 10485760 nsslapd-dncachememsize: 10485760 ...
- Make note of the obtained values. You will reset the parameters back to these values after you finish the provisioning.
Adjusting the Database, Domain Entry, and DN Cache Size
For the database cache size:
- Determine the required value.The recommended value is typically between 200 MB and 500 MB. The value appropriate for your use case depends on the memory available on your system:
- More than 8 GB of memory
500 MB - 8 GB - 4 GB of memory
200 MB - Less than 4 GB of memory
100 MB
- Set the determined value by using this template:
dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: db_cache_size_in_bytes
For an example of modifying LDAP attributes using theldapmodify
utility, see Example 37.1, “Usingldapmodify
to Change an LDAP Attribute”.
Example 37.1. Using ldapmodify
to Change an LDAP Attribute
- Run the
ldapmodify
command, and then add the statements to modify the attribute value. For example:# ldapmodify -D "cn=directory manager" -w secret -x dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: 200000000
- Press Ctrl+D to confirm and send the changes to the server. If the operation finishes successfully, the following message is displayed:
modifying entry "cn=config,cn=ldbm database,cn=plugins,cn=config"
For the domain entry cache size:
- Determine the required value.The recommended value is between 100 MB and 400 MB. The appropriate value depends on the memory available on your system:
- More than 4 GB of memory
400 MB - 2 GB - 4 GB of memory
200 MB - Less than 2 GB of memory
100 MB
If you are provisioning a large static group, it is recommended that the entry cache is large enough to fit all entries: groups and members. - Set the determined value by using this template:
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-cachememsize nsslapd-cachememsize: entry_cache_size_in_bytes
For the domain name (DN) cache size:
- For the best possible performance, it is recommended that the DN cache fits all the DNs of the provisioned entries. To estimate the value appropriate for your use case:
- Determine the number of all DN entries in the file. The DN entries are on lines starting with
dn:
. For example, using# grep
,sed
, andwc
:# grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -l 92200
- Determine the size of all DN entry strings in the LDIF file.
# grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -c 9802460
- Get the average DN size: divide the size of all DN entry strings by the number of all the DN entries in the file.For example: 9,802,460 / 92,200 ≈ 106
- Get the average memory size: multiple the average DN size by 2, and then add 32 to the result.For example: (106 * 2) + 32 = 244
- Get the appropriate DN cache size: multiply the average memory size by the total number of DN entries in the LDIF file.For example: 244 * 92,200 = 22,496,800
- Set the determined value by using this template:
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify Replace: nsslapd-dncachememsize Nsslapd-dncachememsize: dn_cache_size
Disabling Unnecessary Services and Adjusting Database Locks
- Disable the MemberOf and Schema Compatibility plug-ins:
dn: cn=MemberOf Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
dn: cn=Schema Compatibility,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
Disabling MemberOf significantly speeds up the provisioning. Disabling Schema Compatibility also helps reduce the duration of the operation.For an example of modifying LDAP attributes using theldapmodify
utility, see Example 37.1, “Usingldapmodify
to Change an LDAP Attribute”. - If no replicas are installed in your topology (as recommended in the section called “Recommendations and Prerequisites for Bulk Provisioning”), disable the Content Synchronization and Retro Changelog plug-ins:
dn: cn=Content Synchronization,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
dn: cn=Retro Changelog Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
Disabling these additional plug-ins helps improve the performance of the provisioning. - Stop the IdM server. This also stops the DS instance.
# ipactl stop
Stopping DS is required to set the number of database locks in the next step. You will restart it again later. - Adjust the number of database locks. The appropriate value equals half the number of provisioned entries.
- the minimum value is 10,000
- the maximum value is 200,000
Because DS is stopped, you must set the value by modifying the/etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
file:dn: cn=config,cn=ldbm database,cn=plugins,cn=config ... nsslapd-db-locks: db_lock_number
IdM accesses a large number of database pages when computing membership. The more pages it accesses, the more locks are required for the provisioning. - Start DS:
# systemctl start dirsrv.target
Importing the Entries
To import the new entries from the LDIF file to the IdM LDAP instance. For example, using the
ldapadd
utility:
# ldapadd -D "binddn" -y password_file -f ldif_file
For details on using
ldapadd
, see the ldapadd(1) man page.
Re-enabling the Disabled Services and Restoring the Original Attribute Values
- Enable MemberOf:
dn: cn=MemberOf Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
For an example of modifying LDAP attributes using theldapmodify
utility, see Example 37.1, “Usingldapmodify
to Change an LDAP Attribute”. - Restart DS:
# systemctl restart dirsrv.target
Restarting DS at this point is required because you enabled MemberOf in the previous step. - Run the
fixup-memberof.pl
script with the(objectClass=*)
filter to regenerate and update thememberOf
attribute on all provisioned entries. For example:# fixup-memberof.pl -D "cn=directory manager" -j password_file -Z server_id -b "suffix" -f "(objectClass=*)" -P LDAP
Runningfixup-memberof.pl
is necessary because the MemberOf plug-in was disabled when you imported the entries. To be able to continue with the provisioning, the script must complete successfully.For details onfixup-memberof.pl
, see the fixup-memberof.pl(8) man page. - Enable the Schema Compatibility plug-in:
dn: cn=Schema Compatibility,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
- If you disabled the Content Synchronization and Retro Changelog plug-ins in the section called “Disabling Unnecessary Services and Adjusting Database Locks”, re-enable them:
dn: cn=Content Synchronization,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
dn: cn=Retro Changelog Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
- Restore the original values for the database cache, entry cache, and DN cache size that you backed up in the section called “Backing up the Current DS Tuning Parameter Values”:
dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: backup_db_cache_size dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify Replace: nsslapd-dncachememsize Nsslapd-dncachememsize: backup_dn_cache_size - replace: nsslapd-cachememsize nsslapd-cachememsize: backup_entry_cache_size
- Stop DS:
# systemctl stop dirsrv.target
- Restore the original value for database locks that you backed up in the section called “Backing up the Current DS Tuning Parameter Values”. Because DS is stopped, you must set the value by modifying the
/etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
file:dn: cn=config,cn=ldbm database,cn=plugins,cn=config ... nsslapd-db-locks: backup_db_lock_number
- Start the IdM server:
# ipactl start
This starts all IdM services, including DS.