Search
SupportConsoleDevelopersStart a trialContact

Chapter 37. Performance Tuning for Bulk Provisioning of Entries

download PDF
Adding a large number of entries using the usual workflow, such as Chapter 11, Managing User Accounts for adding users, can be very slow. This chapter describes how to tune the process to ensure the provisioning is completed as quickly as possible.
As part of the procedure:
  • Identity Management (IdM) reads entries to be provisioned from an LDIF file and then imports them to the target IdM LDAP instance.
  • The administrator sets custom values for certain attributes, such as cache sizes, and disables the MemberOf and Schema Compatibility plug-ins. The procedure includes running the fixup-memberof.pl plug-in on the provisioned entries to compensate for disabling MemberOf.
This procedure has been designed and tested to provision the following entry types: user, user group, host, host group, sudo rules, and host-based access control (HBAC) rules.

Recommendations and Prerequisites for Bulk Provisioning

Recommendations:
  • When provisioning a large number of entries (10,000 or more), do not allow any LDAP client to access the server on which the entries are provisioned or to rely on the information from the server. For example, you can achieve this by disabling ports 389 and 636 on the server and using LDAPI to work over Unix sockets.
    Reason: The MemberOf plug-in is disabled on the server, which means that membership information on the server is not valid.
  • Stop applications that are not required to be running during the provisioning.
    Reason: This helps free as much memory on the machine as possible. The free memory will be used by the file system cache, thus improving the performance of the provisioning.
    Note that the procedure below already includes steps to stop the IdM services, and restart only the Directory Server (DS) instance. IdM services, especially tomcat, consume a lot of memory, but are not used during the provisioning.
  • Run the procedure on a fresh IdM deployment with only one server. Create replicas only after the provisioning has been completed.
    Reason: The provisioning throughput is much faster than replication. In a deployment with more that one server, information on the replicas would become significantly outdated.
Prerequisites:
  • Generate an LDIF file containing the entries you want to provision. For example, if you are migrating an existing IdM deployment, create the LDIF file by exporting all the entries using the ldapsearch utility.
    For details on the LDIF format, see About the LDIF File Format in the Red Hat Directory Server 10 Administration Guide.

Backing up the Current DS Tuning Parameter Values

  1. Retrieve the current values for the DS tuning parameters:
    • the database cache size and database locks: 
      # ldapsearch -D "cn=directory manager" -w secret -b "cn=config,cn=ldbm database,cn=plugins,cn=config" nsslapd-dbcachesize nsslapd-db-locks

...
nsslapd-dbcachesize: 10000000
nsslapd-db-locks: 50000
...
    • the entry cache size and DN cache size: 
      # ldapsearch -D "cn=directory manager" -w secret -b "cn=userRoot,cn=ldbm database,cn=plugins,cn=config" nsslapd-cachememsize nsslapd-dncachememsize

...
nsslapd-cachememsize: 10485760
nsslapd-dncachememsize: 10485760
...
  2. Make note of the obtained values. You will reset the parameters back to these values after you finish the provisioning.

Adjusting the Database, Domain Entry, and DN Cache Size

For the database cache size:
  1. Determine the required value.
    The recommended value is typically between 200 MB and 500 MB. The value appropriate for your use case depends on the memory available on your system:
    • More than 8 GB of memory 500 MB
    • 8 GB - 4 GB of memory 200 MB
    • Less than 4 GB of memory 100 MB
  2. Set the determined value by using this template: 
    dn: cn=config,cn=ldbm database,cn=plugins,cn=config
changetype: modify
replace: nsslapd-dbcachesize
nsslapd-dbcachesize: db_cache_size_in_bytes
    For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, “Using ldapmodify to Change an LDAP Attribute”.

Example 37.1. Using ldapmodify to Change an LDAP Attribute

  1. Run the ldapmodify command, and then add the statements to modify the attribute value. For example: 
    # ldapmodify -D "cn=directory manager" -w secret -x
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
changetype: modify
replace: nsslapd-dbcachesize
nsslapd-dbcachesize: 200000000
  2. Press Ctrl+D to confirm and send the changes to the server. If the operation finishes successfully, the following message is displayed: 
    modifying entry "cn=config,cn=ldbm database,cn=plugins,cn=config"
For the domain entry cache size:
  1. Determine the required value.
    The recommended value is between 100 MB and 400 MB. The appropriate value depends on the memory available on your system:
    • More than 4 GB of memory 400 MB
    • 2 GB - 4 GB of memory 200 MB
    • Less than 2 GB of memory 100 MB
    If you are provisioning a large static group, it is recommended that the entry cache is large enough to fit all entries: groups and members.
  2. Set the determined value by using this template: 
    dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
changetype: modify
replace: nsslapd-cachememsize
nsslapd-cachememsize: entry_cache_size_in_bytes
For the domain name (DN) cache size:
  1. For the best possible performance, it is recommended that the DN cache fits all the DNs of the provisioned entries. To estimate the value appropriate for your use case:
    1. Determine the number of all DN entries in the file. The DN entries are on lines starting with dn: . For example, using # grep, sed, and wc: 
      # grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -l
92200
    2. Determine the size of all DN entry strings in the LDIF file. 
      # grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -c
9802460
    3. Get the average DN size: divide the size of all DN entry strings by the number of all the DN entries in the file.
      For example: 9,802,460 / 92,200 ≈ 106
    4. Get the average memory size: multiple the average DN size by 2, and then add 32 to the result.
      For example: (106 * 2) + 32 = 244
    5. Get the appropriate DN cache size: multiply the average memory size by the total number of DN entries in the LDIF file.
      For example: 244 * 92,200 = 22,496,800
  2. Set the determined value by using this template: 
    dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
changetype: modify
Replace: nsslapd-dncachememsize
Nsslapd-dncachememsize: dn_cache_size

Disabling Unnecessary Services and Adjusting Database Locks

  1. Disable the MemberOf and Schema Compatibility plug-ins: 
    dn: cn=MemberOf Plugin,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off
    dn: cn=Schema Compatibility,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off
    Disabling MemberOf significantly speeds up the provisioning. Disabling Schema Compatibility also helps reduce the duration of the operation.
    For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, “Using ldapmodify to Change an LDAP Attribute”.
  2. If no replicas are installed in your topology (as recommended in the section called “Recommendations and Prerequisites for Bulk Provisioning”), disable the Content Synchronization and Retro Changelog plug-ins: 
    dn: cn=Content Synchronization,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off
    dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: off
    Disabling these additional plug-ins helps improve the performance of the provisioning.
  3. Stop the IdM server. This also stops the DS instance. 
    # ipactl stop
    Stopping DS is required to set the number of database locks in the next step. You will restart it again later.
  4. Adjust the number of database locks. The appropriate value equals half the number of provisioned entries.
    • the minimum value is 10,000
    • the maximum value is 200,000
    Because DS is stopped, you must set the value by modifying the /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif file: 
    dn: cn=config,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-db-locks: db_lock_number
    IdM accesses a large number of database pages when computing membership. The more pages it accesses, the more locks are required for the provisioning.
  5. Start DS: 
    # systemctl start dirsrv.target

Importing the Entries

To import the new entries from the LDIF file to the IdM LDAP instance. For example, using the ldapadd utility: 
# ldapadd -D "binddn" -y password_file -f ldif_file
For details on using ldapadd, see the ldapadd(1) man page.

Re-enabling the Disabled Services and Restoring the Original Attribute Values

  1. Enable MemberOf: 
    dn: cn=MemberOf Plugin,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on
    For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, “Using ldapmodify to Change an LDAP Attribute”.
  2. Restart DS: 
    # systemctl restart dirsrv.target
    Restarting DS at this point is required because you enabled MemberOf in the previous step.
  3. Run the fixup-memberof.pl script with the (objectClass=*) filter to regenerate and update the memberOf attribute on all provisioned entries. For example: 
    # fixup-memberof.pl -D "cn=directory manager" -j password_file -Z server_id -b "suffix" -f "(objectClass=*)" -P LDAP
    Running fixup-memberof.pl is necessary because the MemberOf plug-in was disabled when you imported the entries. To be able to continue with the provisioning, the script must complete successfully.
    For details on fixup-memberof.pl, see the fixup-memberof.pl(8) man page.
  4. Enable the Schema Compatibility plug-in: 
    dn: cn=Schema Compatibility,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on
  5. If you disabled the Content Synchronization and Retro Changelog plug-ins in the section called “Disabling Unnecessary Services and Adjusting Database Locks”, re-enable them: 
    dn: cn=Content Synchronization,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on
    dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginEnabled
nsslapd-pluginEnabled: on
  6. Restore the original values for the database cache, entry cache, and DN cache size that you backed up in the section called “Backing up the Current DS Tuning Parameter Values”: 
    dn: cn=config,cn=ldbm database,cn=plugins,cn=config
changetype: modify
replace: nsslapd-dbcachesize
nsslapd-dbcachesize: backup_db_cache_size

dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
changetype: modify
Replace: nsslapd-dncachememsize
Nsslapd-dncachememsize: backup_dn_cache_size
-
replace: nsslapd-cachememsize
nsslapd-cachememsize: backup_entry_cache_size
  7. Stop DS: 
    # systemctl stop dirsrv.target
  8. Restore the original value for database locks that you backed up in the section called “Backing up the Current DS Tuning Parameter Values”. Because DS is stopped, you must set the value by modifying the /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif file: 
    dn: cn=config,cn=ldbm database,cn=plugins,cn=config
...
nsslapd-db-locks: backup_db_lock_number
  9. Start the IdM server: 
    # ipactl start
    This starts all IdM services, including DS.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.