Chapter 8. Monitoring server and database activity
You can monitor Directory Server and replication topology by using the web console and the dsconf utility. You can configure Directory Server to record events to log files that you can then use to troubleshoot, monitor, and analyze the directory activity.
8.1. Monitoring Directory Server activity Copy linkLink copied to clipboard!
The Red Hat Directory Server tracks and records performance data by using performance counters and logs.
- Performance counters provide a measurement of the Directory Server performance. Performance counters focus on the operations and information of Directory Server, configured databases, and database links (chaining databases).
Log files record events that happen during server activity. To monitor the performance, you can use the following logs:
- Access log
- Error log
- Audit log
- Audit fail log
Security log
For more details about log files, see Types of log files in Directory Server.
The information on the current Directory Server activities is available through the web console or by using the command line. You can also monitor the cache activity of all databases.
The access log is buffered and it enables full access logging even with highly loaded servers. However, there is a discrepancy between when the event occurs on the server and the time the event is recorded in the log.
8.1.1. Monitoring Directory Server using the command line Copy linkLink copied to clipboard!
With the dsconf
command, you can monitor the disk usage, query server statistics that are stored in the directory, and other metrics to track the performance.
Prerequisite
-
Ensure the server is running to use the
dconf
utility.
Procedure
To monitor the server performance using the command-line, run:
dsconf <instance_name> monitor server
# dsconf <instance_name> monitor server
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.1.2. Monitoring Directory Server using the web console Copy linkLink copied to clipboard!
The web console is a browser-based graphical user interface (GUI) that enables users to perform administrative tasks. The Directory Server packages automatically install the Directory Server user interface for the web console.
Procedure
To open Directory Server in the web console, connect to the web console running on port 9090 on the Directory Server host.
https://server.example.com:9090
https://server.example.com:9090
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Log in as the
root
user or withsudo
privileges. Under the Monitoring tab, select
.
8.1.3. Server monitoring attributes Copy linkLink copied to clipboard!
The dsconf
command returns the following attributes when monitoring Directory Server.
Attribute | Description |
---|---|
| Identifies the current directory version number. |
| The current number of active threads handling requests. The internal server tasks, such as replication or chaining, can create additional threads when required. |
| Provides the following summary information for each open connection when you bind to the directory as the Directory Manager:
By default, this information is available to the Directory Manager. However, you can edit the Access Control Instruction (ACI) attribute in directory entries and configure additional users on access privileges to the information. |
| Identifies the number of connections currently in service by the directory. |
| Identifies the number of connections the server handles after it starts. |
|
Displays the connections currently in a |
|
Displays the number of times a connection hits the |
|
Shows the number of file descriptors available for the directory. Each connection requires one file descriptor for every open index, log file management, and for |
| Identifies the number of threads waiting to read data from a client. |
| Identifies the number of operations the server initiates after it starts. |
| Identifies the number of operations the server completes. |
| Identifies the number of entries sent to clients after the server starts. |
| Identifies the number of bytes sent to clients after the server starts. |
| Identifies the server snapshot time. The time display is Greenwich Mean Time (GMT) in UTC format. |
| Identifies the time when the server starts. The time display is Greenwich Mean Time (GMT) in UTC format. |
| Identifies the number of back ends (databases) the server services. |
8.1.4. Server information Copy linkLink copied to clipboard!
The Directory Server displays the following fields under the Server Information menu.
Field | Description |
---|---|
| Displays the name of the Directory Server instance. |
| Identifies the current server version. |
| The date and time the server is up and running. |
| The measure of time the instance is up and running. |
| The current number of active threads that handle requests. The internal server tasks, such as replication or chaining, can create additional threads when required. |
| The total number of threads waiting to be read from the client. Threads may not be immediately read if the server receives new request from the client and halts the request transmission. Generally, waiting threads indicate a slow network or a slow client. |
|
Displays all connections that are currently in a |
|
Displays the number of times a connection hits |
| The total number of connections established to a Directory Server instance. |
| The total number of open connections. Each connection can start multiple operations and therefore multiple threads. |
| The number of operations initiated by a connection. |
| The number of operations completed by the server for all connections. |
| The number of entries sent to clients after the server starts. |
8.2. Using the health check to identify problems Copy linkLink copied to clipboard!
You can perform a health check to analyze the Directory Server instance for potential issues and to get recommended solutions.
8.2.1. Running the Directory Server health check Copy linkLink copied to clipboard!
Use the dsctl healthcheck
command to run a health check.
Procedure
To run a health check, enter:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To display the output in JSON format, pass the
--json
parameter to the command:dsctl --json <instance_name> healthcheck
# dsctl --json <instance_name> healthcheck
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example 8.1. Possible report of the health check
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example 8.2. Possible report of the health check in JSON format
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.2.2. Overview of health checks Copy linkLink copied to clipboard!
The dsctl healthcheck
command performs the following checks:
Component | Severity | Result code | Description |
---|---|---|---|
Back end | Medium | DSBLE0006 | Berkeley database (BDB) is still used as a backend. BDB is deprecated and should not be used as a backend. |
Back end | Low | DSBLE0005 | Backend configuration attributes mismatch. Found configuration attributes that are not applicable for the configured backend type. |
Back end | Low | DSBLE0003 | The database was not initialized. The database was created, but it is empty. |
Back end | Medium | DSBLE0001 | The mapping tree entry for a back end is missing in the configuration. |
Config | Low | DSCLE0001 | High-resolution time stamps are disabled. |
Config | High | DSVIRTLE0001 | A virtual attribute is incorrectly indexed. Indexed attributes used by roles or Class of Service (CoS) definitions can corrupt search results. |
Operating System | Medium | DSPERMLE0001 |
The permissions set on the |
Operating System | High | DSDSLE0001 | Low disk space. |
Operating System | High | DSPERMLE0002 |
The permissions set on the |
Plug-ins | Low | DSRILE0001 | An update delay is set for the Referential Integrity plug-in. This can cause replication issues. |
Plug-ins | High | DSRILE0002 | The Referential Integrity plug-in misses indexes. The plug-in queries certain attributes for every delete operation if they are not indexed. This can cause hard-to-detect unindexed searches and high CPU usage. |
Replication | Low | DSREPLLE0002 | Conflict entries exist in the database. |
Replication | Low | DSSKEWLE0001 | The replication time skew is larger than 6 hours and lower than 12 hours. |
Replication | Medium | DSCLLE0001 | Changelog trimming is disabled. In this case, the changelog grows without limits. |
Replication | Medium | DSREPLLE0004 | The health check failed to retrieve the replication status. |
Replication | Medium | DSREPLLE0003 | The topology is out of synchronization, but the replication is working. |
Replication | Medium | DSREPLLE0005 | A remote replica is not reachable. |
Replication | Medium | DSSKEWLE0002 | The replication time skew is larger than 12 hours and lower than 24 hours. |
Replication | High | DSREPLLE0001 | The topology is out of synchronization, and the replication is not working. |
Replication | High | DSSKEWLE0003 | The replication time skew is larger than 24 hours. Replication sessions could break. |
Security | Medium | DSELE0001 | The minimum TLS version is set to a value lower than TLS 1.2. |
Security | High | DSCLE0002 | A password storage scheme is weak. |
Server | High | DSBLE0002 | The health check failed to query the back end. |
Transparent Huge Pages (THP) | Medium | DSTHPLE0001 | THP are enabled and might have an impact on the Directory Server performance. |
TLS certificates | Medium | DSCERTLE0001 | The server certificate expires within the next 30 days. |
TLS certificates | High | DSCERTLE0002 | The server certificate has expired. |
Indexes | Low | DSMOLE0002 | If the substring index is configured for a membership attribute. The removal of a member from a large group can be slow. |
8.3. Configuring log files in Directory Server Copy linkLink copied to clipboard!
Directory Server records events to log files. Use these files to troubleshoot, monitor, and analyze the directory activity.
In Directory Server, you can enable or disable logging, configure the log level, define logging policies, compress log files, and perform other operations.
8.3.1. Types of log files in Directory Server Copy linkLink copied to clipboard!
Directory Server has the following log file types that are stored the /var/log/dirsrv/slapd-<instance_name>/
directory:
- Access log (access). Enabled by default
- Contains information on client connections and connection attempts to the Directory Server instance. Note that because the access log is buffered, you can notice a discrepancy between when the event occurs on the server and the time the event is recorded in the log.
- Error log (error). Enabled by default
- Contains detailed messages of errors and events that the directory experiences during normal operations.
If Directory Server fails to write messages to the error
log file, the server sends an error message to the syslog
service and exits.
- Audit log (audit). Disabled by default
-
Records changes made to each database and to the server configuration. If you enable audit logging, Directory Server records only successful operations to the
audit
log file. - Audit fail log (audit-failure). Disabled by default
-
Records failed change operations. With the default settings, Directory Server writes failed operations to the same file as the audit log. To write failed operations to a separate file, set a path to this file in the
nsslapd-auditfaillog
configuration attribute. For details, see nsslapd-auditfaillog section. - Security log (security). Enabled by default
- Records authentication events, authorization issues, DoS/TCP attacks, and other security events.
For more detailed information about Directory Server log files, see Log files reference.
8.3.2. Displaying log files Copy linkLink copied to clipboard!
You can display Directory Server log files using the command line or the web console.
8.3.2.1. Displaying log files using the command line Copy linkLink copied to clipboard!
Use the utilities included in Red Hat Enterprise Linux, such as less
, more
, and cat
, to view the log files.
Prerequisites
- You enabled logging as described in Enabling or disabling logging using the command line.
Procedure
To display the log files, use the following commands:
less /var/log/dirsrv/slapd-<instance_name>/access less /var/log/dirsrv/slapd-<instance_name>/errors less /var/log/dirsrv/slapd-<instance_name>/audit less /var/log/dirsrv/slapd-<instance_name>/access less /var/log/dirsrv/slapd-<instance_name>/security
# less /var/log/dirsrv/slapd-<instance_name>/access # less /var/log/dirsrv/slapd-<instance_name>/errors # less /var/log/dirsrv/slapd-<instance_name>/audit # less /var/log/dirsrv/slapd-<instance_name>/access # less /var/log/dirsrv/slapd-<instance_name>/security
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note that by default the audit log and the audit fail log write operations to the same file.
To display the locations of log files, use the command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
If you have not enabled logging for a specified log type, Directory Server does not create the corresponding log file.
8.3.2.2. Displaying log files using the web console Copy linkLink copied to clipboard!
To view Directory Server log files use the tab of the web console.
Prerequisites
- You are logged in to the web console.
Procedure
- Select the instance.
-
Navigate to
. In the list of log types, select the log you want to display:
Optional: Apply the following settings to the log viewer:
- Set the number of records to display.
- Enable automatic display of new log entries by selecting Continuously Refresh.
- Click the button to apply the changes.
8.3.3. Enabling or disabling logging Copy linkLink copied to clipboard!
By default, Directory Server enables access, error, security logging, and disables audit and audit fail logging.
- IMPORTANT
- Every 2000 accesses to the directory increases the access log file by approximately 1 MB. However, before disabling the access logging, consider that this information can help to troubleshoot problems.
8.3.3.1. Enabling or disabling logging using the command line Copy linkLink copied to clipboard!
Use the dsconf config replace
command to modify the following attributes in the cn=config
DN entry that manage the Directory Server logging feature:
-
nsslapd-accesslog-logging-enabled
(access log) -
nsslapd-errorlog-logging-enabled
(error log) -
nsslapd-auditlog-logging-enabled
(audit log) -
nsslapd-auditfaillog-logging-enabled
(audit fail log) -
nsslapd-securitylog-logging-enabled
(security log)
Procedure
To enable logging, set the corresponding attribute value to
on
. For example, use the following command to enable the audit logging:dsconf <instance_name> config replace nsslapd-auditlog-logging-enabled=on
# dsconf <instance_name> config replace nsslapd-auditlog-logging-enabled=on
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteMake sure that the
nsslapd-auditlog
attribute contains a valid path and a filename of the log file. Otherwise, you cannot enable the logging.To disable logging, set the corresponding attribute to
off
. For example, use the following command to disable the error logging:dsconf <instance_name> config replace nsslapd-errorlog-logging-enabled=off
# dsconf <instance_name> config replace nsslapd-errorlog-logging-enabled=off
Copy to Clipboard Copied! Toggle word wrap Toggle overflow When you disable logging, Directory Server stops to record new events to a log file. However, the log file remains in the
/var/log/dirsrv/slapd-<instance_name>/
directory.
Verification
Check if the log directory now contains the log files:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.3.2. Enabling or disabling logging using the web console Copy linkLink copied to clipboard!
To enable or disable logging for an instance use the tab in the web console.
Prerequisites
- You are logged in to the web console.
Procedure
- Select the instance.
-
Navigate to
. - Select the log type you want to configure, for example, Access Log.
- Enable or disable the logging for the selected log type.
Optional: Configure additional settings, such as the log level, the log rotation policy, and the log buffering.
- Click the button to apply the changes.
Verification
-
Navigate to
and see if Directory Server now logs the events.
8.3.4. Defining a log rotation policy Copy linkLink copied to clipboard!
Directory Server periodically rotates the current log file and creates a new one. However, you can change the default behavior by setting a rotation policy using the command line or the web console.
You can manage the following rotation settings:
- Maximum number of logs
-
Sets the maximum number of log files to keep. When the number of files is reached, Directory Server deletes the oldest log file before creating the new one. By default, it is
10
for the access log, and1
for other logs. - Maximum log size (in MB)
-
Sets the maximum size of a log file in megabytes before it is rotated. By default, it is
100
MB for all logs. - Create new log every
- Sets the maximum age of a log file. By default, Directory Server rotates all logs every week.
- Time of day
- Set the time when the log file is rotated. This setting is not enabled by default for all logs.
- Access mode
-
The access mode sets the file permissions on newly created log files. By default, it is
600
for all logs.
8.3.4.1. Configuring a log rotation policy using the command line Copy linkLink copied to clipboard!
You can use the dsconf config replace
command to modify the following attributes in the cn=config
DN entry that manage rotation policies:
access log | error log | audit log | audit fail log | security log | |
---|---|---|---|---|---|
Maximum number of logs | nsslapd-accesslog-maxlogsperdir | nsslapd-errorlog-maxlogsperdir | nsslapd-auditlog-maxlogsperdir | nsslapd-auditfaillog-maxlogsperdir | nsslapd-securitylog-maxlogsperdir |
Maximum log size (in MB) | nsslapd-accesslog-maxlogsize | nsslapd-errorlog-maxlogsize | nsslapd-auditlog-maxlogsize | nsslapd-auditfaillog-maxlogsize | nsslapd-securitylog-maxlogsize |
Create new log every | nsslapd-accesslog-logrotationtime, nsslapd-accesslog-logrotationtimeunit | nsslapd-errorlog-logrotationtime, nsslapd-errorlog-logrotationtimeunit | nsslapd-auditlog-logrotationtime, nsslapd-auditlog-logrotationtimeunit | nsslapd-auditfaillog-logrotationtime, nsslapd-auditfaillog-logrotationtimeunit | nsslapd-securitylog-logrotationtime, nsslapd-securitylog-logrotationtimeunit |
Time of day | nsslapd-accesslog-logrotationsynchour, nsslapd-accesslog-logrotationsyncmin | nsslapd-errorlog-logrotationsynchour, nsslapd-errorlog-logrotationsyncmin | nsslapd-auditlog-logrotationsynchour, nsslapd-auditlog-logrotationsyncmin | nsslapd-auditfaillog-logrotationsynchour, nsslapd-auditfaillog-logrotationsyncmin | nsslapd-securitylog-logrotationsynchour, nsslapd-securitylog-logrotationsyncmin |
Access mode | nsslapd-accesslog-mode | nsslapd-errorlog-mode | nsslapd-auditlog-mode | nsslapd-auditfaillog-mode | nsslapd-securitylog-mode |
Procedure
To configure the error log to use access mode 600, to keep maximum 2 logs, and to rotate log files with a 100 MB size or every 5 days, enter:
dsconf <instance_name> config replace nsslapd-errorlog-mode=600 nsslapd-errorlog-maxlogsperdir=2 nsslapd-errorlog-maxlogsize=100 nsslapd-errorlog-logrotationtime=5 nsslapd-errorlog-logrotationtimeunit=day
# dsconf <instance_name> config replace nsslapd-errorlog-mode=600 nsslapd-errorlog-maxlogsperdir=2 nsslapd-errorlog-maxlogsize=100 nsslapd-errorlog-logrotationtime=5 nsslapd-errorlog-logrotationtimeunit=day
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For more details about rotation policy attributes, see corresponding sections in Core server configuration attributes description.
8.3.4.2. Configuring a log rotation policy using the web console Copy linkLink copied to clipboard!
To periodically archive the current log file and create a new one, set a log file rotation policy by using the web console.
Prerequisites
- You are logged in to the web console.
Procedure
- Select the instance.
Navigate to
and select the log type, for example, Error Log
.The Error Log Settings page opens.
- Click the Rotation Policy tab.
Configure rotation policy parameters. For example, set maximum 3 log files, the log size maximum 110 MB, and creation of a new log file every 3 days.
- Click the button to apply changes.
8.3.5. Defining a log deletion policy Copy linkLink copied to clipboard!
Directory Server automatically deletes old archived log files if you set a deletion policy.
You can only set a log file deletion policy if you have a log file rotation policy set. Directory Server applies the deletion policy at the time of log rotation.
You can set the following configuration attributes to manage the log file deletion policy:
- Log archive exceeds (in MB)
- If the size of a log file of one type exceeds the configured value, the oldest log file of this type is automatically deleted.
- Free disk space (in MB)
- When the free disk space reaches this value, the oldest archived log file is automatically deleted.
- Log file is older than
- When a log file is older than the configured time, it is automatically deleted.
8.3.5.1. Configuring a log deletion policy using the command line Copy linkLink copied to clipboard!
You can use the dsconf config replace
command to modify the following attributes in the cn=config
DN entry that manage deletion policies:
access log | error log | audit log | audit fail log | security log | |
---|---|---|---|---|---|
Log archive exceeds (in MB) | nsslapd-accesslog-logmaxdiskspace | nsslapd-errorlog-logmaxdiskspace | nsslapd-auditlog-logmaxdiskspace | nsslapd-auditfaillog-logmaxdiskspace | nsslapd-securitylog-logmaxdiskspace |
Free disk space (in MB) | nsslapd-accesslog-logminfreediskspace | nsslapd-errorlog-logminfreediskspace | nsslapd-auditlog-logminfreediskspace | nsslapd-auditfaillog-logminfreediskspace | nsslapd-securitylog-logminfreediskspace |
Log file is older than | nsslapd-accesslog-logexpirationtime, nsslapd-accesslog-logexpirationtimeunit | nsslapd-errorlog-logminfreediskspace, nsslapd-errorlog-logexpirationtimeunit | nsslapd-auditlog-logminfreediskspace, nsslapd-auditlog-logexpirationtimeunit | nsslapd-auditfaillog-logminfreediskspace, nsslapd-auditfaillog-logexpirationtimeunit | nsslapd-securitylog-logminfreediskspace, nsslapd-securitylog-logexpirationtimeunit |
Procedure
For example, to auto-delete the oldest access log file if the total size of all access log files exceeds 500 MB, enter:
dsconf <instance_name> config replace nsslapd-accesslog-logmaxdiskspace=500
# dsconf <instance_name> config replace nsslapd-accesslog-logmaxdiskspace=500
Copy to Clipboard Copied! Toggle word wrap Toggle overflow For more details about deletion policy attributes, see corresponding sections in Core server configuration attributes description.
8.3.5.2. Configuring a log deletion policy using the web console Copy linkLink copied to clipboard!
To automatically delete old archived log files, set a log deletion policy by using the web console.
Prerequisites
- You are logged in to the web console.
Procedure
- Select the instance.
-
Navigate to
and select the log type, for example, Access Log
. The Access Log Settings page opens. - Click the Deletion Policy tab.
Configure deletion policy parameters. For example, set maximum archive size to 600 MB and the log file age to 3 weeks.
- Click the button to apply changes.
8.3.6. Manual log file rotation Copy linkLink copied to clipboard!
You can rotate log files manually only if you did not configure an automatic log file rotation or deletion policies.
Procedure
Stop the instance:
dsctl <instance_name> stop
# dsctl <instance_name> stop
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Go to the log files directory. By default, Directory Server stores access, error, audit, audit fail log, and security files in the
/var/log/dirsrv/slapd-instance/
directory. - Move or rename the log file you want to rotate to make it available for future reference.
Start the instance:
dsctl <instance_name> restart
# dsctl <instance_name> restart
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.7. Configuring log levels Copy linkLink copied to clipboard!
To manage how detailed logs are, and therefore the amount of information that is logged, you can specify log levels for access logging and error logging.
Changing the default log level can lead to very large log files. Red Hat recommends that you do not change the default logging values without being asked to do so by Red Hat technical support.
8.3.7.1. Configuring log levels using the command line Copy linkLink copied to clipboard!
You can adjust log levels by setting the following configuration attributes:
-
nsslapd-accesslog-level
for the access log -
nsslapd-errorlog-level
for the error log
Use the dsconf config replace
command to modify the log level attributes. The attribute values are additive: for example, if you set a log level value of 12, it includes levels 8 and 4.
Prerequisites
- You enabled access and error logging.
Procedure
To enable Logging internal access operations (4) and Logging for connections, operations, and results (256) for the access log, set the
nsslapd-accesslog-level
attribute to 260 (4 + 256) with the following command:dsconf <instance_name> config replace nsslapd-accesslog-level=260
# dsconf <instance_name> config replace nsslapd-accesslog-level=260
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To enable Search filter logging (32) and Config file processing (64) log levels for the error log, set the
nsslapd-errorlog-level
attribute to 96 (32 + 64) with the following command:dsconf <instance_name> config replace nsslapd-errorlog-level=96
# dsconf <instance_name> config replace nsslapd-errorlog-level=96
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
When you set the access log level to Logging internal access operations (4), do the following to see if Directory Server started to log internal access events:
Restart the instance to trigger internal events by command:
dsctl <instance_name> restart
# dsctl <instance_name> restart Instance "<instance_name>" has been restarted
Copy to Clipboard Copied! Toggle word wrap Toggle overflow View the access log file and find internal operation records:
cat /var/log/dirsrv/slapd-instance_name/access ... [08/Nov/2022:16:29:05.556977401 -0500] conn=2 (Internal) op=1(1)(1) SRCH base="cn=config,cn=WritersData,cn=ldbm database,cn=plugins,cn=config" scope=1 filter="objectclass=vlvsearch" attrs=ALL [08/Nov/2022:16:29:05.557250374 -0500] conn=2 (Internal) op=1(1)(1) RESULT err=0 tag=48 nentries=0 wtime=0.000016828 optime=0.000274854 etime=0.000288952 ...
# cat /var/log/dirsrv/slapd-instance_name/access ... [08/Nov/2022:16:29:05.556977401 -0500] conn=2 (Internal) op=1(1)(1) SRCH base="cn=config,cn=WritersData,cn=ldbm database,cn=plugins,cn=config" scope=1 filter="objectclass=vlvsearch" attrs=ALL [08/Nov/2022:16:29:05.557250374 -0500] conn=2 (Internal) op=1(1)(1) RESULT err=0 tag=48 nentries=0 wtime=0.000016828 optime=0.000274854 etime=0.000288952 ...
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.7.2. Configuring log levels using the web console Copy linkLink copied to clipboard!
To manage how detailed logs are, specify log levels for access logging and error logging.
Prerequisites
- You are logged in to the web console.
- You enabled access and error logging.
Procedure
- Select the instance.
-
Navigate to
. -
Select the log type, for example,
Access Log
. Click the
button to see all available log levels for the log type.-
Select log levels, for example,
Default Logging
andInternal Operations
levels. - Click the button to apply changes.
Verification
To see if Directory Server started to log internal access events, do the following:
- Restart the instance by clicking button and then selecting . Directory Server restarts the instance and generates internal events.
-
Navigate to
. Refresh access log and view recorded internal events:
[08/Nov/2022:17:04:17.035502206 -0500] conn=6 (Internal) op=1(2)(1) SRCH base="cn=config,cn=Example database,cn=ldbm database,cn=plugins,cn=config" scope=1 filter="objectclass=vlvsearch" attrs=ALL [08/Nov/2022:17:04:17.035579829 -0500] conn=6 (Internal) op=1(2)(1) RESULT err=0 tag=48 nentries=0 wtime=0.000004563 optime=0.000078000 etime=0.000081911
[08/Nov/2022:17:04:17.035502206 -0500] conn=6 (Internal) op=1(2)(1) SRCH base="cn=config,cn=Example database,cn=ldbm database,cn=plugins,cn=config" scope=1 filter="objectclass=vlvsearch" attrs=ALL [08/Nov/2022:17:04:17.035579829 -0500] conn=6 (Internal) op=1(2)(1) RESULT err=0 tag=48 nentries=0 wtime=0.000004563 optime=0.000078000 etime=0.000081911
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.8. Configuring logging for plug-ins Copy linkLink copied to clipboard!
By default, Directory Server does not log internal events which plug-ins initiate. To debug plug-in operations, you can enable access and audit logging for all plug-ins, or for specific plug-ins.
8.3.8.1. Configuring logging for all plug-ins Copy linkLink copied to clipboard!
Use nsslapd-plugin-logging
attribute to configure logging for all plug-ins.
Procedure
To enable access and audit logging for all plug-ins, use the following command:
dsconf <instance_name> config replace nsslapd-plugin-logging=on
# dsconf <instance_name> config replace nsslapd-plugin-logging=on
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.8.2. Configuring logging for a specific plugin Copy linkLink copied to clipboard!
Use nsslapd-logAccess
and nsslapd-logAudit
attributes to configure logging for a plug-in.
Prerequisites
-
The
nsslapd-accesslog
attribute contains valid path and the filename for the access log file. -
The
nsslapd-auditlog
attribute contains valid path and the filename for the audit log file.
Procedure
To enable access and audit logging for a specific plug-in, modify
nsslapd-logAccess
andnsslapd-logAudit
attributes using the LDAP interface:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.9. Logging statistics per search operation Copy linkLink copied to clipboard!
During some search operations, especially with filters such as (cn=user*)
, the time the server spends for receiving the tasks and then sending the result back (etime
) can be very long.
Expanding the access log with information related to indexes used during search operation helps to diagnose why etime
value is resource expensive.
Use the nsslapd-statlog-level
attribute to enable collecting statistics, such as a number of index lookups (database read operations) and overall duration of index lookups for each search operation, with minimal impact on the server.
Prerequisites
- You enabled access logging.
Procedure
Enable search operation metrics:
dsconf <instance_name> config replace nsslapd-statlog-level=1
# dsconf <instance_name> config replace nsslapd-statlog-level=1
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Restart the instance:
dsctl <instance_name> restart
# dsctl <instance_name> restart
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Perform a search operation:
ldapsearch -D "cn=Directory Manager" -H ldap://server.example.com -b "dc=example,dc=com" -s sub -x "cn=user*"
# ldapsearch -D "cn=Directory Manager" -H ldap://server.example.com -b "dc=example,dc=com" -s sub -x "cn=user*"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow View the access log file and find the search statistics records:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3.10. Compressing log files Copy linkLink copied to clipboard!
To save disc space, you can enable log file compression that compresses archived logs into .gzip
files.
Use the dsconf config replace
command to modify the following attributes that manage log file compression:
-
nsslapd-accesslog-compress
(access log) -
nsslapd-errorlog-compress
(error log) -
nsslapd-auditlog-compress
(audit log) -
nsslapd-auditfaillog-compress
(audit fail log) -
nsslapd-securitylog-compress
(security log)
By default, Directory Server compresses only archived security log files.
Procedure
To enable log file compression, run:
dsconf <instance_name> config replace nsslapd-accesslog-compress=on nsslapd-errorlog-compress=on
# dsconf <instance_name> config replace nsslapd-accesslog-compress=on nsslapd-errorlog-compress=on
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The command enables compression for access and error logs.
To disable log file compression, run:
dsconf <instance_name> config replace nsslapd-accesslog-compress=off
# dsconf <instance_name> config replace nsslapd-accesslog-compress=off
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The command disables compression for the access log.
Verification
Check that the log file directory contains compressed logs files:
# ls /var/log/dirsrv/slapd-instance_name/
8.3.11. Disabling log buffering for debugging purposes Copy linkLink copied to clipboard!
For debugging purposes, you can disable log buffering, which is enabled by default for the access, audit, and audit fail logs. With log buffering disabled, Directory Server writes log entries directly to the disk.
Do not disable access logging in a normal operating environment. When you disable the buffering of an active log, Directory Server performance decreases, especially under heavy load.
8.3.11.1. Disabling log buffering using the command line Copy linkLink copied to clipboard!
Use the dsconf
utility to disable Directory Server log buffering. When log buffering is disabled, Directory Server writes log entries directly to the disk.
The following procedure disables the access log buffering.
Procedure
To disable the access log buffering, enter:
dsconf <instance_name> config replace nsslapd-accesslog-logbuffering=off
# dsconf <instance_name> config replace nsslapd-accesslog-logbuffering=off
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteBoth the audit and audit fail logs use the
nsslapd-auditlog-logbuffering
attribute for the buffering configuration. Therefore, when you setnsslapd-auditlog-logbuffering
tooff
, buffering for both logs is disabled.
Verification
Display the access log in continuous mode:
tail -f /var/log/dirsrv/slapd-<instance_name>/access
# tail -f /var/log/dirsrv/slapd-<instance_name>/access
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Perform actions in the directory, such as searches.
- Monitor the access log. Log entries appear without delay at the moment when users perform actions in the directory.
8.3.11.2. Disabling log buffering using the web console Copy linkLink copied to clipboard!
Use the web console to disable Directory Server log buffering. When log buffering is disabled, Directory Server writes log entries directly to the disk.
The following procedure disables the access log buffering.
Procedure
-
Navigate to
. -
Deselect
Access Log Buffering Enabled
. - Click .
To disable the audit fail log buffering, disable buffering for the the audit log on the
Verification
-
Navigate to
. -
Select
Continuously Refresh
. - Perform actions in the directory, such as searches.
- Monitor the access log. Log entries appear without delay at the moment when users perform actions in the directory.
8.4. Monitoring the replication topology Copy linkLink copied to clipboard!
Use the command line of the web console to monitor the replication topology.
8.4.1. Monitoring the replication topology using the command line Copy linkLink copied to clipboard!
To monitor the state of the directory data replication between suppliers, consumers, and hubs, you can use replication topology report that provides information on the replication progress, replica IDs, number of changes, and other parameters. To generate the report faster and make it more readable, you can configure your own credentials and aliases.
8.4.1.1. Displaying a replication topology report using the command line Copy linkLink copied to clipboard!
To view overall information about the replication status for each agreement in your replication topology, you can display the replication topology report. To do so, use the dsconf replication monitor
command.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumers.
Procedure
To view a replication topology report, enter:
dsconf <instance_name> replication monitor
# dsconf <instance_name> replication monitor
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
dsconf
utility will request authentication credentials for each instance in the topology:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.4.1.2. Setting credentials for replication monitoring in the .dsrc file Copy linkLink copied to clipboard!
By default, the dsconf replication monitor
command asks for bind DNs and passwords when authenticating to remote instances. To generate the report faster and easier in the future, you can set the bind DNs, and optionally passwords, for each server in the topology in the user’s ~/.dsrc
file.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumers.
Procedure
-
Optional: Create the
~/.dsrc
file. In the
~/.dsrc
file, set the bind DNs, and passwords. For example:[repl-monitor-connections] connection1 = server1.example.com:389:cn=Directory Manager:* connection2 = server2.example.com:389:cn=Directory Manager:[~/pwd.txt] connection3 = hub1.example.com:389:cn=Directory Manager:S3cret
[repl-monitor-connections] connection1 = server1.example.com:389:cn=Directory Manager:* connection2 = server2.example.com:389:cn=Directory Manager:[~/pwd.txt] connection3 = hub1.example.com:389:cn=Directory Manager:S3cret
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This example uses connection1 to connection3 as keys for each entry. However, you can use any unique key.
When you run the
dsconf replication monitor
command, thedsconf
utility connects to all servers configured in replication agreements of the instance. If the utility finds the hostname in~/.dsrc
, it uses the defined credentials to authenticate to the remote server. In the example above,dsconf
uses the following credentials when connecting to a server:Expand Hostname Bind DN Password setup method server1.example.com
cn=Directory Manager
Requests the password
server2.example.com
cn=Directory Manager
Reads the password from
~/pwd.txt
hub1.example.com
cn=Directory Manager
S3cret
Verification
-
Run the
dsconf replication monitor
command to see Ifdsconf
utility uses credentials configured in the~/.dsrc
file. For more information, see Displaying a replication topology report using the command.
8.4.1.3. Using aliases in the replication topology monitoring output Copy linkLink copied to clipboard!
To make the report more readable, you can set your own aliases that will be displayed in the report output. By default, the replication monitoring report contains the hostnames of remote servers.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumers.
Procedure
If you want to see aliases in the report, use one of the following methods:
Define the aliases in the
~/.dsrc
file:[repl-monitor-aliases] M1 = server1.example.com:389 M2 = server2.example.com:389
[repl-monitor-aliases] M1 = server1.example.com:389 M2 = server2.example.com:389
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Define the aliases by passing the
-a alias=host_name:port
parameter to thedsconf replication monitor
command:dsconf <instance_name> replication monitor -a M1=server1.example.com:389 M2=server2.example.com:389
# dsconf <instance_name> replication monitor -a M1=server1.example.com:389 M2=server2.example.com:389dsconf <instance_name> replication monitor -a M1=server1.example.com:389 M2=server2.example.com:389dsconf <instance_name> replication monitor -a M1=server1.example.com:389 M2=server2.example.com:389
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
In both cases, the dsconf replication monitor
command displays the alias in the output:
8.4.2. Monitoring the replication topology using the web console Copy linkLink copied to clipboard!
To monitor the state of the directory data replication between suppliers, consumers, and hubs, you can use replication topology report that provides information on the replication progress, replica IDs, number of changes, and other parameters. To generate the report faster and make it more readable, you can configure your own credentials and aliases.
8.4.2.1. Displaying a replication topology report using the web console Copy linkLink copied to clipboard!
To view overall information about the replication status for each agreement in your replication topology, you can display the replication topology report.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumers.
- You are logged in to the web console.
Procedure
-
Navigate to
. The page opens. - Click .
Enter the passwords for login to remote instances and click
. Directory Server uses bind DNs values from existing replication agreements.The replication topology report will be generated on the tab.
NoteTo generate another replication topology report, go to the tab.
8.4.2.2. Setting credentials for replication monitoring using the web console Copy linkLink copied to clipboard!
To generate the replication topology report faster and easier, you can set your own bind DNs, and optionally passwords, for each server in the topology for authentication. In this case, you do not need to confirm replication credentials each time you want to generate a replication topology report. By default, Directory Server takes these credentials from existing replication agreements.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumer.
- You are logged in to the web console.
Procedure
-
Navigate to
. The page opens. - Click .
Enter replication login credentials you want to use for authentication to remote instances:
-
Hostname
. A remote instance hostname you want the server to authenticate to. -
Port
. A remote instance port. -
Bind DN
. Bind DN used for authentication to the remote instance. -
Password
. A password used for authentication. -
Interactive Input
. If checked, Directory Server will ask for a password every time you generate a replication topology report.
-
- Click .
Verification
Generate the replication topology report to see If the report asks for the credentials. For more information, see Displaying a replication topology report using the web console.
8.4.2.3. Configuring replication naming aliases using the web console Copy linkLink copied to clipboard!
To make the report more readable, you can set your own aliases that will be displayed in the report output. By default, the replication monitoring report contains the hostnames of servers.
Prerequisites
- The host is a member of replication topology.
- You initialized the consumers.
- You are logged in to the web console.
Procedure
-
Navigate to
. - Click .
Enter alias details:
-
Alias
. An alias that will be displayed in the replication topology report. -
Hostname
. An instance hostname. -
Port
. An instance port.
-
- Click .
Verification
- Generate the replication topology report to see If the report uses new aliases. For more information, see Displaying a replication topology report using the web console.
8.5. Tracking the bind DN for plug-in-initiated updates Copy linkLink copied to clipboard!
In Directory Server, you can track which user performed an action that caused a plug-in to update an entry. If the tracking is enabled and a plug-in changes an entry as a consequence of an action performed by a user, you can see the user’s name in the modifiersname
attribute of updated entry.
8.5.1. Tracking user information for entry modifications performed by a plug-in Copy linkLink copied to clipboard!
When the user performs an action that changes an entry, it can trigger other, automatic changes across the directory tree. By default, Directory Server is not tracking the name of the user who performed the action that has initiated the data modification. To track the user information, you can use the nsslapd-plugin-binddn-tracking
parameter.
For example, when the administrator deletes a user, the Referential Integrity Postoperation plug-in automatically removes the user from all groups. You can see the initial action in the entry as being performed by the user account bound to the server. But all related updates are, by default, shown as being performed by the plug-in, with no information about which user initiated the update.
A second example might be using the MemberOf plug-in to update user entries with group membership. The update to the group account is shown as being performed by the bound user, while the edit to the user entry is shown as being performed by the MemberOf plug-in:
dn: cn=example_group,ou=groups,dc=example,dc=com modifiersname: uid=example,ou=people,dc=example,dc=com dn: uid=example,ou=people,dc=example,dc=com modifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
dn: cn=example_group,ou=groups,dc=example,dc=com
modifiersname: uid=example,ou=people,dc=example,dc=com
dn: uid=example,ou=people,dc=example,dc=com
modifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
The nsslapd-plugin-binddn-tracking
parameter enables the server to track which user originated an update operation, as well as the internal plug-in which actually performed the operation. The bound user is shown in the modifiersname
and creatorsname
operational attributes, while the plug-in which performed the update is shown in the internalModifiersname
and internalCreatorsname
operational attributes. For example:
dn: uid=example,ou=people,dc=example,dc=com modifiersname: uid=admin,ou=people,dc=example,dc=com internalModifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
dn: uid=example,ou=people,dc=example,dc=com
modifiersname: uid=admin,ou=people,dc=example,dc=com
internalModifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
The nsslapd-plugin-binddn-tracking
parameter tracks and maintains the relationship between the bound user and all updates performed for that connection.
The internalModifiersname
and internalCreatorsname
attributes always show a plug-in as the identity. The value of the attribute is:
-
cn=ldbm database,cn=plugins,cn=config
when the core Directory Server performs the change -
cn=the DN of the plug-in,cn=plugins,cn=config
when a plug-in changed the entry
8.5.2. Enabling tracking the bind DN for plug-in-initiated updates using the command line Copy linkLink copied to clipboard!
For data updates initiated by a plug-in, you often need to know which user has performed the action that led to the update. In the command line, set up the nsslapd-plugin-binddn-tracking
parameter to track such user information.
Procedure
Set the
nsslapd-plugin-binddn-tracking
parameter toon
:dsconf <instance_name> config replace nsslapd-plugin-binddn-tracking=on
# dsconf <instance_name> config replace nsslapd-plugin-binddn-tracking=on
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
-
Display the
modifiersname
andinternalModifiersname
attributes of an entry that was changed by a plug-in. For example, if thememberOf
attribute is enabled, display the attributes of a user after you added the user to a group:
ldapsearch -D "cn=Directory Manager" -W -H ldap://server.example.com -x -b "uid=example-user,ou=People,dc=example,dc=com" -s base -x internalModifiersname -x modifiersname
# ldapsearch -D "cn=Directory Manager" -W -H ldap://server.example.com -x -b "uid=example-user,ou=People,dc=example,dc=com" -s base -x internalModifiersname -x modifiersname
dn: uid=example-user,ou=people,dc=example,dc=com
modifiersname: uid=admin,ou=people,dc=example,dc=com
internalModifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
8.5.3. Enabling tracking the bind DN for plug-in-initiated updates using the web console Copy linkLink copied to clipboard!
For data updates initiated by a plug-in, you often need to know which user has performed the action that led to the update. Using the web console, you can enable tracking of the user information.
Prerequisites
- You are logged in to the Directory Server instance in the web console.
Procedure
-
Open the
menu. -
On the Advanced Settings tab, select
Enable Plugin Bind DN Tracking
. - Click .
Verification
-
Display the
modifiersname
andinternalModifiersname
attributes of an entry that was changed by a plug-in. For example, if thememberOf
attribute is enabled, display the attributes of a user after you added the user to a group:
ldapsearch -D "cn=Directory Manager" -W -H ldap://server.example.com -x -b "uid=example-user,ou=People,dc=example,dc=com" -s base -x internalModifiersname -x modifiersname
# ldapsearch -D "cn=Directory Manager" -W -H ldap://server.example.com -x -b "uid=example-user,ou=People,dc=example,dc=com" -s base -x internalModifiersname -x modifiersname
dn: uid=example-user,ou=people,dc=example,dc=com
modifiersname: uid=admin,ou=people,dc=example,dc=com
internalModifiersname: cn=MemberOf Plugin,cn=plugins,cn=config
8.6. Monitoring the database activity Copy linkLink copied to clipboard!
Administrators should monitor the database activity to ensure that tuning settings, such as caches, are properly configured.
8.6.1. Monitoring the database activity using the command line Copy linkLink copied to clipboard!
To display the monitoring activity using the command line, display the dynamically-updated read-only attributes stored in the cn=monitor,cn=database_name,cn=ldbm database,cn=plugins,cn=config
.
Procedure
To display the current activity of a database, enter:
dsconf <instance_name> monitor backend userRoot
# dsconf <instance_name> monitor backend userRoot
Copy to Clipboard Copied! Toggle word wrap Toggle overflow This command displays the activity of the
userRoot
database.
8.6.2. Monitoring the database activity using the web console Copy linkLink copied to clipboard!
In the web console, Directory Server displays the values of the dynamically-updated read-only monitoring attributes from the cn=monitor,cn=database_name,cn=ldbm database,cn=plugins,cn=config
in the `Monitoring´ tab.
Procedure
-
Navigate to
. - Display the cache values on the Entry Cache and DN Cache tabs.
8.6.3. Database monitoring attributes Copy linkLink copied to clipboard!
Attribute | Description |
---|---|
|
Indicates whether the database is in read-only mode ( |
| The total number of successful entry cache lookups. The value is the total number of times the server could retrieve an entry from the entry cache without reloading it from the database. |
| The total number of entry cache lookups since you started the instance. The value is the total number, since the instance has been started, Directory Server tried to retrieve entry from the entry cache. |
| The number of entry cache tries to successful entry cache lookups. This number is based on the total lookups and hits since you last started the instance. The closer the entry cache hit ratio is to 100%, the better. Whenever an operation attempts to find an entry that is not present in the entry cache, the server needs to access the database to obtain the entry. Thus, as this ratio drops towards zero, the number of disk accesses increases, and directory search performance decreases. To improve this ratio, increase the size of the entry cache of the database.
To improve this ratio, increase the size of the entry cache by increasing the value of the |
| The total size, in bytes, of directory entries currently present in the entry cache.
To increase the size of the entries which can be present in the cache, increase the value of the |
| The maximum size, in bytes, of directory entries that Directory Server can maintain in the entry cache.
To increase the size of the entries which can be present in the cache, increase the value of the |
| The current number of entries stored in the entry cache of a given backend. |
| The maximum number of entries stored in the entry cache of a database.
To tune this value, increase the value of the |
| The number of times the server could process a request by obtaining a normalized distinguished name (DN) from the DN cache rather than normalizing it again. |
| The total number of DN cache accesses since you started the instance. |
| The ratio of cache tries to successful DN cache hits. The closer this value is to 100%, the better. |
| The total size, in bytes, of DN currently present in the DN cache.
To increase the size of the entries which can be present in the DN cache, increase the value of the |
| The maximum size, in bytes, of DNs that Directory Server can maintain in the DN cache.
To increase the size of the entries which can be present in the cache, increase the value of the |
| The number of DNs currently present in the DN cache. |
| The maximum number of DNs allowed in the DN cache. |
8.7. Getting Directory Server access log using the command line Copy linkLink copied to clipboard!
The logconv.pl
command analyzes Directory Server access logs, extracts usage statistics, and counts occurrences of significant events specified on the command line. The logconv.pl
command prints a list of total operations, total number of connections, counts per each operation type, counts for some extended operations like persistent searches, and bind information.
The logconv.pl
command has the following syntax:
logconv.pl /path/to/accesslog
# logconv.pl /path/to/accesslog
To analyze multiple access log files, you can use the following format with an asterisk (*):
logconv.pl /var/log/dirsrv/slapd-<instance_name>/access*
# logconv.pl /var/log/dirsrv/slapd-<instance_name>/access*
The logconv.pl
command generates following three types of statistics that are useful for monitoring the Directory Server and optimizing Directory Server configuration:
- Counts of events, such as total binds and total searches performed.
-
Lists of the most frequently occurring parameters in LDAP requests. For example, the
logconv.pl
command generates lists of the top ten bind DNs, base DNs, filter strings, and attributes returned. -
Counts of occurrences for error codes such as those defined in
ldap.h
.
8.7.1. Analyzing Directory Server access logs using the command line Copy linkLink copied to clipboard!
The logconv.pl
command analyzes Directory Server access logs and extracts usage statistics and counts occurrences of significant events.
The logconv.pl
takes following options:
-
-S
: specifies the time to begin the log file analysis. -
-E
: specifies the time to stop the log files analysis. -
-bc
: generates a report based on the number of DNs used to connect to the server and the total connection codes the server returns. -
-m
: generates output data per second (-m
) to a specified CSV output file. -
-M
: generates output data with counts per minute (-M
) to a specified CSV output file.
Procedure
To generate a simple access log summary, run the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
logconv.pl
script prints a list of total operations, total number of connections, counts per each operation type, counts for some extended operations like persistent searches, and bind information.Optional: If you must enable additional connection summaries passed as a single option, such as the number of DNs used to connect to the server (
b
) and the total connection codes the server (c
) returns, specify the-bc
option as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: If you must enable data output for a certain start (
-S
) and end time (-E
) or within a specific range, run the following commandCopy to Clipboard Copied! Toggle word wrap Toggle overflow When start and end times are set, the
logconv.pl
command first prints the time range given, then the summary for that period.Optional: If you must enable data output with counts per minute (
-M
) or per second (-m
), run the following command:logconv.pl -m|-M outputFile accessLogFile
# logconv.pl -m|-M outputFile accessLogFile
Copy to Clipboard Copied! Toggle word wrap Toggle overflow