Red Hat SummitChapter 2. Configuring Logging
This chapter describes how to configure logging for various Ceph subsystems.
Logging is resource intensive. Also, verbose logging can generate a huge amount of data in a relatively short time. It you are encountering problems in a specific subsystem of the cluster, enable logging only of that subsystem. See Section 2.1, “Ceph Subsystems” for more information.
In addition, consider setting up a rotation of log files. See Section 2.4, “Accelerating Log Rotation” for details.
Once you fix any problems you encounter, change the subsystems log and memory levels to their default values. See Appendix A, Subsystems Default Logging Levels Values for list of all Ceph subsystems and their default values.
You can configure Ceph logging by:
-
Using the
cephcommand at runtime. This is the most common approach. See Section 2.2, “Configuring Logging at Runtime” for details. - Updating the Ceph configuration file. Use this approach if you are encountering problems when starting the cluster. See Section 2.3, “Configuring Logging in the Ceph Configuration File” for details.
2.1. Ceph Subsystems
This section contains information about Ceph subsystems and their logging levels.
Understanding Ceph Subsystems and Their Logging Levels
Ceph consists of several subsystems. Each subsystem has a logging level of its:
-
Output logs that are stored by default in
/var/log/ceph/directory (log level) - Logs that are stored in a memory cache (memory level)
In general, Ceph does not send logs stored in memory to the output logs unless:
- A fatal signal is raised
- An assert in source code is triggered
- You request it
You can set different values for each of these subsystems. Ceph logging levels operate on scale of 1 to 20, where 1 is terse and 20 is verbose.
Use a single value for the log level and memory level to set them both to the same value. For example, debug_osd = 5 sets the debug level for the ceph-osd daemon to 5.
To use different values for the output log level and the memory level, separate the values with a forward slash (/). For example, debug_mon = 1/5 sets the debug log level for the ceph-mon daemon to 1 and its memory log level to 5.
The Most Used Ceph Subsystems and Their Default Values
| Subsystem | Log Level | Memory Level | Description |
|---|---|---|---|
|
| 1 | 5 | The administration socket |
|
| 1 | 5 | Authentication |
|
| 0 | 5 |
Any application or library that uses |
|
| 1 | 5 | The FileStore OSD back end |
|
| 1 | 5 | The OSD journal |
|
| 1 | 5 | The Metadata Servers |
|
| 0 | 5 | The Monitor client handles communication between most Ceph daemons and Monitors |
|
| 1 | 5 | Monitors |
|
| 0 | 5 | The messaging system between Ceph components |
|
| 0 | 5 | The OSD Daemons |
|
| 0 | 5 | The algorithm that Monitors use to establish a consensus |
|
| 0 | 5 | Reliable Autonomic Distributed Object Store, a core component of Ceph |
|
| 0 | 5 | The Ceph Block Devices |
|
| 1 | 5 | The Ceph Object Gateway |
Example Log Outputs
The following examples show the type of messages in the logs when you increase the verbosity for the Monitors and OSDs.
Monitor Debug Settings
debug_ms = 5 debug_mon = 20 debug_paxos = 20 debug_auth = 20
debug_ms = 5
debug_mon = 20
debug_paxos = 20
debug_auth = 20Example Log Output of Monitor Debug Settings
OSD Debug Settings
debug_ms = 5 debug_osd = 20 debug_filestore = 20 debug_journal = 20
debug_ms = 5
debug_osd = 20
debug_filestore = 20
debug_journal = 20Example Log Output of OSD Debug Settings
See Also
2.2. Configuring Logging at Runtime
To activate the Ceph debugging output, dout(), at runtime:
ceph tell <type>.<id> injectargs --debug-<subsystem> <value> [--<name> <value>]
ceph tell <type>.<id> injectargs --debug-<subsystem> <value> [--<name> <value>]Replace:
-
<type>with the type of Ceph daemons (osd,mon, ormds) -
<id>with a specific ID of the Ceph daemon. Alternatively, use*to apply the runtime setting to all daemons of a particular type. -
<subsystem>with a specific subsystem. See Section 2.1, “Ceph Subsystems” for details. -
<value>with a number from1to20, where1is terse and20is verbose
For example, to set the log level for the OSD subsystem on the OSD named osd.0 to 0 and the memory level to 5:
ceph tell osd.0 injectargs --debug-osd 0/5
# ceph tell osd.0 injectargs --debug-osd 0/5To see the configuration settings at runtime:
-
Log in to the host with a running Ceph daemon, for example
ceph-osdorceph-mon. Display the configuration:
ceph daemon <name> config show | less
ceph daemon <name> config show | lessCopy to Clipboard Copied! Toggle word wrap Toggle overflow Specify the name of the Ceph daemon, for example:
ceph daemon osd.0 config show | less
# ceph daemon osd.0 config show | lessCopy to Clipboard Copied! Toggle word wrap Toggle overflow
See Also
- Section 2.3, “Configuring Logging in the Ceph Configuration File”
- The Logging Configuration Reference chapter in the Configuration Guide for Red Hat Ceph Storage 2
2.3. Configuring Logging in the Ceph Configuration File
To activate Ceph debugging output, dout() at boot time, add the debugging settings to the Ceph configuration file.
-
For subsystems common to each daemon, add the settings under the
[global]section. -
For subsystems for particular daemons, add the settings under a daemon section, such as
[mon],[osd], or[mds].
For example:
See Also
- Section 2.1, “Ceph Subsystems”
- Section 2.2, “Configuring Logging at Runtime”
- The Logging Configuration Reference chapter in the Configuration Guide for Red Hat Ceph Storage 2
2.4. Accelerating Log Rotation
Increasing debugging level for Ceph components might generate a huge amount of data. If you have almost full disks, you can accelerate log rotation by modifying the Ceph log rotation file at /etc/logrotate.d/ceph. The Cron job scheduler uses this file to schedule log rotation.
Procedure: Accelerating Log Rotation
Add the size setting after the rotation frequency to the log rotation file:
rotate 7 weekly size <size> compress sharedscripts
rotate 7 weekly size <size> compress sharedscriptsCopy to Clipboard Copied! Toggle word wrap Toggle overflow For example, to rotate a log file when it reaches 500 MB:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Open the
crontabeditor:crontab -e
$ crontab -eCopy to Clipboard Copied! Toggle word wrap Toggle overflow Add an entry to check the
/etc/logrotate.d/cephfile. For example, to instruct Cron to check/etc/logrotate.d/cephevery 30 minutes:30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1Copy to Clipboard Copied! Toggle word wrap Toggle overflow
See Also
- The Scheduling a Recurring Job Using Cron section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}