Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 10. Logging and Debugging
Usually, you add debugging to the Ceph configuration at runtime. You can also add Ceph debug logging to the Ceph configuration file if you are encountering issues when starting your cluster. Also, view Ceph log files under /var/log/ceph.
When debug output slows down your system, the latency can hide race conditions.
Logging is resource intensive. If you are encountering a problem in a specific area of your cluster, enable logging for that area of the cluster. For example, if your OSDs are running fine, but your gateways are not, you should start by enabling debug logging for the specific gateway instance(s) giving you trouble. Enable logging for each subsystem as needed.
Verbose logging can generate over 1GB of data per hour. If your OS disk reaches its capacity, the node will stop working.
If you enable or increase the rate of Ceph logging, ensure that you have sufficient disk space on your OS disk. See Accelerating Log Rotation for details on rotating log files. When your system is running well, remove unnecessary debugging settings to ensure your cluster runs optimally. Logging debug output messages is relatively slow, and a waste of resources when operating your cluster.
See Subsystem, Log and Debug Settings for details on available settings.
10.1. Runtime
To see the configuration settings at runtime, log in to a host with a running daemon and execute the following command:
ceph --admin-daemon </path/to/admin/socket> config show | less ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
ceph --admin-daemon </path/to/admin/socket> config show | less
ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
To activate Ceph’s debugging output (dout()) at runtime, use the ceph tell command to inject arguments into the runtime configuration:
ceph tell <daemon-type>.<daemon id or *> injectargs --<name> <value> [--<name> <value>]
ceph tell <daemon-type>.<daemon id or *> injectargs --<name> <value> [--<name> <value>]
Replace <daemon-type> with one of osd or mon. To apply the runtime setting to all daemons of a particular type, use *, or specify a specific daemon’s ID (that is, its number or letter). For example, to increase debug logging for a ceph-osd daemon named osd.0:
ceph tell osd.0 injectargs --debug-osd 0/5
ceph tell osd.0 injectargs --debug-osd 0/5
The ceph tell command goes through the monitors. If you cannot bind to the monitor, you can still make the change by logging into the host of the daemon whose configuration you want to change by using the ceph --admin-daemon command. For example:
sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set debug_osd 0/5
sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set debug_osd 0/5See Subsystem, Log and Debug Settings for details on available settings.
10.2. Boot Time
To activate Ceph’s debugging output (dout()) at boot time, add the debug settings to the Ceph configuration file. Subsystems common to each daemon can be set under the [global] section in the configuration file. Subsystems for particular daemons are set under the daemon section in the configuration file (that is, [mon], [osd]). For example:
See Subsystem, Log and Debug Settings for details on available settings.
10.3. Accelerating Log Rotation
If your OS disk is relatively full, you can accelerate log rotation by modifying the Ceph log rotation file at /etc/logrotate.d/ceph. Add a size setting after the rotation frequency to accelerate log rotation by using the Cron utility if your logs exceed the size setting. For example, the default setting looks like this:
rotate 7 weekly compress sharedscripts
rotate 7
weekly
compress
sharedscripts
Modify the configuration by adding the size setting.
rotate 7 weekly size 500M compress sharedscripts
rotate 7
weekly
size 500M
compress
sharedscripts
Then, start the crontab editor for your user space.
crontab -e
crontab -e
Finally, add an entry to check the etc/logrotate.d/ceph file.
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
The preceding example checks the etc/logrotate.d/ceph file every 30 minutes.
10.4. Valgrind
Debugging might also require you to track down memory and threading issues. You can run a single daemon, a type of daemon, or the whole cluster with the Valgrind utility. You should only use Valgrind when developing or debugging Ceph. Valgrind is computationally expensive, and will slow down your system otherwise. Valgrind messages are logged to stderr.
10.5. Subsystems, Log and Debug Settings
In most cases, you will enable debug logging output by using subsystems and on a temporary basis.
10.5.1. Subsystems
Each subsystem has a logging level for its output logs, and for its logs in-memory. You can set different values for each of these subsystems by setting a log file level and a memory level for debug logging. Ceph’s logging levels operate on a scale of 1 to 20, where 1 is terse and 20 is verbose.
A debug logging setting can take a single value for the log level and the memory level, which sets them both as the same value. For example, if you specify debug_ms = 5, Ceph will treat it as a log level and a memory level of 5. You can also specify them separately. The first setting is the log level, and the second setting is the memory level. You must separate them with a forward slash (/). For example, if you want to set the ms subsystem’s debug logging level to 1 and its memory level to 5, you would specify it as debug_ms = 1/5. For example:
debug_<subsystem> = <log-level>/<memory-level> #for example debug_osd = 1/20
debug_<subsystem> = <log-level>/<memory-level>
#for example
debug_osd = 1/20The following table provides a list of Ceph subsystems and their default log and memory levels. Once you complete your logging efforts, restore the subsystems to their default level or to a level suitable for normal operations.
| Subsystem | Log Level | Memory Level |
|---|---|---|
|
| 0 | 5 |
|
| 0 | 1 |
|
| 0 | 1 |
|
| 0 | 1 |
|
| 0 | 0 |
|
| 0 | 1 |
|
| 0 | 1 |
|
| 0 | 1 |
|
| 0 | 1 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 0 | 5 |
|
| 1 | 3 |
|
| 1 | 3 |
|
| 0 | 5 |
|
| 1 | 5 |
|
| 0 | 10 |
|
| 1 | 5 |
|
| 0 | 5 |
|
| 1 | 5 |
|
| 1 | 1 |
|
| 1 | 5 |
|
| 1 | 5 |
|
| 1 | 5 |
|
| 1 | 10 |
|
| 1 | 5 |
|
| 1 | 5 |
|
| 1 | 1 |
|
| 0 | 0 |
|
| 1 | 5 |
Here are examples of the type of messages you will see in the logs when the verbosity is increased for the monitors and the OSDs.
Monitor Debug Settings
debug_ms = 5 debug_mon = 20 debug_paxos = 20 debug_auth = 20
debug_ms = 5
debug_mon = 20
debug_paxos = 20
debug_auth = 20Example Log Output
OSD Debug Settings
debug_ms = 5 debug_osd = 20 debug_filestore = 20 debug_journal = 20
debug_ms = 5
debug_osd = 20
debug_filestore = 20
debug_journal = 20Example Log Output
10.5.2. Logging Settings
Logging and debugging settings are not required in a Ceph configuration file, but you can override default settings as needed. Ceph supports the following settings:
- log_file
- Description
- The location of the logging file for your cluster.
- Type
- String
- Required
- No
- Default
-
/var/log/ceph/$cluster-$name.log
- log_max_new
- Description
- The maximum number of new log files.
- Type
- Integer
- Required
- No
- Default
-
1000
- log_max_recent
- Description
- The maximum number of recent events to include in a log file.
- Type
- Integer
- Required
- No
- Default
-
1000000
- log_to_stderr
- Description
-
Determines if logging messages appear in
stderr. - Type
- Boolean
- Required
- No
- Default
-
true
- err_to_stderr
- Description
-
Determines if error messages appear in
stderr. - Type
- Boolean
- Required
- No
- Default
-
true
- log_to_syslog
- Description
-
Determines if logging messages appear in
syslog. - Type
- Boolean
- Required
- No
- Default
-
false
- err_to_syslog
- Description
-
Determines if error messages appear in
syslog. - Type
- Boolean
- Required
- No
- Default
-
false
- log_flush_on_exit
- Description
- Determines if Ceph flushes the log files after exit.
- Type
- Boolean
- Required
- No
- Default
-
true
- clog_to_monitors
- Description
-
Determines if
clogmessages will be sent to monitors. - Type
- Boolean
- Required
- No
- Default
-
true
- clog_to_syslog
- Description
-
Determines if
clogmessages will be sent tosyslog. - Type
- Boolean
- Required
- No
- Default
-
false
- mon_cluster_log_to_syslog
- Description
-
Determines if the cluster log will be output to
syslog. - Type
- Boolean
- Required
- No
- Default
-
false
- mon_cluster_log_file
- Description
- The location of the cluster’s log file.
- Type
- String
- Required
- No
- Default
-
/var/log/ceph/$cluster.log
10.5.2.1. OSD
- osd_preserve_trimmed_log
- Description
- Preserves trimmed logs after trimming.
- Type
- Boolean
- Required
- No
- Default
-
false
- osd_tmapput_sets_uses_tmap
- Description
-
Uses
tmap. For debug only. - Type
- Boolean
- Required
- No
- Default
-
false
- osd_min_pg_log_entries
- Description
- The minimum number of log entries for placement groups.
- Type
- 32-bit Unsigned Integer
- Required
- No
- Default
- 1000
- osd_op_log_threshold
- Description
- How many op log messages to show up in one pass.
- Type
- Integer
- Required
- No
- Default
- 5
10.5.2.2. File Store
- filestore_debug_omap_check
- Description
- Debugging check on synchronization. This is an expensive operation.
- Type
- Boolean
- Required
- No
- Default
- 0
10.5.2.3. RADOS Gateway
- rgw_log_nonexistent_bucket
- Description
- Log non-existent buckets.
- Type
- Boolean
- Required
- No
- Default
-
false
- rgw_log_object_name
- Description
- Log an object’s name.
- Type
- String
- Required
- No
- Default
-
%Y-%m-%d-%H-%i-%n
- rgw_log_object_name_utc
- Description
- Object log name contains UTC.
- Type
- Boolean
- Required
- No
- Default
-
false
- rgw_enable_ops_log
- Description
- Enables logging of every RGW operation.
- Type
- Boolean
- Required
- No
- Default
-
true
- rgw_enable_usage_log
- Description
- Enable logging of RGW’s bandwidth usage.
- Type
- Boolean
- Required
- No
- Default
-
true
- rgw_usage_log_flush_threshold
- Description
- Threshold to flush pending log data.
- Type
- Integer
- Required
- No
- Default
-
1024
- rgw_usage_log_tick_interval
- Description
-
Flush pending log data every
sseconds. - Type
- Integer
- Required
- No
- Default
- 30
- rgw_intent_log_object_name
- Description, Type
- String
- Required
- No
- Default
-
%Y-%m-%d-%i-%n
- rgw_intent_log_object_name utc
- Description
- Include a UTC time stamp in the intent log object name.
- Type
- Boolean
- Required
- No
- Default
-
false
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}