Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 29. Logging and Debugging

Typically, when you add debugging to your Ceph configuration, you do so at runtime. You can also add Ceph debug logging to your Ceph configuration file if you are encountering issues when starting your cluster. You may view Ceph log files under /var/log/ceph (the default location).

Tip

When debug output slows down your system, the latency can hide race conditions.

Logging is resource intensive. If you are encountering a problem in a specific area of your cluster, enable logging for that area of the cluster. For example, if your OSDs are running fine, but your gateways are not, you should start by enabling debug logging for the specific gateway instance(s) giving you trouble. Enable logging for each subsystem as needed.

Important

Verbose logging can generate over 1GB of data per hour. If your OS disk reaches its capacity, the node will stop working.

If you enable or increase the rate of Ceph logging, ensure that you have sufficient disk space on your OS disk. See Accelerating Log Rotation for details on rotating log files. When your system is running well, remove unnecessary debugging settings to ensure your cluster runs optimally. Logging debug output messages is relatively slow, and a waste of resources when operating your cluster.

See Subsystem, Log and Debug Settings for details on available settings.

29.1. Runtime
Copy link

If you would like to see the configuration settings at runtime, you must log in to a host with a running daemon and execute the following: 

ceph --admin-daemon </path/to/admin/socket> config show | less
ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
Copy to Clipboard Toggle word wrap

To activate Ceph’s debugging output (i.e., dout()) at runtime, use the ceph tell command to inject arguments into the runtime configuration: 

ceph tell <daemon-type>.<daemon id or *> injectargs --<name> <value> [--<name> <value>]
Copy to Clipboard Toggle word wrap

Replace <daemon-type> with one of osd or mon. You may apply the runtime setting to all daemons of a particular type with *, or specify a specific daemon’s ID (i.e., its number or letter). For example, to increase debug logging for a ceph-osd daemon named osd.0, execute the following: 

ceph tell osd.0 injectargs --debug-osd 0/5
Copy to Clipboard Toggle word wrap

The ceph tell command goes through the monitors. If you cannot bind to the monitor, you can still make the change by logging into the host of the daemon whose configuration you’d like to change using ceph --admin-daemon. For example: 

sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set debug_osd 0/5
Copy to Clipboard Toggle word wrap

See Subsystem, Log and Debug Settings for details on available settings.

29.2. Boot Time
Copy link

To activate Ceph’s debugging output (i.e., dout()) at boot time, you must add settings to your Ceph configuration file. Subsystems common to each daemon may be set under [global] in your configuration file. Subsystems for particular daemons are set under the daemon section in your configuration file (e.g., [mon], [osd]). For example: 

[global]
    debug ms = 1/5

[mon]
    debug mon = 20
    debug paxos = 1/5
    debug auth = 2

[osd]
    debug osd = 1/5
    debug filestore = 1/5
    debug journal = 1
    debug monc = 5/20
Copy to Clipboard Toggle word wrap

See Subsystem, Log and Debug Settings for details on available settings.

29.3. Accelerating Log Rotation
Copy link

If your OS disk is relatively full, you can accelerate log rotation by modifying the Ceph log rotation file at /etc/logrotate.d/ceph. Add a size setting after the rotation frequency to accelerate log rotation (via cronjob) if your logs exceed the size setting. For example, the default setting looks like this: 

rotate 7
weekly
compress
sharedscripts
Copy to Clipboard Toggle word wrap

Modify it by adding a size setting. 

rotate 7
weekly
size 500M
compress
sharedscripts
Copy to Clipboard Toggle word wrap

Then, start the crontab editor for your user space. 

crontab -e
Copy to Clipboard Toggle word wrap

Finally, add an entry to check the etc/logrotate.d/ceph file. 

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
Copy to Clipboard Toggle word wrap

The preceding example checks the etc/logrotate.d/ceph file every 30 minutes.

29.4. Valgrind
Copy link

Debugging may also require you to track down memory and threading issues. You can run a single daemon, a type of daemon, or the whole cluster with Valgrind. You should only use Valgrind when developing or debugging Ceph. Valgrind is computationally expensive, and will slow down your system otherwise. Valgrind messages are logged to stderr.

29.5. Subsystem, Log and Debug Settings
Copy link

In most cases, you will enable debug logging output via subsystems.

29.5.1. Ceph Subsystems
Copy link

Each subsystem has a logging level for its output logs, and for its logs in-memory. You may set different values for each of these subsystems by setting a log file level and a memory level for debug logging. Ceph’s logging levels operate on a scale of 1 to 20, where 1 is terse and 20 is verbose.

A debug logging setting can take a single value for the log level and the memory level, which sets them both as the same value. For example, if you specify debug ms = 5, Ceph will treat it as a log level and a memory level of 5. You may also specify them separately. The first setting is the log level, and the second setting is the memory level. You must separate them with a forward slash (/). For example, if you want to set the ms subsystem’s debug logging level to 1 and its memory level to 5, you would specify it as debug ms = 1/5. For example: 

debug <subsystem> = <log-level>/<memory-level>
#for example
debug osd log = 1/20
Copy to Clipboard Toggle word wrap

The following table provides a list of Ceph subsystems and their default log and memory levels. Once you complete your logging efforts, restore the subsystems to their default level or to a level suitable for normal operations.

Expand
SubsystemLog LevelMemory Level

default

0

5

lockdep

0

5

context

0

5

crush

1

5

buffer

0

0

timer

0

5

filer

0

5

objecter

0

0

rados

0

5

rbd

0

5

journaler

0

5

objectcacher

0

5

client

0

5

osd

0

5

optracker

0

5

objclass

0

5

filestore

1

5

journal

1

5

ms

0

5

mon

1

5

monc

0

5

paxos

0

5

tp

0

5

auth

1

5

finisher

1

5

heartbeatmap

1

5

perfcounter

1

5

rgw

1

5

javaclient

1

5

asok

1

5

throttle

1

5

29.5.2. Logging Settings
Copy link

Logging and debugging settings are not required in a Ceph configuration file, but you may override default settings as needed. Ceph supports the following settings:

log file

Description
The location of the logging file for your cluster.
Type
String
Required
No
Default
/var/log/ceph/$cluster-$name.log

log max new

Description
The maximum number of new log files.
Type
Integer
Required
No
Default
1000

log max recent

Description
The maximum number of recent events to include in a log file.
Type
Integer
Required
No
Default
1000000

log to stderr

Description
Determines if logging messages should appear in stderr.
Type
Boolean
Required
No
Default
true

err to stderr

Description
Determines if error messages should appear in stderr.
Type
Boolean
Required
No
Default
true

log to syslog

Description
Determines if logging messages should appear in syslog.
Type
Boolean
Required
No
Default
false

err to syslog

Description
Determines if error messages should appear in syslog.
Type
Boolean
Required
No
Default
false

log flush on exit

Description
Determines if Ceph should flush the log files after exit.
Type
Boolean
Required
No
Default
true

clog to monitors

Description
Determines if clog messages should be sent to monitors.
Type
Boolean
Required
No
Default
true

clog to syslog

Description
Determines if clog messages should be sent to syslog.
Type
Boolean
Required
No
Default
false

mon cluster log to syslog

Description
Determines if the cluster log should be output to the syslog.
Type
Boolean
Required
No
Default
false

mon cluster log file

Description
The location of the cluster’s log file.
Type
String
Required
No
Default
/var/log/ceph/$cluster.log

29.5.3. OSD
Copy link

osd preserve trimmed log

Description
Preserves trimmed logs after trimming.
Type
Boolean
Required
No
Default
false

osd tmapput sets uses tmap

Description
Uses tmap. For debug only.
Type
Boolean
Required
No
Default
false

osd min pg log entries

Description
The minimum number of log entries for placement groups.
Type
32-bit Unsigned Integer
Required
No
Default
1000

osd op log threshold

Description
How many op log messages to show up in one pass.
Type
Integer
Required
No
Default
5

29.5.4. Filestore
Copy link

filestore debug omap check

Description
Debugging check on synchronization. This is an expensive operation.
Type
Boolean
Required
No
Default
0

29.5.5. RADOS Gateway
Copy link

rgw log nonexistent bucket

Description
Log non-existent buckets.
Type
Boolean
Required
No
Default
false

rgw log object name

Description
Log an object’s name.
Type
String
Required
No
Default
%Y-%m-%d-%H-%i-%n

rgw log object name utc

Description
Object log name contains UTC.
Type
Boolean
Required
No
Default
false

rgw enable ops log

Description
Enables logging of every RGW operation.
Type
Boolean
Required
No
Default
true

rgw enable usage log

Description
Enable logging of RGW’s bandwidth usage.
Type
Boolean
Required
No
Default
true

rgw usage log flush threshold

Description
Threshold to flush pending log data.
Type
Integer
Required
No
Default
1024

rgw usage log tick interval

Description
Flush pending log data every s seconds.
Type
Integer
Required
No
Default
30

rgw intent log object name

Description, Type
String
Required
No
Default
%Y-%m-%d-%i-%n

rgw intent log object name utc

Description
Include a UTC timestamp in the intent log object name.
Type
Boolean
Required
No
Default
false
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat