Appendix F. Object Storage Daemon (OSD) configuration options
The following are Ceph Object Storage Daemon (OSD) configuration options that can be set during deployment.
You can set these configuration options with the ceph config set osd CONFIGURATION_OPTION VALUE command.
- osd_uuid
- Description
- The universally unique identifier (UUID) for the Ceph OSD.
- Type
- UUID
- Default
- The UUID.
- Note
-
The
osd uuidapplies to a single Ceph OSD. Thefsidapplies to the entire cluster.
- osd_data
- Description
The path to the OSD’s data. You must create the directory when deploying Ceph. Mount a drive for OSD data at this mount point.
IMPORTANT: Red Hat does not recommend changing the default.
- Type
- String
- Default
-
/var/lib/ceph/osd/$cluster-$id
- osd_max_write_size
- Description
- The maximum size of a write in megabytes.
- Type
- 32-bit Integer
- Default
-
90
- osd_client_message_size_cap
- Description
- The largest client data message allowed in memory.
- Type
- 64-bit Integer Unsigned
- Default
-
500MB default.
500*1024L*1024L
- osd_class_dir
- Description
- The class path for RADOS class plug-ins.
- Type
- String
- Default
-
$libdir/rados-classes
- osd_max_scrubs
- Description
- The maximum number of simultaneous scrub operations for a Ceph OSD.
- Type
- 32-bit Int
- Default
-
1
- osd_scrub_thread_timeout
- Description
- The maximum time in seconds before timing out a scrub thread.
- Type
- 32-bit Integer
- Default
-
60
- osd_scrub_finalize_thread_timeout
- Description
- The maximum time in seconds before timing out a scrub finalize thread.
- Type
- 32-bit Integer
- Default
-
60*10
- osd_scrub_begin_hour
- Description
-
This restricts scrubbing to this hour of the day or later. Use
osd_scrub_begin_hour = 0andosd_scrub_end_hour = 0to allow scrubbing the entire day. Along withosd_scrub_end_hour, they define a time window, in which the scrubs can happen. But a scrub is performed no matter whether the time window allows or not, as long as the placement group’s scrub interval exceedsosd_scrub_max_interval. - Type
- Integer
- Default
-
0 - Allowed range
-
[0,23]
- osd_scrub_end_hour
- Description
-
This restricts scrubbing to the hour earlier than this. Use
osd_scrub_begin_hour = 0andosd_scrub_end_hour = 0to allow scrubbing for the entire day. Along withosd_scrub_begin_hour, they define a time window, in which the scrubs can happen. But a scrub is performed no matter whether the time window allows or not, as long as the placement group’s scrub interval exceedsosd_scrub_max_interval. - Type
- Integer
- Default
-
0 - Allowed range
-
[0,23]
- osd_scrub_load_threshold
- Description
-
The maximum load. Ceph will not scrub when the system load (as defined by the
getloadavg()function) is higher than this number. Default is0.5. - Type
- Float
- Default
-
0.5
- osd_scrub_min_interval
- Description
- The minimum interval in seconds for scrubbing the Ceph OSD when the Red Hat Ceph Storage cluster load is low.
- Type
- Float
- Default
-
Once per day.
60*60*24
- osd_scrub_max_interval
- Description
- The maximum interval in seconds for scrubbing the Ceph OSD irrespective of cluster load.
- Type
- Float
- Default
-
Once per week.
7*60*60*24
- osd_scrub_interval_randomize_ratio
- Description
-
Takes the ratio and randomizes the scheduled scrub between
osd scrub min intervalandosd scrub max interval. - Type
- Float
- Default
-
0.5.
- mon_warn_not_scrubbed
- Description
-
Number of seconds after
osd_scrub_intervalto warn about any PGs that were not scrubbed. - Type
- Integer
- Default
-
0(no warning).
- osd_scrub_chunk_min
- Description
-
The object store is partitioned into chunks which end on hash boundaries. For chunky scrubs, Ceph scrubs objects one chunk at a time with writes blocked for that chunk. The
osd scrub chunk minsetting represents the minimum number of chunks to scrub. - Type
- 32-bit Integer
- Default
-
5
- osd_scrub_chunk_max
- Description
- The maximum number of chunks to scrub.
- Type
- 32-bit Integer
- Default
-
25
- osd_scrub_sleep
- Description
- The time to sleep between deep scrub operations.
- Type
- Float
- Default
-
0(or off).
- osd_scrub_during_recovery
- Description
- Allows scrubbing during recovery.
- Type
- Bool
- Default
-
false
- osd_scrub_invalid_stats
- Description
- Forces extra scrub to fix stats marked as invalid.
- Type
- Bool
- Default
-
true
- osd_scrub_priority
- Description
- Controls queue priority of scrub operations versus client I/O.
- Type
- Unsigned 32-bit Integer
- Default
-
5
- osd_requested_scrub_priority
- Description
-
The priority set for user requested scrub on the work queue. If this value were to be smaller than
osd_client_op_priority, it can be boosted to the value ofosd_client_op_prioritywhen scrub is blocking client operations. - Type
- Unsigned 32-bit Integer
- Default
-
120
- osd_scrub_cost
- Description
- Cost of scrub operations in megabytes for queue scheduling purposes.
- Type
- Unsigned 32-bit Integer
- Default
-
52428800
- osd_deep_scrub_interval
- Description
-
The interval for deep scrubbing, that is fully reading all data. The
osd scrub load thresholdparameter does not affect this setting. - Type
- Float
- Default
-
Once per week.
60*60*24*7
- osd_deep_scrub_stride
- Description
- Read size when doing a deep scrub.
- Type
- 32-bit Integer
- Default
-
512 KB.
524288
- mon_warn_not_deep_scrubbed
- Description
-
Number of seconds after
osd_deep_scrub_intervalto warn about any PGs that were not scrubbed. - Type
- Integer
- Default
-
0(no warning)
- osd_deep_scrub_randomize_ratio
- Description
-
The rate at which scrubs will randomly become deep scrubs (even before
osd_deep_scrub_intervalhas passed). - Type
- Float
- Default
-
0.15or 15%
- osd_deep_scrub_update_digest_min_age
- Description
- How many seconds old objects must be before scrub updates the whole-object digest.
- Type
- Integer
- Default
-
7200(120 hours)
- osd_deep_scrub_large_omap_object_key_threshold
- Description
- Warning when you encounter an object with more OMAP keys than this.
- Type
- Integer
- Default
-
200000
- osd_deep_scrub_large_omap_object_value_sum_threshold
- Description
- Warning when you encounter an object with more OMAP key bytes than this.
- Type
- Integer
- Default
-
1 G
- osd_delete_sleep
- Description
- Time in seconds to sleep before the next removal transaction. This throttles the placement group deletion process.
- Type
- Float
- Default
-
0.0
- osd_delete_sleep_hdd
- Description
- Time in seconds to sleep before the next removal transaction for HDDs.
- Type
- Float
- Default
-
5.0
- osd_delete_sleep_ssd
- Description
- Time in seconds to sleep before the next removal transaction for SSDs.
- Type
- Float
- Default
-
1.0
- osd_delete_sleep_hybrid
- Description
- Time in seconds to sleep before the next removal transaction when Ceph OSD data is on HDD and OSD journal or WAL and DB is on SSD.
- Type
- Float
- Default
-
1.0
- osd_op_num_shards
- Description
- The number of shards for client operations.
- Type
- 32-bit Integer
- Default
-
0
- osd_op_num_threads_per_shard
- Description
- The number of threads per shard for client operations.
- Type
- 32-bit Integer
- Default
-
0
- osd_op_num_shards_hdd
- Description
- The number of shards for HDD operations.
- Type
- 32-bit Integer
- Default
-
5
- osd_op_num_threads_per_shard_hdd
- Description
- The number of threads per shard for HDD operations.
- Type
- 32-bit Integer
- Default
-
1
- osd_op_num_shards_ssd
- Description
- The number of shards for SSD operations.
- Type
- 32-bit Integer
- Default
-
8
- osd_op_num_threads_per_shard_ssd
- Description
- The number of threads per shard for SSD operations.
- Type
- 32-bit Integer
- Default
-
2
- osd_op_queue
- Description
- Sets the type of queue to be used for operation prioritizing within Ceph OSDs. Requires a restart of the OSD daemons.
- Type
- String
- Default
-
wpq - Valid choices
-
wpq,mclock_scheduler,debug_random
Important
The mClock OSD scheduler is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. See the support scope for Red Hat Technology Preview features for more details.
osd_op_queue_cut_off- Description
Selects which priority operations are sent to the strict queue and which are sent to the normal queue. Requires a restart of the OSD daemons.
The low setting sends all replication and higher operations to the strict queue, while the high option sends only replication acknowledgment operations and higher to the strict queue.
The high setting helps when some Ceph OSDs in the cluster are very busy, especially when combined with the
wpqoption in theosd_op_queuesetting. Ceph OSDs that are very busy handling replication traffic can deplete primary client traffic on these OSDs without these settings.- Type
- String
- Default
-
high - Valid choices
-
low,high,debug_random
- osd_client_op_priority
- Description
-
The priority set for client operations. It is relative to
osd recovery op priority. - Type
- 32-bit Integer
- Default
-
63 - Valid Range
- 1-63
- osd_recovery_op_priority
- Description
-
The priority set for recovery operations. It is relative to
osd client op priority. - Type
- 32-bit Integer
- Default
-
3 - Valid Range
- 1-63
- osd_op_thread_timeout
- Description
- The Ceph OSD operation thread timeout in seconds.
- Type
- 32-bit Integer
- Default
-
15
- osd_op_complaint_time
- Description
- An operation becomes complaint worthy after the specified number of seconds have elapsed.
- Type
- Float
- Default
-
30
- osd_disk_threads
- Description
- The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming.
- Type
- 32-bit Integer
- Default
-
1
- osd_op_history_size
- Description
- The maximum number of completed operations to track.
- Type
- 32-bit Unsigned Integer
- Default
-
20
- osd_op_history_duration
- Description
- The oldest completed operation to track.
- Type
- 32-bit Unsigned Integer
- Default
-
600
- osd_op_log_threshold
- Description
- How many operations logs to display at once.
- Type
- 32-bit Integer
- Default
-
5
- osd_op_timeout
- Description
- The time in seconds after which running OSD operations time out.
- Type
- Integer
- Default
-
0
Important
Do not set the osd op timeout option unless your clients can handle the consequences. For example, setting this parameter on clients running in virtual machines can lead to data corruption because the virtual machines interpret this timeout as a hardware failure.
- osd_max_backfills
- Description
- The maximum number of backfill operations allowed to or from a single OSD.
- Type
- 64-bit Unsigned Integer
- Default
-
1
- osd_backfill_scan_min
- Description
- The minimum number of objects per backfill scan.
- Type
- 32-bit Integer
- Default
-
64
- osd_backfill_scan_max
- Description
- The maximum number of objects per backfill scan.
- Type
- 32-bit Integer
- Default
-
512
- osd_backfill_full_ratio
- Description
- Refuse to accept backfill requests when the Ceph OSD’s full ratio is above this value.
- Type
- Float
- Default
-
0.85
- osd_backfill_retry_interval
- Description
- The number of seconds to wait before retrying backfill requests.
- Type
- Double
- Default
-
30.000000
- osd_map_dedup
- Description
- Enable removing duplicates in the OSD map.
- Type
- Boolean
- Default
-
true
- osd_map_cache_size
- Description
- The size of the OSD map cache in megabytes.
- Type
- 32-bit Integer
- Default
-
50
- osd_map_cache_bl_size
- Description
- The size of the in-memory OSD map cache in OSD daemons.
- Type
- 32-bit Integer
- Default
-
50
- osd_map_cache_bl_inc_size
- Description
- The size of the in-memory OSD map cache incrementals in OSD daemons.
- Type
- 32-bit Integer
- Default
-
100
- osd_map_message_max
- Description
- The maximum map entries allowed per MOSDMap message.
- Type
- 32-bit Integer
- Default
-
40
- osd_snap_trim_thread_timeout
- Description
- The maximum time in seconds before timing out a snap trim thread.
- Type
- 32-bit Integer
- Default
-
60*60*1
- osd_pg_max_concurrent_snap_trims
- Description
- The max number of parallel snap trims/PG. This controls how many objects per PG to trim at once.
- Type
- 32-bit Integer
- Default
-
2
- osd_snap_trim_sleep
- Description
- Insert a sleep between every trim operation a PG issues.
- Type
- 32-bit Integer
- Default
-
0
- osd_snap_trim_sleep_hdd
- Description
- Time in seconds to sleep before the next snapshot trimming for HDDs.
- Type
- Float
- Default
-
5.0
- osd_snap_trim_sleep_ssd
- Description
- Time in seconds to sleep before the next snapshot trimming operation for SSD OSDs, including NVMe.
- Type
- Float
- Default
-
0.0
- osd_snap_trim_sleep_hybrid
- Description
- Time in seconds to sleep before the next snapshot trimming operation when OSD data is on an HDD and the OSD journal or WAL and DB is on an SSD.
- Type
- Float
- Default
-
2.0
- osd_max_trimming_pgs
- Description
- The max number of trimming PGs
- Type
- 32-bit Integer
- Default
-
2
- osd_backlog_thread_timeout
- Description
- The maximum time in seconds before timing out a backlog thread.
- Type
- 32-bit Integer
- Default
-
60*60*1
- osd_default_notify_timeout
- Description
- The OSD default notification timeout (in seconds).
- Type
- 32-bit Integer Unsigned
- Default
-
30
- osd_check_for_log_corruption
- Description
- Check log files for corruption. Can be computationally expensive.
- Type
- Boolean
- Default
-
false
- osd_remove_thread_timeout
- Description
- The maximum time in seconds before timing out a remove OSD thread.
- Type
- 32-bit Integer
- Default
-
60*60
- osd_command_thread_timeout
- Description
- The maximum time in seconds before timing out a command thread.
- Type
- 32-bit Integer
- Default
-
10*60
- osd_command_max_records
- Description
- Limits the number of lost objects to return.
- Type
- 32-bit Integer
- Default
-
256
- osd_auto_upgrade_tmap
- Description
-
Uses
tmapforomapon old objects. - Type
- Boolean
- Default
-
true
- osd_tmapput_sets_users_tmap
- Description
-
Uses
tmapfor debugging only. - Type
- Boolean
- Default
-
false
- osd_preserve_trimmed_log
- Description
- Preserves trimmed log files, but uses more disk space.
- Type
- Boolean
- Default
-
false
- osd_recovery_delay_start
- Description
- After peering completes, Ceph delays for the specified number of seconds before starting to recover objects.
- Type
- Float
- Default
-
0
- osd_recovery_max_active
- Description
- The number of active recovery requests per OSD at one time. More requests will accelerate recovery, but the requests place an increased load on the cluster.
- Type
- 32-bit Integer
- Default
-
0
- osd_recovery_max_active_hdd
- Description
- The number of active recovery requests per Ceph OSD at one time, if the primary device is HDD.
- Type
- Integer
- Default
-
3
- osd_recovery_max_active_ssd
- Description
- The number of active recovery requests per Ceph OSD at one time, if the primary device is SSD.
- Type
- Integer
- Default
-
10
- osd_recovery_sleep
- Description
- Time in seconds to sleep before the next recovery or backfill operation. Increasing this value slows down recovery operation while client operations are less impacted.
- Type
- Float
- Default
-
0.0
- osd_recovery_sleep_hdd
- Description
- Time in seconds to sleep before the next recovery or backfill operation for HDDs.
- Type
- Float
- Default
-
0.1
- osd_recovery_sleep_ssd
- Description
- Time in seconds to sleep before the next recovery or backfill operation for SSDs.
- Type
- Float
- Default
-
0.0
- osd_recovery_sleep_hybrid
- Description
- Time in seconds to sleep before the next recovery or backfill operation when Ceph OSD data is on HDD and OSD journal or WAL and DB is on SSD.
- Type
- Float
- Default
-
0.025
- osd_recovery_max_chunk
- Description
- The maximum size of a recovered chunk of data to push.
- Type
- 64-bit Integer Unsigned
- Default
-
8388608
- osd_recovery_threads
- Description
- The number of threads for recovering data.
- Type
- 32-bit Integer
- Default
-
1
- osd_recovery_thread_timeout
- Description
- The maximum time in seconds before timing out a recovery thread.
- Type
- 32-bit Integer
- Default
-
30
- osd_recover_clone_overlap
- Description
-
Preserves clone overlap during recovery. Should always be set to
true. - Type
- Boolean
- Default
-
true
- rados_osd_op_timeout
- Description
- Number of seconds that RADOS waits for a response from the OSD before returning an error from a RADOS operation. A value of 0 means no limit.
- Type
- Double
- Default
- 0
