Chapter 8. File Store Configuration Reference
8.1. Extended Attributes
Extended attributes (XATTRs) are an important aspect in the CephFS configuration. Some file systems have limits on the number of bytes stored in extended attributes. Additionally, in some cases, the file system might not be as fast as an alternative method of storing extended attributes. The following settings improve CephFS performance by using a method of storing extended attributes that is extrinsic to the underlying file system.
Ceph extended attributes are stored as inline xattr
, using the extended attributes provided by the underlying file system, if it does not impose a size limit. If there is a size limit (4KB total on ext4, for instance), some Ceph extended attributes will be stored in an key-value database called omap
when the filestore max inline xattr size
or filestore max inline xattrs
threshold are reached.
- filestore_xattr_use_omap
- Description
-
Use object map for XATTRS. Set to
true
for ext4 file systems. - Type
- Boolean
- Required
- No
- Default
-
false
- filestore_omap_header_cache_size
- Description
-
Determines the size of the LRU used to cache object
omap
headers. Larger values use more memory but can reduce lookups onomap
. (Experts only). - Type
- Integer
- Default
-
1024
- filestore_omap_backend
- Description
-
Used to determine which back end is used for the
omap
. Can be set toleveldb
orrocksdb
. (Experts only.rocksdb
is experimental.) - Type
- String
- Default
-
leveldb
- filestore_debug_omap_check
- Description
- Debugging check on synchronization. Expensive. For debugging only.
- Type
- Boolean
- Required
- No
- Default
-
0
- filestore_max_inline_xattr_size
- Description
- The maximum size of an extended attribute stored in a file system (that is, XFS, btrfs, ext4, and others) per object. Should not be larger than the file system can handle.
- Type
- Unsigned 32-bit Integer
- Required
- No
- Default
-
512
- filestore_max_inline_xattrs
- Description
- The maximum number of extended attributes stored in the file system per object.
- Type
- 32-bit Integer
- Required
- No
- Default
-
2
- filestore_max_inline_xattr_size_xfs
- Description
- The maximum size of an extended attribute stored in the file system for XFS file systems per object. Should not be larger than the file system can handle.
- Type
- Unsigned 32-bit Integer
- Default
-
65536
- filestore_max_inline_xattr_size_btrfs
- Description
- The maximum size of an extended attribute stored in the file system for btrfs per object. Should not be larger than the file system can handle.
- Type
- Unsigned 32-bit Integer
- Default
-
2048
- filestore_max_inline_xattr_size_other
- Description
- The maximum size of an extended attribute stored in the file system for file systems other than btrfs or XFS per object. Should not be larger than the file system can handle.
- Type
- Unsigned 32-bit Integer
- Default
-
512
- filestore_max_inline_xattrs
- Description
- The maximum number of extended attributes stored in the file system per object. Overrides fine-grained settings.
- Type
- Unsigned 32-bit Integer
- Default
-
0
- filestore_max_inline_xattrs_xfs
- Description
- The maximum number of extended attributes stored in an XFS file system per object.
- Type
- Unsigned 32-bit Integer
- Default
-
10
- filestore_max_inline_xattrs_btrfs
- Description
- The maximum number of extended attributes stored in a btrfs file system per object.
- Type
- Unsigned 32-bit Integer
- Default
-
10
- filestore_max_inline_xattrs_other
- Description
- The maximum number of extended attributes stored in file systems other than btrfs or XFS per object.
- Type
- Unsigned 32-bit Integer
- Default
-
2
8.2. Synchronization Intervals
Periodically, the file store needs to quiesce write operations and synchronize the file system, which creates a consistent commit point. It can then free journal entries up to the commit point. Synchronizing more frequently tends to reduce the time required to perform synchronization, and reduces the amount of data that needs to remain in the journal. Less frequent synchronization allows the backing file system to coalesce small writes and metadata updates more optimally—potentially resulting in more efficient synchronization.
- filestore_max_sync_interval
- Description
- The maximum interval in seconds for synchronizing the file store.
- Type
- Double
- Required
- No
- Default
-
5
- filestore_min_sync_interval
- Description
- The minimum interval in seconds for synchronizing the file store.
- Type
- Double
- Required
- No
- Default
-
.01
8.3. Flusher
The file store flusher forces data from large write operations to be written out using the sync file range
option before the synchronization in order to reduce the cost of the eventual synchronization. In practice, disabling the file store flusher seems to improve performance in some cases.
- filestore_flusher
- Description
- Enables the file store flusher.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_flusher_max_fds
- Description
- Sets the maximum number of file descriptors for the flusher.
- Type
- Integer
- Required
- No
- Default
-
512
- filestore_sync_flush
- Description
- Enables the synchronization flusher.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_fsync_flushes_journal_data
- Description
- Flush journal data during file system synchronization.
- Type
- Boolean
- Required
- No
- Default
-
false
8.4. Queue
The following settings provide limits on the size of the file store queue.
- filestore_queue_max_ops
- Description
- Defines the maximum number of operations in progress that the file store accepts before blocking on queuing new operations.
- Type
- Integer
- Required
- No. Minimal impact on performance.
- Default
-
500
- filestore_queue_max_bytes
- Description
- The maximum number of bytes for an operation.
- Type
- Integer
- Required
- No
- Default
-
100 << 20
- filestore_queue_committing_max_ops
- Description
- The maximum number of operations that the file store can commit.
- Type
- Integer
- Required
- No
- Default
-
500
- filestore_queue_committing_max_bytes
- Description
- The maximum number of bytes that the file store can commit.
- Type
- Integer
- Required
- No
- Default
-
100 << 20
8.5. Writeback Throttle
Ceph replicates some of the write-back behavior in the kernel, because the page cache tends to keep dirty data round too long.
- filestore_wbthrottle_enable
- Description
- Enables the file store write-back throttle. The file store write-back throttle is used to prevent large amounts of uncommitted data from building up before each file store sync. (Experts only).
- Type
- Boolean
- Default
-
true
- filestore_wbthrottle_btrfs_bytes_start_flusher
- Description
- Dirty bytes threshold at which Ceph begins background flushing for the btrfs file system.
- Type
- 64-bit Unsigned Integer
- Default
-
41943040
- filestore_wbthrottle_btrfs_bytes_hard_limit
- Description
- Dirty bytes threshold at which Ceph begins to throttle I/O until the flusher catches up for btrfs.
- Type
- 64-bit Unsigned Integer
- Default
-
419430400
- filestore_wbthrottle_btrfs_ios_start_flusher
- Description
- Dirty I/Os threshold at which Ceph begins background flushing for btrfs.
- Type
- 64-bit Unsigned Integer
- Default
-
500
- filestore_wbthrottle_btrfs_ios_hard_limit
- Description
- Dirty I/Os threshold at which Ceph begins to throttle IO until the flusher catches up for btrfs.
- Type
- 64-bit Unsigned Integer
- Default
-
5000
- filestore_wbthrottle_btrfs_inodes_start_flusher
- Description
- Dirty inodes threshold at which Ceph begins background flushing for btrfs.
- Type
- 64-bit Unsigned Integer
- Default
-
500
- filestore_wbthrottle_btrfs_inodes_hard_limit
- Description
-
Dirty inodes threshold at which Ceph begins to throttle IO until the flusher catches up for btrfs. Must be less than the
fd
limit. - Type
- 64-bit Unsigned Integer
- Default
-
5000
- filestore_wbthrottle_xfs_bytes_start_flusher
- Description
- Dirty bytes threshold at which Ceph begins background flushing for the XFS file system.
- Type
- 64-bit Unsigned Integer
- Default
-
41943040
- filestore_wbthrottle_xfs_bytes_hard_limit
- Description
- Dirty bytes threshold at which Ceph begins to throttle IO until the flusher catches up for XFS.
- Type
- 64-bit Unsigned Integer
- Default
-
419430400
- filestore_wbthrottle_xfs_ios_start_flusher
- Description
- Dirty I/Os threshold at which Ceph begins background flushing for XFS.
- Type
- 64-bit Unsigned Integer
- Default
-
500
- filestore_wbthrottle_xfs_ios_hard_limit
- Description
- Dirty I/Os threshold at which Ceph begins to throttle IO until the flusher catches up for XFS.
- Type
- 64-bit Unsigned Integer
- Default
-
5000
- filestore_wbthrottle_xfs_inodes_start_flusher
- Description
- Dirty inodes threshold at which Ceph begins background flushing for XFS.
- Type
- 64-bit Unsigned Integer
- Default
-
500
- filestore_wbthrottle_xfs_inodes_hard_limit
- Description
-
Dirty inodes threshold at which Ceph begins to throttle IO until the flusher catches up for XFS. Must be less than the
fd
limit. - Type
- 64-bit Unsigned Integer
- Default
-
5000
8.6. Timeouts
- filestore_op_threads
- Description
- The number of file system operation threads that execute in parallel.
- Type
- Integer
- Required
- No
- Default
-
2
- filestore_op_thread_timeout
- Description
- The timeout for a file system operation thread (in seconds).
- Type
- Integer
- Required
- No
- Default
-
60
- filestore_op_thread_suicide_timeout
- Description
- The timeout for a commit operation before canceling the commit (in seconds).
- Type
- Integer
- Required
- No
- Default
-
180
8.7. B-Tree File System
- filestore_btrfs_snap
- Description
- Enable snapshots for a btrfs file store.
- Type
- Boolean
- Required
- No. Only used for btrfs.
- Default
-
true
- filestore_btrfs_clone_range
- Description
- Enable cloning ranges for a btrfs file store.
- Type
- Boolean
- Required
- No. Only used for btrfs.
- Default
-
true
8.8. Journal
- filestore_journal_parallel
- Description
- Enables parallel journaling, default for btrfs.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_journal_writeahead
- Description
- Enables write-ahead journaling, default for XFS.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_journal_trailing
- Description
- Deprecated, never use.
- Type
- Boolean
- Required
- No
- Default
-
false
8.9. Miscellaneous
- filestore_merge_threshold
- Description
- Minimum number of files in a subdirectory before merging into parent NOTE: A negative value means to disable subdirectory merging.
- Type
- Integer
- Required
- No
- Default
-
10
- filestore_split_multiple
- Description
-
filestore_split_multiple * abs(filestore_merge_threshold) * 16
is the maximum number of files in a subdirectory before splitting into child directories. - Type
- Integer
- Required
- No
- Default
-
2
- filestore_update_to
- Description
- Limits file store auto upgrade to specified version.
- Type
- Integer
- Required
- No
- Default
-
1000
- filestore_blackhole
- Description
- Drop any new transactions on the floor.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_dump_file
- Description
- File onto which store transaction dumps.
- Type
- Boolean
- Required
- No
- Default
-
false
- filestore_kill_at
- Description
- Inject a failure at the n’th opportunity.
- Type
- String
- Required
- No
- Default
-
false
- filestore_fail_eio
- Description
- Fail or terminate unexpectedly on EIO.
- Type
- Boolean
- Required
- No
- Default
-
true