Chapter 6. The ceph-volume utility
As a storage administrator, you can prepare, create, and activate Ceph OSDs using the ceph-volume utility. The ceph-volume utility is a single purpose command-line tool to deploy logical volumes as OSDs. It uses a plugin-type framework to deploying OSDs with different device technologies. The ceph-volume utility follows a similar workflow of the ceph-disk utility for deploying OSDs, with a predictable, and robust way of preparing, activating, and starting OSDs. Currently, the ceph-volume utility only supports the lvm plugin, with the plan to support others technologies in the future.
The ceph-disk command is deprecated.
6.1. Prerequisites
- A running Red Hat Ceph Storage cluster.
6.2. Ceph volume lvm plugin
By making use of LVM tags, the lvm sub-command is able to store and re-discover by querying devices associated with OSDs so they can be activated. This includes support for lvm-based technologies like dm-cache as well.
When using ceph-volume, the use of dm-cache is transparent, and treats dm-cache like a logical volume. The performance gains and losses when using dm-cache will depend on the specific workload. Generally, random and sequential reads will see an increase in performance at smaller block sizes. While random and sequential writes will see a decrease in performance at larger block sizes.
To use the LVM plugin, add lvm as a subcommand to the ceph-volume command:
[root@osd ~]# ceph-volume lvm
There are three subcommands to the lvm subcommand, as follows:
Using the create subcommand combines the prepare and activate subcommands into one subcommand.
Additional Resources
-
See the
createsubcommand section for more details.
6.3. Why does ceph-volume replace ceph-disk?
Previous versions of Red Hat Ceph Storage used the ceph-disk utility to prepare, activate, and create OSDs. Starting with Red Hat Ceph Storage 4, ceph-disk is replaced by the ceph-volume utility that aims to be a single purpose command-line tool to deploy logical volumes as OSDs, while maintaining a similar API to ceph-disk when preparing, activating, and creating OSDs.
How does ceph-volume work?
The ceph-volume is a modular tool that currently supports two ways of provisioning hardware devices, legacy ceph-disk devices and LVM (Logical Volume Manager) devices. The ceph-volume lvm command uses the LVM tags to store information about devices specific to Ceph and its relationship with OSDs. It uses these tags to later re-discover and query devices associated with OSDS so that it can activate them. It supports technologies based on LVM and dm-cache as well.
The ceph-volume utility uses dm-cache transparently and treats it as a logical volume. You might consider the performance gains and losses when using dm-cache, depending on the specific workload you are handling. Generally, the performance of random and sequential read operations increases at smaller block sizes; while the performance of random and sequential write operations decreases at larger block sizes. Using ceph-volume does not introduce any significant performance penalties.
The ceph-disk utility is deprecated.
The ceph-volume simple command can handle legacy ceph-disk devices, if these devices are still in use.
How does ceph-disk work?
The ceph-disk utility was required to support many different types of init systems, such as upstart or sysvinit, while being able to discover devices. For this reason, ceph-disk concentrates only on GUID Partition Table (GPT) partitions. Specifically on GPT GUIDs that label devices in a unique way to answer questions like:
-
Is this device a
journal? - Is this device an encrypted data partition?
- Was the device left partially prepared?
To solve these questions, ceph-disk uses UDEV rules to match the GUIDs.
What are disadvantages of using ceph-disk?
Using the UDEV rules to call ceph-disk can lead to a back-and-forth between the ceph-disk systemd unit and the ceph-disk executable. The process is very unreliable and time consuming and can cause OSDs to not come up at all during the boot process of a node. Moreover, it is hard to debug, or even replicate these problems given the asynchronous behavior of UDEV.
Because ceph-disk works with GPT partitions exclusively, it cannot support other technologies, such as Logical Volume Manager (LVM) volumes, or similar device mapper devices.
To ensure the GPT partitions work correctly with the device discovery workflow, ceph-disk requires a large number of special flags to be used. In addition, these partitions require devices to be exclusively owned by Ceph.
6.4. Preparing Ceph OSDs using ceph-volume
The prepare subcommand prepares an OSD back-end object store and consumes logical volumes (LV) for both the OSD data and journal. It does not modify the logical volumes, except for adding some extra metadata tags using LVM. These tags make volumes easier to discover, and they also identify the volumes as part of the Ceph Storage Cluster and the roles of those volumes in the storage cluster.
The BlueStore OSD backend supports the following configurations:
-
A block device, a
block.waldevice, and ablock.dbdevice -
A block device and a
block.waldevice -
A block device and a
block.dbdevice - A single block device
The prepare subcommand accepts a whole device or partition, or a logical volume for block.
Prerequisites
- Root-level access to the OSD nodes.
- Optionally, create logical volumes. If you provide a path to a physical device, the subcommand turns the device into a logical volume. This approach is simpler, but you cannot configure or change the way the logical volume is created.
Procedure
Prepare the LVM volumes:
Syntax
ceph-volume lvm prepare --bluestore --data VOLUME_GROUP/LOGICAL_VOLUME
Example
[root@osd ~]# ceph-volume lvm prepare --bluestore --data example_vg/data_lv
Optionally, if you want to use a separate device for RocksDB, specify the
--block.dband--block.waloptions:Syntax
ceph-volume lvm prepare --bluestore --block.db --block.wal --data VOLUME_GROUP/LOGICAL_VOLUME
Example
[root@osd ~]# ceph-volume lvm prepare --bluestore --block.db --block.wal --data example_vg/data_lv
Optionally, to encrypt data, use the
--dmcryptflag:Syntax
ceph-volume lvm prepare --bluestore --dmcrypt --data VOLUME_GROUP/LOGICAL_VOLUME
Example
[root@osd ~]# ceph-volume lvm prepare --bluestore --dmcrypt --data example_vg/data_lv
Additional Resources
- See the Activating Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
- See the Creating Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
6.5. Activating Ceph OSDs using ceph-volume
The activation process enables a systemd unit at boot time, which allows the correct OSD identifier and its UUID to be enabled and mounted.
Prerequisites
- A running Red Hat Ceph Storage cluster.
- Root-level access to the Ceph OSD node.
-
Ceph OSDs prepared by the
ceph-volumeutility.
Procedure
Get the OSD ID and UUID from an OSD node:
[root@osd ~]# ceph-volume lvm list
Activate the OSD:
Syntax
ceph-volume lvm activate --bluestore OSD_ID OSD_UUID
Example
[root@osd ~]# ceph-volume lvm activate --bluestore 0 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8
To activate all OSDs that are prepared for activation, use the
--alloption:Example
[root@osd ~]# ceph-volume lvm activate --all
Additional Resources
- See the Preparing Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
- See the Creating Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
6.6. Creating Ceph OSDs using ceph-volume
The create subcommand calls the prepare subcommand, and then calls the activate subcommand.
Prerequisites
- A running Red Hat Ceph Storage cluster.
- Root-level access to the Ceph OSD nodes.
If you prefer to have more control over the creation process, you can use the prepare and activate subcommands separately to create the OSD, instead of using create. You can use the two subcommands to gradually introduce new OSDs into a storage cluster, while avoiding having to rebalance large amounts of data. Both approaches work the same way, except that using the create subcommand causes the OSD to become up and in immediately after completion.
Procedure
To create a new OSD:
Syntax
ceph-volume lvm create --bluestore --data VOLUME_GROUP/LOGICAL_VOLUME
Example
[root@osd ~]# ceph-volume lvm create --bluestore --data example_vg/data_lv
Additional Resources
- See the Preparing Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
- See the Activating Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
6.7. Using batch mode with ceph-volume
The batch subcommand automates the creation of multiple OSDs when single devices are provided.
The ceph-volume command decides the best method to use to create the OSDs, based on drive type. Ceph OSD optimization depends on the available devices:
-
If all devices are traditional hard drives,
batchcreates one OSD per device. -
If all devices are solid state drives,
batchcreates two OSDs per device. -
If there is a mix of traditional hard drives and solid state drives,
batchuses the traditional hard drives for data, and creates the largest possible journal (block.db) on the solid state drive.
The batch subcommand does not support the creation of a separate logical volume for the write-ahead-log (block.wal) device.
Prerequisites
- A running Red Hat Ceph Storage cluster.
- Root-level access to the Ceph OSD nodes.
Procedure
To create OSDs on several drives:
Syntax
ceph-volume lvm batch --bluestore PATH_TO_DEVICE [PATH_TO_DEVICE]
Example
[root@osd ~]# ceph-volume lvm batch --bluestore /dev/sda /dev/sdb /dev/nvme0n1
Additional Resources
- See the Creating Ceph OSDs using `ceph-volume` section in the Red Hat Ceph Storage Administration Guide for more details.
