Red Hat SummitEste conteúdo não está disponível no idioma selecionado.
Chapter 10. Using NVMe with LVM Optimally
Summary
The procedures below demonstrate how to deploy Ceph for Object Gateway usage optimally when using high speed NVMe based SSDs (this applies to SATA SSDs too). Journals and bucket indexes will be placed together on high speed storage devices, which can increase performance compared to having all journals on one device. This configuration requires setting
osd_scenariotolvm.Procedures for two example configurations are provided:
- One NVMe device and at least four HDDs using one bucket index: One NVMe device
- Two NVMe devices and at least four HDDs using two bucket indexes: Two NVMe devices
Details
The most basic Ceph setup uses the
osd_scenariosetting ofcollocated. This stores the OSD data and its journal on one storage device together (they are "co-located"). Typical server configurations include both HDDs and SSDs. Since HDDs are usually larger than SSDs, in a collocated configuration to utitlize the most storage space an HDD would be chosen, putting both the OSD data and journal on it alone. However, the journal should ideally be on a faster SSD. Another option is using theosd_scenariosetting ofnon-collocated. This allows configuration of dedicated devices for journals, so you can put the OSD data on HDDs and the journals on SSDs.In addition to OSD data and journals, when using Object Gateway a bucket index needs to be stored on a device. In this case Ceph is often configured so that HDDs hold the OSD data, one SSD holds the journals, and another SSD holds the bucket indexes. This can create highly imbalanced situations where the SSD with all the journals becomes saturated while the SSD with bucket indexes is underutilized.
The solution is to set
osd_scenariotolvmand use Logical Volume Manager (LVM) to divide up single SSD devices for more than one purpose. This allows journals and bucket indexes to exist side by side on a single device. Most importantly, it allows journals to exist on more than one SSD, spreading the intense IO data transfer of the journals across more than one device.The normal Ansible playbooks provided by the
ceph-ansibleRPM used to install Ceph (site.yml, osds.yml, etc.) don’t support using one device for more than one purpose.In the future the normal Ansible playbooks will support using one device for more than one purpose. In the meantime the playbooks
lv-create.ymlandlv-vars.yamlare being provided to facilitate creating the required Logicial Volumes (LVs) for optimal SSD usage. Afterlv-create.ymlis runsite.ymlcan be run normally and it will use the newly created LVs.ImportantThese procedures only apply to the FileStore storage backend, not the newer BlueStore storage backend.
10.1. Using One NVMe Device
Follow this procedure to deploy Ceph for Object Gateway usage with one NVMe device.
10.1.1. Purge Any Existing Ceph Cluster
If Ceph is already configured, purge it in order to start over. The Ansible playbook, purge-cluster.yml, is provided for this purpose.
ansible-playbook infrastructure-playbooks/purge-cluster.yml -i hosts
$ ansible-playbook infrastructure-playbooks/purge-cluster.yml -i hosts
For more information on how to use purge-cluster.yml see Purging a Ceph Cluster by Using Ansible in the Installation Guide.
Purging the cluster may not be enough to prepare the servers for redeploying Ceph using the following procedures. Any file system, GPT, RAID, or other signatures on storage devices used by Ceph may cause problems. Instructions to remove any signatures using wipefs are provided under Run The lv-create.yml Ansible Playbook.
10.1.2. Configure The Cluster for Normal Installation
Setting aside any NVMe and/or LVM considerations, configure the cluster as you would normally but stop before running the Ansible playbook. Afterwards, the cluster installation configuration will be adjusted specifically for optimal NVMe/LVM usage to support the Object Gateway. Only at that time should the Ansible playbook be ran.
To configure the storage cluster for normal installation consult the Red Hat Ceph Storage Installation Guide. In particular, complete the steps in Installing a Red Hat Ceph Storage Cluster through Step 9 creating an Ansible log directory. Stop before Step 10, when ansible-playbook site.yml -i hosts is ran.
Do not run ansible-playbook site.yml -i hosts until all the steps after this and before Install Ceph for NVMe and Verify Success have been completed.
10.1.3. Identify The NVMe and HDD Devices
Use lsblk to identify the NVMe and HDD devices connected to the server. Example output from lsblk is listed below:
In this example the following raw block devices will be used:
NVMe devices
-
/dev/nvme0n1
HDD devices
-
/dev/sdc -
/dev/sdd -
/dev/sde -
/dev/sdf
The file lv_vars.yaml configures logical volume creation on the chosen devices. It creates journals on NVMe, an NVMe based bucket index, and HDD based OSDs. The actual creation of logical volumes is initiated by lv-create.yml, which reads lv_vars.yaml.
That file should only have one NVMe device referenced in it at a time. For information on using Ceph with two NVMe devices optimally see Using Two NVMe Devices.
10.1.4. Add The Devices to lv_vars.yaml
As
root, navigate to the/usr/share/ceph-ansible/directory:cd /usr/share/ceph-ansible
# cd /usr/share/ceph-ansibleCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the file so it includes the following lines:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.1.5. Run The lv-create.yml Ansible Playbook
The purpose of the lv-create.yml playbook is to create logical volumes for the object gateway bucket index, and journals, on a single NVMe. It does this by using osd_scenario=lvm. The lv-create.yml Ansible playbook makes it easier to configure Ceph in this way by automating some of the complex LVM creation and configuration.
Ensure the storage devices are raw
Before running
lv-create.ymlto create the logical volumes on the NVMe devices and HDD devices, ensure there are no file system, GPT, RAID, or other signatures on them.If they are not raw, when you run
lv-create.ymlit might fail with the following error:device /dev/sdc excluded by a filter
device /dev/sdc excluded by a filterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wipe storage device signatures (optional):
If the devices have signatures you can use
wipefsto erase them.An example of using
wipefsto erase the devices is shown below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
lv-teardown.ymlAnsible playbook:Always run
lv-teardown.ymlbefore runninglv-create.yml:Run the
lv-teardown.ymlAnsible playbook:ansible-playbook infrastructure-playbooks/lv-teardown.yml -i hosts
$ ansible-playbook infrastructure-playbooks/lv-teardown.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow WarningProceed with caution when running the
lv-teardown.ymlAnsible script. It destroys data. Ensure you have backups of any important data.Run the
lv-create.ymlAnsible playbook:ansible-playbook infrastructure-playbooks/lv-create.yml -i hosts
$ ansible-playbook infrastructure-playbooks/lv-create.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Once
lv-create.ymlcompletes without error continue to the next section to verify it worked properly.
10.1.6. Verify LVM Configuration
Review
lv-created.log:Once the
lv-create.ymlAnsible playbook completes successfully, configuration information will be written tolv-created.log. Later this information will be copied intogroup_vars/osds.yml. Openlv-created.logand look for information similar to the below example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Review LVM configuration
Based on the example of one NVMe device and four HDDs the following Logical Volumes (LVs) should be created:
One journal LV per HDD placed on NVMe (four LVs on /dev/nvme0n1)
One data LV per HDD placed on each HDD (one LV per HDD)
One journal LV for bucket index placed on NVMe (one LV on /dev/nvme0n1)
One data LV for bucket index placed on NVMe (one LV on /dev/nvme0n1)
The LVs can be seen in
lsblkandlvscanoutput. In the example explained above, there should be ten LVs for Ceph. As a rough sanity check you could count the Ceph LVs to make sure there are at least ten, but ideally you would make sure the appropriate LVs were created on the right storage devices (NVMe vs HDD).Example output from
lsblkis shown below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
lvscanoutput is below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.1.7. Edit The osds.yml and all.yml Ansible Playbooks
-
Copy the previously mentioned configuration information from
lv-created.logintogroup_vars/osds.ymlunder thelvm_volumes:line. Set
osd_scenario:tolvm:osd_scenario: lvm
osd_scenario: lvmCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set
osd_objectstore: filestoreinall.ymlandosds.yml.The
osds.ymlfile should look similar to this:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.1.8. Install Ceph for NVMe and Verify Success
After configuring Ceph for installation to use NVMe with LVM optimally, install it.
Run the
site.ymlAnsible playbook to install Cephansible-playbook -v site.yml -i hosts
$ ansible-playbook -v site.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify Ceph is running properly after install completes
ceph -s
# ceph -sCopy to Clipboard Copied! Toggle word wrap Toggle overflow ceph osd tree
# ceph osd treeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example
ceph -soutput showing Ceph is running properly:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
ceph osd treeoutput showing Ceph is running properly:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Ceph is now set up to use one NVMe device and LVM optimally for the Ceph Object Gateway.
10.2. Using Two NVMe Devices
Follow this procedure to deploy Ceph for Object Gateway usage with two NVMe devices.
10.2.1. Purge Any Existing Ceph Cluster
If Ceph is already configured, purge it in order to start over. An ansible playbook file named purge-cluster.yml is provided for this purpose.
ansible-playbook purge-cluster.yml -i hosts
$ ansible-playbook purge-cluster.yml -i hosts
For more information on how to use purge-cluster.yml see Purging a Ceph Cluster by Using Ansible in the Installation Guide.
Purging the cluster may not be enough to prepare the servers for redeploying Ceph using the following procedures. Any file system, GPT, RAID, or other signatures on storage devices used by Ceph may cause problems. Instructions to remove any signatures using wipefs are provided under Run The lv-create.yml Ansible Playbook.
10.2.2. Configure The Cluster for Normal Installation
Setting aside any NVMe and/or LVM considerations, configure the cluster as you would normally but stop before running Ansible playbook. Afterwards, the cluster installation configuration will be adjusted specifically for optimal NVMe/LVM usage to support the Object Gateway. Only at that time should Ansible playbook be ran.
To configure the cluster for normal installation consult the Installation Guide. In particular, complete the steps in Installing a Red Hat Ceph Storage Cluster through Step 9 creating an Ansible log directory. Stop before Step 10 when ansible-playbook site.yml -i hosts is ran.
Do not run ansible-playbook site.yml -i hosts until all the steps after this and before Install Ceph for NVMe and Verify Success have been completed.
10.2.3. Identify The NVMe and HDD Devices
Use lsblk to identify the NVMe and HDD devices connected to the server. Example output from lsblk is listed below:
In this example the following raw block devices will be used:
NVMe devices
-
/dev/nvme0n1 -
/dev/nvme1n1
HDD devices
-
/dev/sdc -
/dev/sdd -
/dev/sde -
/dev/sdf
The file lv_vars.yaml configures logical volume creation on the chosen devices. It creates journals on NVMe, an NVMe based bucket index, and HDD based OSDs. The actual creation of logical volumes is initiated by lv-create.yml, which reads lv_vars.yaml.
That file should only have one NVMe device referenced in it at a time. It should also only reference the HDD devices to be associated with that particular NVMe device. For OSDs that contain more than one NVMe device edit lv_vars.yaml for each NVMe and run lv-create.yml repeatedly for each NVMe. This is explained below.
In the example this means lv-create.yml will first be run on /dev/nvme0n1 and then again on /dev/nvme1n1.
10.2.4. Add The Devices to lv_vars.yaml
As
root, navigate to the/usr/share/ceph-ansible/directory:cd /usr/share/ceph-ansible
# cd /usr/share/ceph-ansibleCopy to Clipboard Copied! Toggle word wrap Toggle overflow As
root, copy thelv_vars.yamlAnsible playbook to the current directory:cp infrastructure-playbooks/vars/lv_vars.yaml .
# cp infrastructure-playbooks/vars/lv_vars.yaml .Copy to Clipboard Copied! Toggle word wrap Toggle overflow For the first run edit the file so it includes the following lines:
nvme_device: /dev/nvme0n1 hdd_devices: - /dev/sdc - /dev/sdd
nvme_device: /dev/nvme0n1 hdd_devices: - /dev/sdc - /dev/sddCopy to Clipboard Copied! Toggle word wrap Toggle overflow
The journal size, number of bucket indexes, their sizes and names, and the bucket indexes' journal names can all be adjusted in lv_vars.yaml. See the comments within the file for more information.
10.2.5. Run The lv-create.yml Ansible Playbook
The purpose of the lv-create.yml playbook is to create logical volumes for the object gateway bucket index, and journals, on a single NVMe. It does this by using osd_scenario=lvm as opposed to using osd_scenario=non-collocated. The lv-create.yml Ansible playbook makes it easier to configure Ceph in this way by automating some of the complex LVM creation and configuration.
As
root, copy thelv-create.ymlAnsible playbook to the current directory:cp infrastructure-playbooks/lv-create.yml .
# cp infrastructure-playbooks/lv-create.yml .Copy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure the storage devices are raw
Before running
lv-create.ymlto create the logical volumes on the NVMe devices and HDD devices, ensure there are no file system, GPT, RAID, or other signatures on them.If they are not raw, when you run
lv-create.ymlit may fail with the following error:device /dev/sdc excluded by a filter
device /dev/sdc excluded by a filterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Wipe storage device signatures (optional)
If the devices have signatures you can use
wipefsto erase them.An example of using
wipefsto erase the devices is shown below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
lv-teardown.ymlAnsible playbook:Always run
lv-teardown.ymlbefore runninglv-create.yml:As
root, copy thelv-teardown.ymlAnsible playbook to the current directory:cp infrastructure-playbooks/lv-teardown.yml .
# cp infrastructure-playbooks/lv-teardown.yml .Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
lv-teardown.ymlAnsible playbook:ansible-playbook lv-teardown.yml -i hosts
$ ansible-playbook lv-teardown.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow WarningProceed with caution when running the
lv-teardown.ymlAnsible script. It destroys data. Ensure you have backups of any important data.Run the
lv-create.ymlAnsible playbook:ansible-playbook lv-create.yml -i hosts
$ ansible-playbook lv-create.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow
10.2.6. Copy First NVMe LVM Configuration
Review
lv-created.logOnce the
lv-create.ymlAnsible playbook completes successfully, configuration information will be written tolv-created.log. Openlv-created.logand look for information similar to the below example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Copy this information into
group_vars/osds.ymlunderlvm_volumes:.
10.2.7. Run The lv-create.yml Playbook on NVMe device two
The following instructions are abbreviated steps to set up a second NVMe device. Consult the related steps above for further context if needed.
Modify
lv-vars.yamlto use the second NVMe and associated HDDs.Following the previous example,
lv-vars.yamlwill now have the following devices set:nvme_device: /dev/nvme1n1 hdd_devices: - /dev/sde - /dev/sdf
nvme_device: /dev/nvme1n1 hdd_devices: - /dev/sde - /dev/sdfCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run
lv-teardown.yml:ansible-playbook lv-teardown.yml -i hosts
$ ansible-playbook lv-teardown.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run
lv-create.ymlagainansible-playbook lv-create.yml -i hosts
$ ansible-playbook lv-create.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow
10.2.8. Copy Second NVMe LVM Configuration
Review
lv-created.logOnce the
lv-create.ymlAnsible playbook completes successfully, configuration information will be written tolv-created.log. Openlv-created.logand look for information similar to the below example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Copy this information into
group_vars/osds.ymlunder the already entered information underlvm_volumes:.
10.2.9. Verify LVM Configuration
Review LVM Configuration
Based on the example of two NVMe device and four HDDs the following Logical Volumes (LVs) should be created:
One journal LV per HDD placed on both NVMe devices (two LVs on /dev/nvme0n1, two on /dev/nvme1n1)
One data LV per HDD placed on each HDD (one LV per HDD)
One journal LV per bucket index placed on NVMe (one LV on /dev/nvme0n1, one LV on /dev/nvme1n1)
One data LV per bucket index placed on both NVMe devices (one LV on /dev/nvme0n1, one LV on /dev/nvme1n1)
The LVs can be seen in
lsblkandlvscanoutput. In the example explained above, there should be twelve LVs for Ceph. As a rough sanity check you could count the Ceph LVs to make sure there are at least twelve, but ideally you would make sure the appropriate LVs were created on the right storage devices (NVMe vs HDD).Example output from
lsblkis shown below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output from
lvscanis shown below:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.2.10. Edit The osds.yml and all.yml Ansible Playbooks
Set
osd_objectstoretobluestoreIn addition to adding the second set of information from
lv-create.logintoosds.yml,osd_objectstorealso needs to be set tobluestorein both theosds.ymlandall.ymlfiles.The line should look like this in both
osds.ymlandall.yml:osd_objectstore: bluestore
osd_objectstore: bluestoreCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set
osd_scenariotolvminosds.ymlThe
osds.ymlfile should look similar to the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.2.11. Install Ceph for NVMe and Verify Success
Run the
site.ymlAnsible playbook to install Cephansible-playbook -v site.yml -i hosts
$ ansible-playbook -v site.yml -i hostsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify Ceph is running properly after install completes
ceph -s
# ceph -sCopy to Clipboard Copied! Toggle word wrap Toggle overflow ceph osd tree
# ceph osd treeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example
ceph -soutput showing Ceph is running properly:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
ceph osd treeoutput showing Ceph is running properly:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Ceph is now set up to use two NVMe devices and LVM optimally for Object Storage Gateway.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}