Chapter 9. BlueStore

Procedure

1. Add a new OSD node to the [osds] section in Ansible inventory file, by default located at /etc/ansible/hosts.

   [osds]
   node1
   node2
   node3
   HOST_NAME

   Replace:

   • HOST_NAME with the name of the OSD node

   Example

   [osds]
   node1
   node2
   node3
   node4

2. Navigate to the /usr/share/ceph-ansible/ directory.

   [user@admin ~]$ cd /usr/share/ceph-ansible

3. Create the host_vars directory.

   [root@admin ceph-ansible] mkdir host_vars

4. Create the configuration file for the newly added OSD in host_vars.

   [root@admin ceph-ansible] touch host_vars/HOST_NAME.yml

   Replace:

   • HOST_NAME with the host name of the newly added OSD

   Example

   [root@admin ceph-ansible] touch host_vars/node4.yml

5. Add the following setting to the newly created file:

   osd_objectstore: bluestore

   Note

   To use BlueStore for all OSDs, add osd_objectstore:bluestore to the group_vars/all.yml file.

6. Configure the BlueStore OSDs, in host_vars/HOST_NAME.yml:

   Syntax

   lvm_volumes:
     - data: DATALV
       data_vg: DATAVG

   Replace:

   • DATALV with the data logical volume name
   • DATAVG with the data logical volume group name

   Example

   lvm_volumes:
     - data: data-lv1
       data_vg: vg1

7. Optional. If you want to store the block.wal and block.db on dedicated logical volumes, edit the host_vars/HOST_NAME.yml file as follows:

   lvm_volumes:
     - data: DATALV
       wal: WALLV
       wal_vg: VG
       db: DBLV
       db_vg: VG

   Replace:

   • DATALV with the logical volume where the data should be contained
   • WALLV with the logical volume where the write-ahead-log should be contained
   • VG with the volume group the WAL and/or DB device LVs are on
   • DBLV with the logical volume the BlueStore internal metadata should be contained

   Example

   lvm_volumes:
     - data: data-lv3
       wal: wal-lv1
       wal_vg: vg3
       db: db-lv3
       db_vg: vg3

   Note

   When using lvm_volumes: with osd_objectstore: bluestore the lvm_volumes YAML dictionary must contain at least data. When defining wal or db, it must have both the LV name and VG name (db and wal are not required). This allows for four combinations: just data, data and wal, data and wal and db, or data and db. Data can be a raw device, lv or partition. The wal and db can be a lv or partition. When specifying a raw device or partition ceph-volume will put logical volumes on top of them.

   Note

   Currently, ceph-ansible does not create the volume groups or the logical volumes. This must be done before running the Anisble playbook.

8. Optional: You can override the block.db default size in the group_vars/all.yml file:

   Syntax

   ceph_conf_overrides:
     osd:
       bluestore_block_db_size: VALUE

   Example

   ceph_conf_overrides:
     osd:
       bluestore_block_db_size: 24336000000

   Note

   The value of bluestore_block_db_size should be greater than 2 GB.

9. Open and edit the group_vars/all.yml file, and uncomment the osd_memory_target option. Adjust the value on how much memory you want the OSD to consume.

   Note

   The default value for the osd_memory_target option is 4000000000, which is 4 GB. This option pins the BlueStore cache in memory.

   Important

   The osd_memory_target option only applies to BlueStore-backed OSDs.

10. Run the following Ansible playbook:

    [user@admin ceph-ansible]$ ansible-playbook site.yml

11. From a Ceph Monitor node, verify that the new OSD has been successfully added:

    [root@mon ~]# ceph osd tree

Procedure

1. Optional: If redeploying an existing OSD node, use the shrink-osd.yml Ansible playbook to remove the OSD from the cluster.

   ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=OSD_ID

   Example

   [admin@admin ceph-ansible]$ ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=1

2. If redeploying an existing OSD node, wipe the OSD drives and reinstall the OS.
3. Prepare the node for OSD provisioning using Ansible. Examples of preparation tasks include enabling Red Hat Ceph Storage repositories, adding an Ansible user, and enabling password-less SSH login.

4. Add the bluestore_min_alloc_size to the ceph_conf_overrides section of the group_vars/all.yml Ansible playbook:

   ceph_conf_overrides:
     osd:
       bluestore_min_alloc_size: 4096

5. If deploying a new node, add it to the Ansible inventory file, normally /etc/ansible/hosts:

   [osds]
   OSD_NODE_NAME

   Example

   [osds]
   osd1 devices="[ '/dev/sdb' ]"

6. If redeploying an existing OSD, copy the admin keyring file in the Ceph Monitor node to the node where you want to deploy the OSD.

7. Provision the OSD node using Ansible:

   ansible-playbook -v site.yml -l OSD_NODE_NAME

   Example

   [admin@admin ceph-ansible]$ ansible-playbook -v site.yml -l osd1

8. After the playbook finishes, verify the setting using the ceph daemon command:

   ceph daemon OSD.ID config get bluestore_min_alloc_size

   Example

   [root@osd1 ~]# ceph daemon osd.1 config get bluestore_min_alloc_size
   {
       "bluestore_min_alloc_size": "4096"
   }

   You can see bluestore_min_alloc_size is set to 4096 bytes, which is equivalent to 4 KiB.