Chapter 10. The mClock OSD scheduler

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the osd_mclock_profile option:

   Syntax

   ceph config set osd.OSD_ID osd_mclock_profile VALUE

   Example

   [ceph: root@host01 /]# ceph config set osd.0 osd_mclock_profile high_recovery_ops

   This example changes the profile to allow faster recoveries on osd.0.

   Note

   For optimal performance the profile must be set on all Ceph OSDs by using the following command:

   Syntax

   ceph config set osd osd_mclock_profile VALUE

Switch from built-in profile to custom profile

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Switch to the custom profile:

   Syntax

   ceph config set osd.OSD_ID osd_mclock_profile custom

   Example

   [ceph: root@host01 /]# ceph config set osd.0 osd_mclock_profile custom

   Note

   For optimal performance the profile must be set on all Ceph OSDs by using the following command:

   Example

   [ceph: root@host01 /]# ceph config set osd osd_mclock_profile custom

3. Optional: After switching to the custom profile, modify the desired mClock configuration options:

   Syntax

   ceph config set osd.OSD_ID MCLOCK_CONFIGURATION_OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set osd.0 osd_mclock_scheduler_client_res 0.5

   This example changes the client reservation IOPS ratio for a specific OSD osd.0 to 0.5 (50%)

   Important

   Change the reservations of other services, such as background recovery and background best-effort accordingly to ensure that the sum of the reservations does not exceed the maximum proportion (1.0) of the IOPS capacity of the OSD.

Switch from custom profile to built-in profile

1. Log into the cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the desired built-in profile:

   Syntax

   ceph config set osd osd_mclock_profile MCLOCK_PROFILE

   Example

   [ceph: root@host01 /]# ceph config set osd osd_mclock_profile high_client_ops

   This example sets the built-in profile to high_client_ops on all Ceph OSDs.

3. Determine the existing custom mClock configuration settings in the database:

   Example

   [ceph: root@host01 /]# ceph config dump

4. Remove the custom mClock configuration settings determined earlier:

   Syntax

   ceph config rm osd MCLOCK_CONFIGURATION_OPTION

   Example

   [ceph: root@host01 /]# ceph config rm osd osd_mclock_scheduler_client_res

   This example removes the configuration option osd_mclock_scheduler_client_res that was set on all Ceph OSDs.

   After all existing custom mClock configuration settings are removed from the central configuration database, the configuration settings related to high_client_ops are applied.

5. Verify the settings on Ceph OSDs:

   Syntax

   ceph config show osd.OSD_ID

   Example

   [ceph: root@host01 /]# ceph config show osd.0

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Run the following command to override the mClock settings:

   Syntax

   ceph tell osd.OSD_ID injectargs '--MCLOCK_CONFIGURATION_OPTION=VALUE'

   Example

   [ceph: root@host01 /]# ceph tell osd.0 injectargs '--osd_mclock_profile=high_recovery_ops'

   This example overrides the osd_mclock_profile option on osd.0.

3. Optional: You can use the alternative to the previous ceph tell osd.OSD_ID injectargs command:

   Syntax

   ceph daemon osd.OSD_ID config set MCLOCK_CONFIGURATION_OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph daemon osd.0 config set osd_mclock_profile high_recovery_ops

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the osd_mclock_override_recovery_settings configuration option to true on all Ceph OSDs:

   Example

   [ceph: root@host01 /]# ceph config set osd osd_mclock_override_recovery_settings true

3. Set the desired backfills or recovery option:

   Syntax

   ceph config set osd OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set osd osd_max_backfills_ 5

4. Wait a few seconds and verify the configuration for the specific OSD:

   Syntax

   ceph config show osd.OSD_ID_ | grep OPTION

   Example

   [ceph: root@host01 /]# ceph config show osd.0 | grep osd_max_backfills

5. Reset the osd_mclock_override_recovery_settings configuration option to false on all OSDs:

   Example

   [ceph: root@host01 /]# ceph config set osd osd_mclock_override_recovery_settings false

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Verify the capacity of a Ceph OSD:

   Syntax

   ceph config show osd.OSD_ID osd_mclock_max_capacity_iops_[hdd, ssd]

   Example

   [ceph: root@host01 /]# ceph config show osd.0 osd_mclock_max_capacity_iops_ssd
   
   21500.000000

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Benchmark a Ceph OSD:

   Syntax

   ceph tell osd.OSD_ID bench [TOTAL_BYTES] [BYTES_PER_WRITE] [OBJ_SIZE] [NUM_OBJS]

   where:

   • TOTAL_BYTES: Total number of bytes to write.
   • BYTES_PER_WRITE: Block size per write.
   • OBJ_SIZE: Bytes per object.
   • NUM_OBJS: Number of objects to write.

   Example

   [ceph: root@host01 /]# ceph tell osd.0 bench 12288000 4096 4194304 100
   {
       "bytes_written": 12288000,
       "blocksize": 4096,
       "elapsed_sec": 1.3718913019999999,
       "bytes_per_sec": 8956977.8466311768,
       "iops": 2186.7621695876896
   }

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Run a simple 4KiB random write workload on an OSD:

   Syntax

   ceph tell osd.OSD_ID bench 12288000 4096 4194304 100

   Example

   [ceph: root@host01 /]# ceph tell osd.0 bench 12288000 4096 4194304 100
   {
       "bytes_written": 12288000,
       "blocksize": 4096,
       "elapsed_sec": 1.3718913019999999,
       "bytes_per_sec": 8956977.8466311768,
       "iops": 2186.7621695876896 1
   }

   1

   The overall throughput obtained from the output of the osd bench command. This value is the baseline throughput, when the default BlueStore throttle options are in effect.

3. Note the overall throughput, that is IOPS, obtained from the output of the previous command.

4. If the intent is to determine the BlueStore throttle values for your environment, set bluestore_throttle_bytes and bluestore_throttle_deferred_bytes options to 32 KiB, that is, 32768 Bytes:

   Syntax

   ceph config set osd.OSD_ID bluestore_throttle_bytes 32768
   ceph config set osd.OSD_ID bluestore_throttle_deferred_bytes 32768

   Example

   [ceph: root@host01 /]# ceph config set osd.0 bluestore_throttle_bytes 32768
   [ceph: root@host01 /]# ceph config set osd.0 bluestore_throttle_deferred_bytes 32768

   Otherwise, you can skip to the next section Specifying maximum OSD capacity.

5. Run the 4KiB random write test as before using an OSD bench command:

   Example

   [ceph: root@host01 /]# ceph tell osd.0 bench 12288000 4096 4194304 100

6. Notice the overall throughput from the output and compare the value against the baseline throughput recorded earlier.
7. If the throughput does not match with the baseline, increase the BlueStore throttle options by multiplying by 2.
8. Repeat the steps by running the 4KiB random write test, comparing the value against the baseline throughput, and increasing the BlueStore throttle options by multiplying by 2, until the obtained throughput is very close to the baseline value.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set osd_mclock_max_capacity_iops_[hdd, ssd] option for an OSD:

   Syntax

   ceph config set osd.OSD_ID osd_mclock_max_capacity_iops_[hdd,ssd] VALUE

   Example

   [ceph: root@host01 /]# ceph config set osd.0 osd_mclock_max_capacity_iops_hdd 350

   This example sets the maximum capacity for osd.0, where an underlying device type is HDD, to 350 IOPS.