Chapter 10. Troubleshooting scrub and deep-scrub issues

Prequisites

1. A running Red Hat Ceph Storage cluster in a healthy state.
2. Root-level access to the node.

Procedure

1. Detect low measured IOPS reported by OSD bench during OSD boot-up and fallback to default IOPS setting defined for osd_mclock_max_capacity_iops_[hdd|ssd]. The fallback is triggered if the reported IOPS falls below a threshold determined by osd_mclock_iops_capacity_low_threshold_[hdd|ssd]. A cluster warning is also logged.

   Example:

   $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]

   Copy to Clipboard Toggle word wrap

2. [Optional]: Perform the following steps if you have not yet upgraded to 7 from 6 (before the upgrade):

   1. For clusters already affected by the issue, remove the IOPS capacity setting on the OSD(s) before upgrading to the release with the fix by running the following command:

      Example:

      $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]

      Copy to Clipboard Toggle word wrap

   2. Set the osd_mclock_force_run_benchmark_on_init option for the affected OSD to true before the upgrade:

      Example:

      $ ceph config set osd.X osd_mclock_force_run_benchmark_on_init true

      Copy to Clipboard Toggle word wrap

      After upgrading to the release with this fix, the IOPS capacity reflects the default setting or the new one reported by the OSD bench.

3. [Optional]: Perform the following steps if you have already upgraded from 6 to 7 (after upgrade):

   1. If you were unable to perform the above steps before upgrade, you re-run the OSD bench again after upgrading by removing the osd_mclock_max_capacity_iops_[hdd|ssd] setting:

      Example:

      $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]

      Copy to Clipboard Toggle word wrap

   2. Set osd_mclock_force_run_benchmark_on_init to true.

      Example:

      $ ceph config set osd.X osd_mclock_force_run_benchmark_on_init true

      Copy to Clipboard Toggle word wrap

   3. Restart the OSD.

      After the OSD restarts, the IOPS capacity reflects the default setting or the new setting reported by the OSD bench.