Este conteúdo não está disponível no idioma selecionado.

Chapter 10. Troubleshooting scrub and deep-scrub issues


Learn to troubleshoot the scrub and deep-scrub issues.

10.1. Addressing the scrub slowness issue while upgrading from 6 to 7

Learn to troubleshoot the scrub slowness issue which seen after upgrading from 6 to 7.

Scrub slowness is caused by the automated OSD benchmark setting a very low value for osd_mclock_max_capacity_iops_hdd. Due to this, scrub operations are impacted since the IOPS capacity of an OSD plays a significant role in determining the bandwidth the scrub operation receives. To further increase the problem, scrubs receive only a fraction of the total IOPS capacity based on the QoS allocation defined by the mClock profile.

Due to this, the Ceph cluster reports the expected scrub completion time in multiples of days or weeks.

Prequisites

  1. A running Red Hat Ceph Storage cluster in a healthy state.
  2. Root-level access to the node.

Procedure

  1. Detect low measured IOPS reported by OSD bench during OSD boot-up and fallback to default IOPS setting defined for osd_mclock_max_capacity_iops_[hdd|ssd]. The fallback is triggered if the reported IOPS falls below a threshold determined by osd_mclock_iops_capacity_low_threshold_[hdd|ssd]. A cluster warning is also logged.

    Example:

    $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]
    Copy to Clipboard Toggle word wrap

  2. [Optional]: Perform the following steps if you have not yet upgraded to 7 from 6 (before the upgrade):

    1. For clusters already affected by the issue, remove the IOPS capacity setting on the OSD(s) before upgrading to the release with the fix by running the following command:

      Example:

      $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]
      Copy to Clipboard Toggle word wrap

    2. Set the osd_mclock_force_run_benchmark_on_init option for the affected OSD to true before the upgrade:

      Example:

      $ ceph config set osd.X osd_mclock_force_run_benchmark_on_init true
      Copy to Clipboard Toggle word wrap

      After upgrading to the release with this fix, the IOPS capacity reflects the default setting or the new one reported by the OSD bench.

  3. [Optional]: Perform the following steps if you have already upgraded from 6 to 7 (after upgrade):

    1. If you were unable to perform the above steps before upgrade, you re-run the OSD bench again after upgrading by removing the osd_mclock_max_capacity_iops_[hdd|ssd] setting:

      Example:

      $ ceph config rm osd.X osd_mclock_max_capacity_iops_[hdd|ssd]
      Copy to Clipboard Toggle word wrap

    2. Set osd_mclock_force_run_benchmark_on_init to true.

      Example:

      $ ceph config set osd.X osd_mclock_force_run_benchmark_on_init true
      Copy to Clipboard Toggle word wrap

    3. Restart the OSD.

      After the OSD restarts, the IOPS capacity reflects the default setting or the new setting reported by the OSD bench.

Voltar ao topo
Red Hat logoGithubredditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar. Explore nossas atualizações recentes.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja o Blog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

Theme

© 2025 Red Hat