Chapter 9. Troubleshooting clusters in stretch mode


You can replace and remove the failed tiebreaker monitors. You can also force the cluster into the recovery or healthy mode if needed.

When in stretch degraded mode, the cluster goes into the recovery mode automatically after the disconnected data center comes back. If that does not happen, or you want to enable recovery mode early, you can force the stretch cluster into the recovery mode.

Prerequisites

  • A running Red Hat Ceph Storage cluster
  • Stretch mode in enabled on a cluster

Procedure

  1. Force the stretch cluster into the recovery mode:

    Example

    [ceph: root@host01 /]#  ceph osd force_recovery_stretch_mode --yes-i-really-mean-it
    Copy to Clipboard Toggle word wrap

    Note

    The recovery state puts the cluster in the HEALTH_WARN state.

  2. When in recovery mode, the cluster should go back into normal stretch mode after the placement groups are healthy. If that does not happen, you can force the stretch cluster into the healthy mode:

    Example

    [ceph: root@host01 /]#  ceph osd force_healthy_stretch_mode --yes-i-really-mean-it
    Copy to Clipboard Toggle word wrap

    Note

    You can also run this command if you want to force the cross-data-center peering early and you are willing to risk data downtime, or you have verified separately that all the placement groups can peer, even if they are not fully recovered.

    You might also wish to invoke the healthy mode to remove the HEALTH_WARN state, which is generated by the recovery state.

    Note

    The force_recovery_stretch_mode and force_recovery_healthy_mode commands should not be necessary, as they are included in the process of managing unanticipated situations.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat