SupportConsoleDevelopersStart a trialContact

Chapter 10. Configuring SBD fencing in a high-availability cluster

Storage-based death (SBD) is a fencing mechanism that provides an alternative to traditional “power fencing” agents. SBD uses a watchdog timer, and optionally shared storage to make nodes self-fence if they become unresponsive or lose quorum.

You can use this approach in environments where network-based power management is unavailable, or where you want to ensure a node reboots if it loses access to the shared storage fabric.

10.1. Overview of SBD fencing
Copy link

Storage-based death (SBD) fencing provides a mechanism for nodes to self-fence in the event certain health checks fail or can’t be performed in time or a reboot is requested by disk-based messaging. You can implement SBD in one of two ways, depending on your hardware and storage availability.

Expand
Table 10.1. Comparison of watchdog-only and poison-pill fencing
FeatureWatchdog-only fencingPoison-pill fencing

Mechanism

Relies on health and quorum monitoring. If a node loses quorum or cannot verify its own health, it stops resetting the watchdog timer. The cluster assumes fencing is successful only after the stonith-watchdog-timeout expires.

Uses shared storage to send explicit "poison pill" messages by the fence_sbd agent. If the target node is responsive, it reads the message and reboots immediately. If the node is unresponsive or storage is inaccessible, it relies on the watchdog timer to trigger the reboot.

Storage requirement

None.

Requires 1 to 3 shared block devices (minimum 4 MB each). Devices must be accessible by all nodes and cannot host a filesystem, be managed by LVM, or be part of a RAID/mirroring layer managed by the cluster.

Reliability

Reliability is derived from a watchdog device (hardware WDT or softdog) supervising the software timer. Hardware WDTs are preferred; softdog (kernel-space) is a supported fallback but might fail if the kernel completely hangs or is starved of resources.

Reliability is derived from the same watchdog device mechanism. Although disks facilitate communication, the watchdog device is the responsible for guarantying a reboot. Because Pacemaker awareness keeps a cluster accessible when disks are not available, even a single shared disk does not represent a single point of failure (SPOF). However, use three disks if you want to be able to fence when a storage device is unavailable.

Performance

Can be configured for faster recovery by reducing network latency blips. However, the cluster must wait for the full stonith-watchdog-timeout to expire before recovering resources.

Potentially provides faster reboot because a responsive target node might react to the fence message immediately. However, it might require higher storage-layer timeouts, such as 30s, to account for storage maintenance or firmware updates. If a node comes back quickly resource recovery time might be short otherwise resource recovery is held back by these storage timeout requirements.

Common use case

Environments without shared storage. Preferred for clusters with at least 3 quorum voters (or 2 nodes and a QDevice).

Primarily used in 2-node clusters to bypass the 3-vote quorum limitation of watchdog-only setups. Also useful for multi-site clusters where the network link between members might be severed but storage remains accessible.

10.2. Configuring watchdog-only SBD fencing
Copy link

In a watchdog-only configuration nodes monitor quorum state and their health as seen by the cluster. If a node is unable to retrieve the cluster view (CIB) or if the cluster determines the node is in an unhealthy state, the local watchdog timer on that node to expire, triggering a reboot.

Prerequisites

  • A running high-availability cluster.
  • A functional watchdog timer (WDT) device available on all nodes at /dev/watchdog or another device path configured on a per node basis.
  • No other fencing devices are active, or if other fencing devices are available in the cluster watchdog-fencing can be limited to those nodes that can’t be fenced by those or a fencing topology can be configured to have watchdog-fencing as fallback
Note

SBD implementation is modular and can be deployed selectively within a cluster. In heterogeneous environments, such as those containing pacemaker-remote nodes or hardware lacking functional watchdog devices, SBD remains a viable fencing solution for compatible members. The absence of universal SBD support does not preclude its selective implementation on supported nodes.

Procedure

  1. On all nodes in the cluster, install the sbd package: 

    # dnf install sbd

  2. On each node, list the available watchdog devices to identify the proper device path: 

    # pcs stonith sbd watchdog list

  3. On each node, verify that the watchdog device functions correctly by triggering a test:

    Warning

    This command reboots the system to confirm the watchdog can successfully trigger a hardware reset. Perform this test only before enabling the SBD daemon; the command does not work after the daemon has taken control of the watchdog device. 

    # pcs stonith sbd watchdog test

  4. On one node, enable SBD fencing to update /etc/sysconfig/sbd on all nodes and enables the sbd service: 

    # pcs stonith sbd enable sbd_watchdog_timeout=5

  5. Set the cluster stonith-watchdog-timeout property. This value determines how long the remaining cluster nodes do wait before assuming a missing node has successfully self-fenced: 

    # pcs property set stonith-watchdog-timeout=10
    Important

    Each node verifies this property individually before bringing up resources. To prevent split-brain scenarios, stonith-watchdog-timeout must be at least as large as the value of SBD_WATCHDOG_TIMEOUT. Do not set this property to -1 for automated calculation, as inconsistent SBD_WATCHDOG_TIMEOUT values across nodes would then lead to different calculations of stonith-watchdog-timeout (a value that needs to be consistent for both the fencing originator and the target - lower for originator than target leads to split-brain) for different nodes.

  6. Restart the cluster to initialize the SBD processes: 

    # pcs cluster stop --all
# pcs cluster start --all

Verification

  1. Verify that the SBD status is active: 

    # pcs stonith sbd status

  2. Manually trigger a fence test to ensure the configuration works:

    Warning

    This command reboots the target node immediately. 

    # pcs stonith fence node_name

In heterogeneous clusters where some nodes lack a functional hardware watchdog, you can create a visible fencing resource to ensure watchdog-only fencing is applied only to supported nodes.

Prerequisites

  • SBD is enabled in watchdog-only mode on the supported nodes.
  • A functional hardware watchdog device is available on the target nodes.

Procedure

  1. Create the watchdog fencing resource. Use the pcmk_host_list attribute to specify only the nodes that possess a functional watchdog device: 

    # pcs stonith create watchdog_fencing fence_watchdog pcmk_host_list="node-01,node-02"
  2. If the cluster contains nodes that are not in the pcmk_host_list, configure an alternative fencing method (such as IPMI or a power-based agent) for those specific nodes.

  3. Configure fencing levels to use the watchdog as a fallback. Set the primary fencing agent at level 1 and the watchdog agent at level 2. 

    # pcs stonith level add 1 node-01 ipmi_fencing
# pcs stonith level add 2 node-01 watchdog_fencing

Verification

  1. Confirm that the resource is restricted to the specified hosts: 

    # pcs stonith config watchdog_fencing

  2. Verify that the cluster recognizes the fencing topology: 

    # pcs stonith level

When using a 2-node cluster, adding a quorum device (corosync-qdevice) provides an additional vote to quorum, making its behavior closer to odd-numbered member clusters. This allows the cluster to bypass the standard 3-node minimum quorum vote limitation for watchdog-only SBD fencing.

When combining a QDevice with watchdog-only SBD fencing, you must strictly align your timeout values, otherwise SBD prevents startup.

Procedure

  1. Determine the required timeout values for your cluster:

    • qdevice-sync_timeout: The default value is 30s.
    • SBD_WATCHDOG_TIMEOUT: If this value is less than the qdevice-sync_timeout, a quorum state update can be delayed long enough to cause a split-brain situation.
    • stonith-watchdog-timeout: This value must be higher than the SBD_WATCHDOG_TIMEOUT to ensure the cluster waits long enough for a node to self-fence. Set this to double the value of the SBD_WATCHDOG_TIMEOUT. Therefore, if SBD_WATCHDOG_TIMEOUT is 30, this should be 60.

  2. Enable SBD and configure the watchdog timeout. By not specifying any block devices, SBD will automatically defaults to watchdog-only fencing: 

    # pcs stonith sbd enable SBD_WATCHDOG_TIMEOUT=30

  3. Configure the stonith-watchdog-timeout cluster property to enable the implicit, "hidden" watchdog STONITH device: 

    # pcs property set stonith-watchdog-timeout=60

  4. Optional: Create a visible watchdog fencing device. Incorporate this device into a fencing topology, such as configuring it as a fallback for other fencing devices that have failed, or limiting it to a certain list of nodes. Configure the device using the fence_watchdog agent: 

    # pcs stonith create sbd_watchdog fence_watchdog

10.5. Configuring poison-pill SBD fencing
Copy link

Poison-pill fencing uses shared block devices to communicate fencing instructions. When a node needs to be fenced, the cluster writes a message to the shared device. The target node reads this message and reboots itself.

Prerequisites

  • A running high-availability cluster.
  • A functional watchdog timer (WDT) device available on all nodes at /dev/watchdog.
  • One to three shared block devices (typically 1-4mb in size) accessible by all nodes.
  • The shared devices must not contain a filesystem or be managed by LVM.

Procedure

  1. On all nodes in the cluster, install the SBD and fence agent packages: 

    # dnf install sbd fence-agents-sbd

  2. On each node, verify that the watchdog device functions correctly by triggering a test:

    Warning

    This command reboots the system to confirm the watchdog can successfully trigger a hardware reset. Perform this test only before enabling the SBD daemon; the command does not work after the daemon takes control of the watchdog device. 

    # pcs stonith sbd watchdog test

  3. Initialize the shared SBD block device. Replace /dev/sdb with the path to your shared device:

    Note

    If you are using multiple shared block devices for redundancy, repeat the device= parameter for each device (for example, device=/dev/sdb device=/dev/sdc). 

    # pcs stonith sbd device setup device=/dev/sdb

  4. Enable SBD with the shared device: 

    # pcs stonith sbd enable device=/dev/sdb sbd_watchdog_timeout=5

  5. Create the fencing device in the cluster. This example uses fence_sbd_storage as the resource name: 

    # pcs stonith create fence_sbd_storage fence_sbd devices=/dev/sdb

  6. Adjust the pcmk_reboot_timeout to ensure the cluster waits long enough for the disk message to be processed. This value should be at least 5 seconds higher than the msgwait value stored on the disk: 

    # pcs stonith update sbd_fence pcmk_reboot_timeout=15

Verification

  1. Verify that the service is enabled and running, the nodes are correctly registered on the SBD device and check their slot status: 

    # pcs sbd status --full

  2. Check the stonith-watchdog-timeout cluster property: 

    # pcs property show stonith-watchdog-timeout

    For a standard poison-pill deployment, this value should be 0 (disabled) to prevent the cluster from prematurely assuming a node has self-fenced before the poison-pill message is acknowledged. In advanced setups where this property is non-zero, ensure that all targets used for poison-pill fencing are explicitly excluded from watchdog-only fencing to avoid split-brain risks.

  3. Manually trigger a fence test to ensure the configuration works:

    Warning

    This command reboots the target node immediately. 

    # pcs stonith fence node_name

When deploying a Red Hat High Availability cluster using poison-pill fencing with Storage-based Death (SBD), configure timing carefully. Proper timeouts ensure that nodes are safely fenced during failures, preventing split-brain conditions while allowing cluster services to stabilize.

Here are the primary considerations for timing configurations in a poison-pill fencing scenario:

SBD_WATCHDOG_TIMEOUT

Keep this at the default of 5 seconds. A higher value gives the cluster more time to resolve temporary connectivity blips without reacting harshly, but it increases the time it takes to recover from legitimate failures. Setting it below the default 5 seconds is cautioned against, as it can make the cluster overly aggressive and lead to instability during high load or brief unresponsiveness.

If your cluster uses a QDevice, set the SBD_WATCHDOG_TIMEOUT parameter higher than qdevice-sync_timeout (which defaults to 30s). Set a difference of 3 to 5 seconds to prevent delayed quorum updates from causing a split-brain situation.

msgwait timeout

The duration that fence_sbd waits after writing the poison-pill message before the fencing action is considered complete. Set this to twice the value of SBD_WATCHDOG_TIMEOUT, for example, 10 seconds, if the watchdog timeout is 5 seconds. If msgwait is set lower than the watchdog timeout, the cluster can prematurely declare the fencing operation successful while the target node is still processing the action or waiting for its hardware watchdog to expire, resulting in a split-brain scenario.

The msgwait duration must be appropriately sized to securely cover three distinct scenarios:

  1. The target successfully reads the message and self-fences: The intended scenario where the target node is still responsive, successfully reads the poison-pill message from the shared block device, and initiates a self-fence.
  2. The target loses access to the disks: The target node is still running but finds that it is not capable of reading from a quorate number of shared disks, causing it to recognize this loss of access and self-fence.

  3. The target is completely unresponsive: The target node is frozen and incapable of doing anything. Because it cannot run actions, it won’t be able to reset the (hardware)-watchdog, which in turn leads to a reboot as well.

    If msgwait is set lower than the timeout derived from preceding considerations, the cluster can prematurely declare the fencing operation successful while the target node is still processing the action or waiting for its hardware watchdog to expire, allowing a split-brain scenario.

pcmk_reboot_timeout (STONITH-level timeout)

This is awaited by the cluster during timeout that dictates how long the STONITH device itself (fence_sbd) awaits during a "reboot" action.

This value should be raised so that it is at least 5 seconds higher than the msgwait attribute. If it is not raised accordingly, the cluster might give up on the fencing operation and report a failure before the msgwait period has been fully exhausted.

SBD_START_DELAY and startup delays

Setting a delay on startup gives the cluster extra time to stabilize and allows storage devices to fully come online before monitoring begins. Furthermore, there is a generic issue with cluster nodes after a quick recovery after fencing that they might still catch the broadcast about their own successful fence. This confuses them and brings them to a permanent error condition under the assumption something had gone wrong within the fencing sublayer.

  • With SBD: An increase of SBD_START_DELAY can be used to mitigate this behavior, purposefully slowing the node’s re-entry so the old broadcast clears.
  • Without SBD: For environments not using SBD, there is the hint to delay the corosync startup directly in the unit-file to achieve the same result.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top