SBD and watchdog fencing - when the node fences itself · Linux

Every fence method covered so far requires the survivor to reach something: a BMC, a hypervisor API, a PDU. SBD inverts the problem. The node fences itself, in hardware, and the survivor only has to wait.

That inversion is why SBD is the answer to two problems the other methods cannot solve: a stretched cluster where the fence path crosses the link that failed, and a node whose kernel has stalled so completely that nothing on it can be trusted.

The watchdog is the mechanism

A hardware watchdog is a timer implemented in the chipset or in the BMC, outside the operating system. Once armed, software must reset it - “pet” it - before it expires. If the timer runs out, the hardware resets the machine. No kernel involvement, no cooperation from the OS, no opportunity for a stalled system to decline.

SBD is a daemon that pets the watchdog only while it is satisfied that this node is entitled to run. The moment it is not satisfied, it stops petting, and the hardware takes the node down within the watchdog timeout.

This is the property that matters: the enforcement lives below the layer that failed. A node whose kernel is deadlocked cannot answer a fence request, cannot shut itself down, and cannot stop its queued I/O - but it also cannot pet the watchdog, so it gets reset anyway.

Read-only / Safewhat device is armed, and by what driver

# wdctl

Device:        /dev/watchdog0
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Pre-timeout:    0 seconds
Timeleft:      27 seconds
FLAG           DESCRIPTION                    STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply               1           0
MAGICCLOSE     Supports magic close char           0           0
SETTIMEOUT     Set timeout (in seconds)            0           0

Illustrative output

An Identity naming a real chipset driver - iTCO_wdt, hpwdt, ipmi_watchdog, or a hypervisor watchdog such as i6300ESB - is what you need to see.

softdog is not a fencing device

softdog is a kernel module that emulates a watchdog with a kernel timer. It is the module that loads when no hardware watchdog is found, and it makes every configuration check pass.

It cannot do the one job that matters. A kernel that is deadlocked, panicked, or stuck in a non-preemptible loop is a kernel that is not running timers, so the software watchdog never fires. Those are precisely the failure modes for which you deployed a watchdog.

The result is a cluster where stonith-watchdog-timeout tells the survivor “assume the peer has self-fenced by now”, the peer has not self-fenced, and the survivor starts the workload on storage the peer is still queued to write to. The assumption is only sound because the enforcement is in hardware; on softdog it is not an assumption, it is a guess.

Confirm with wdctl that the loaded driver is a real device before enabling any watchdog-based fencing. On a VM, enable the hypervisor watchdog device rather than falling back to softdog.

Disk-based SBD: the poison pill

Disk-based SBD uses a small shared block device - a LUN, an iSCSI volume - that every node can reach. The device holds a header and one message slot per node.

To fence a peer, a node writes a reset message into that peer’s slot. The peer’s SBD daemon, which polls its own slot, reads the message and stops petting the watchdog. The hardware does the rest.

# Create the SBD device: watchdog timeout 15s (-1), msgwait 30s (-4)
sudo sbd -d /dev/disk/by-id/wwn-0x600a098000REPLACEME -1 15 -4 30 create

# Read the header back - never trust the create, read it
sudo sbd -d /dev/disk/by-id/wwn-0x600a098000REPLACEME dump

# Slot allocation, one per node
sudo sbd -d /dev/disk/by-id/wwn-0x600a098000REPLACEME list

Use a /dev/disk/by-id/ path, never /dev/sdb. Kernel device names are assigned in discovery order and will differ between nodes and across reboots; an SBD device pointing at the wrong disk is a fence device that fences nothing, or worse, writes its header over something else.

Configuration lives in /etc/sysconfig/sbd on RHEL and SUSE, /etc/default/sbd on Debian and Ubuntu:

SBD_DEVICE="/dev/disk/by-id/wwn-0x600a098000REPLACEME"
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=15
SBD_STARTMODE=clean
SBD_PACEMAKER=yes

The device is then registered with Pacemaker through the fence_sbd agent, so the cluster treats it as a fence device like any other:

pcs stonith create sbd_fence fence_sbd \
    devices="/dev/disk/by-id/wwn-0x600a098000REPLACEME"

# Confirm the parameters your installed agent actually accepts
fence_sbd -o metadata | grep '<parameter name'

Up to three devices can be configured, on independent storage paths, so that losing one array does not disarm fencing.

The strength of disk-based SBD is that it works over the storage path. If the cluster interconnect fails but both nodes still reach the LUN, the fence completes - which is the common two-node failure, and the case where the storage path is exactly the thing you most want to be authoritative about.

Diskless SBD: self-fencing on quorum loss

Diskless SBD drops the shared device. Instead, SBD stops petting the watchdog when the node loses quorum. There is no message and nothing to reach: a node that finds itself in a minority partition resets itself, and the survivors simply wait out the timeout.

# /etc/sysconfig/sbd or /etc/default/sbd - note the empty SBD_DEVICE
SBD_DEVICE=""
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=5
SBD_PACEMAKER=yes

# The survivor's licence to proceed after waiting
sudo pcs property set stonith-watchdog-timeout=10s

stonith-watchdog-timeout is the cluster telling itself: if a node has been gone for this long, and diskless SBD is in use, treat it as self-fenced and recover its resources. That is a genuine assumption rather than a confirmation - and it is sound only because the watchdog is hardware and the timeout leaves margin.

Diskless SBD needs real quorum, so it is not a two-node solution

Diskless SBD self-fences on quorum loss. On a two-node cluster running two_node: 1, quorum is never lost - each node treats one vote as sufficient by design. Nothing ever triggers the self-fence, both nodes keep petting their watchdogs through a partition, and both keep running.

The mechanism is not merely weakened, it is absent. A two-node cluster with diskless SBD and no other fence device has no fencing at all, while pcs status shows a watchdog-enabled cluster and every check looks satisfied.

For two nodes, use one of: disk-based SBD on a shared LUN, a BMC fence device with a tie-breaking delay, or a quorum device in a third location that turns the pair into a genuine three-vote majority system. Diskless SBD becomes appropriate once quorum is real - three nodes or more, or two nodes plus a qdevice.

The timeout relationships

These values are not independent, and getting them wrong produces either a cluster that fences itself under load or a survivor that starts resources before the peer is down.

Value	Where	Meaning
`SBD_WATCHDOG_TIMEOUT`	node config	How long the hardware waits before resetting an unpetted node
`msgwait`	disk header (`-4`)	How long a sender waits for a poison pill to be acted on
`stonith-watchdog-timeout`	cluster property	How long a survivor waits before assuming a peer self-fenced

The rules to hold:

msgwait at least twice SBD_WATCHDOG_TIMEOUT. The message has to be read and the watchdog has to expire; less margin means the sender concludes the fence worked before it has.
stonith-watchdog-timeout comfortably above the watchdog timeout - conventionally at least twice it for diskless SBD. This is the margin that makes the survivor’s assumption safe.
The watchdog timeout above your worst observed I/O stall. Too tight and a busy node with a slow SAN resets itself, turning a latency spike into a node loss. This is the same trade-off as the corosync token timeout, with a harsher penalty.

Read the values back rather than trusting what you set:

sudo sbd -d /dev/disk/by-id/wwn-0x600a098000REPLACEME dump
sudo cibadmin --query --scope crm_config | grep -i watchdog
wdctl

Testing it

The test that matters is the one that proves the hardware resets the node, not that the daemon started.

# Does sbd see a usable watchdog on this host?
sudo sbd query-watchdog

# Deliver a poison pill to a peer - this WILL reset node2
sudo sbd -d /dev/disk/by-id/wwn-0x600a098000REPLACEME message node2 reset

Check the subcommands your installed sbd supports with sbd --help before relying on any of them; the tooling has grown across releases and not every distribution ships the same version.

Then verify the outcome the way this course verifies every fence: from outside. The node must actually reboot, the survivor must recover the resources, and the fence must appear in pcs stonith history show. A daemon that is running is not a fence that works.