Unresponsive is not dead - the evidence problem fencing solves

linux-why-fencing-exists says fencing prevents the minority from writing. This lesson is about the assumption underneath that sentence, because it is the assumption every operator breaks under pressure: that a node which stopped responding has stopped doing things.

It has not. Those are two different claims about two different systems, and confusing them is how a twenty-minute outage becomes a restore from backup.

Two claims that are not the same claim

“node2 is not responding.”

This is a statement about your ability to hear node2. It is produced by the cluster interconnect, by ICMP, by SSH - all paths from you to it.

“node2 has stopped writing to the shared LUN.”

This is a statement about node2’s ability to reach the storage. It is produced by an entirely different path: a different NIC, often a different fabric, sometimes a different physical network with its own switches.

Nothing about the first claim implies the second. They travel over separate hardware, and the failure that broke one has no obligation to have broken the other. A node can be completely silent to the cluster and fully able to write, and that is not an exotic corner case - it is the single most common way a two-writer corruption happens.

Where a silent node keeps writing

Six mechanisms, all of them ordinary:

1. The cluster link failed and the storage path did not. Corosync runs on bond0; the SAN runs on separate HBAs or a storage VLAN. Lose the cluster bond and the node vanishes from the membership while its database keeps committing transactions to a LUN it can still reach perfectly.

2. The corosync process died and the workload did not. The OOM killer picks a process by badness score, not by importance. Kill corosync and the node leaves the membership instantly, while PostgreSQL - which was the thing consuming the memory - carries on serving and writing.

3. The kernel is stalled, with I/O already queued. A node in a long uninterruptible stall answers nothing at all: no ping, no SSH, no token. But writes already submitted to the HBA are queued in hardware, and the controller will issue them when the stall clears. The node looks more dead than a dead node while holding a loaded gun.

4. The hypervisor paused it. A VM stalled by live migration, a snapshot quiesce, or storage contention on the host is not executing instructions at all. When it resumes it continues from the exact instruction it was on, with its page cache intact and its idea of the filesystem unchanged - and writes it.

5. Write caches have not been flushed. A RAID controller with write-back cache, or a disk with its own cache, has accepted data the node considers written. That data lands on the media on the controller’s schedule, not on yours.

6. The partition is asymmetric. node1 cannot hear node2. It does not follow that node2 cannot hear node1, or that node2 cannot hear the storage. Unicast firewall rules, a failed transceiver in one direction, and multicast filtering all produce one-way silence.

Why waiting does not help

The most tempting shortcut is patience: “it has been unreachable for ten minutes, it must be dead.”

Ten minutes is not a bound on anything relevant:

• There is no timeout after which a stalled kernel is guaranteed not to issue queued I/O. Recovery from a storage path failure can take minutes and the queue survives it.
• A paused VM can be resumed by an administrator, by a hypervisor scheduler, or by the completion of a migration, hours later. Nothing decays.
• A network fault that healed is a node rejoining, with its workload still running. A node that comes back after a long absence and finds its resources running elsewhere is the same hazard as one that never left.

The passage of time changes your confidence. It does not change the node’s ability to write.

What counts as evidence

Evidence is a positive observation, made over a path that does not depend on the failed one, that the node has stopped.

And the things routinely mistaken for evidence, none of which are:

Getting the evidence

The commands are short. The discipline is using a path that does not share the failure:

# ipmitool -I lanplus -H 192.0.2.50 -U stonith -f /etc/ipmi/bmc.pw chassis power status

Chassis Power is off

# Did a fence action actually succeed? Not "was one requested".
pcs stonith history show
pcs status --full | sed -n '/Failed Fencing Actions/,/^$/p'

# Who currently holds a reservation on the shared LUN
sudo sg_persist --in --read-reservation /dev/mapper/shared-lun

If the BMC is on the management network and the management network is the one that failed, you do not have evidence - you have a second unanswered question. Go to the console, the PDU, or the rack.

The order that keeps the data

1. Treat the unresponsive node as alive and writing. This is the default assumption, and it is discarded only by evidence.
2. Get a power state from a path that does not depend on the failure: BMC, hypervisor API, PDU, or a physical look at the machine.
3. If the node is running, fix the network. The cluster recovers on its own when the rings return, and no data is at risk.
4. If the node is genuinely down, power it off deliberately so it cannot rejoin mid-recovery with its workload still running.
5. Only then relax quorum or force a resource up, and record it as a decision with a time and an owner.

The step people skip is the second one, and they skip it because the answer feels obvious. The whole discipline is refusing to act on an obvious answer that nothing confirmed.

Knowledge check