Fencing design principles - evaluating any platform · Linux

This part has covered IPMI, Redfish, iLO, iDRAC, hypervisor agents, SCSI reservations and SBD. You will eventually be asked to build a cluster on a platform none of those cover - a cloud that is not AWS, an appliance, a bare-metal provider, a hypervisor with its own API.

“Which agent do I use?” is the wrong first question. The right one is what any method has to do to count as fencing at all, and that question has a hardware-independent answer.

The four properties

A method is a fencing method if and only if it has all four. Three out of four is not a fence.

1. Independence

The fence path must not depend on the thing that failed.

You are fencing because something broke. If the fence travels over the broken thing, it cannot complete. The classic violation is a BMC on the same switch as the cluster interconnect: the switch fails, that failure is the partition that triggers the fence, and the fence request goes out over the switch that is down.

2. Decisiveness

Exactly one side can win.

Fencing exists to resolve a disagreement between two partitions about who owns a resource. A method where both sides can succeed has not resolved anything. Two nodes fencing each other simultaneously over independent BMCs can both win, killing the service - which is why two-node designs need a tie-breaking delay or a third vote, covered in linux-two-node-clusters-and-tie-breaking.

3. Confirmability

It returns a positive confirmation that the target stopped, not that a request was sent.

This is the property most often missing. “The API accepted the power-off call” and “the machine is off” are different facts. An agent that returns success on HTTP 202 has told you about a queue, and Pacemaker will proceed to start resources on the strength of it.

The test is whether you can read the resulting state back over a path independent of the request - a power status query, a reservation listing, a hypervisor instance state.

4. Reachability under the failure it handles

It must work during the specific failure it is there for, and be tested in that failure.

Every fence device works when the cluster is healthy. That test proves nothing, because a healthy cluster does not need fencing. A method is only validated by exercising it while the failure it exists for is in progress.

Three out of four passes design review and fails in production

Each of these is a real design that was signed off:

Independence missing. BMCs on the cluster VLAN. Fencing worked in every test, because every test broke a node rather than the network. The first switch failure produced a partition that also removed the fence path, both sides waited on fences that could never complete, and the service stayed down at both sites.
Decisiveness missing. Two nodes, two independent BMCs, no delay. The first real partition killed both nodes within a second of each other. The fencing worked perfectly, twice.
Confirmability missing. A cloud API that returns immediately and stops the instance a few seconds later. The agent reported success, Pacemaker mounted the volume on the survivor, and the original instance completed two writes on the way down.

Write the four properties down as acceptance criteria and check them individually. The failure mode of an informal review is that everyone agrees fencing is configured, because it is.

Evaluating the common methods

Method	Independent of network failure?	Decisive?	Confirmable?
BMC power off (IPMI, Redfish, iLO, iDRAC)	Only if the BMC is on a separate network	Not on its own between two peers	Yes - read the chassis power state back
PDU outlet off	Yes, if the PDU network is separate	Yes, if only one node per outlet	Yes - read the outlet state
Hypervisor stop / destroy	Depends on where the API lives	Yes - the platform arbitrates	Yes - query the instance state
SCSI-3 persistent reservation	Yes - travels over the storage fabric	Yes - the array arbitrates	Yes - `sg_persist --in --read-reservation`
Disk-based SBD	Yes - travels over the storage fabric	Yes - one slot per node	Indirectly, via the reset and the watchdog
Diskless SBD	Yes - reaches nothing at all	Yes, where quorum is real	No - the survivor waits and assumes
A script that SSHes to the target	No - needs the target healthy and reachable	No	No

Two rows deserve comment.

SCSI reservations and hypervisor APIs are decisive because a third party arbitrates. The array or the platform serialises the requests and one of them loses. This is structurally stronger than two peers racing to power each other off, and it is worth preferring when the platform offers it.

Diskless SBD is the one method with no confirmation, and it is safe anyway - because the survivor’s wait is backed by a hardware timer the failing node cannot influence. That is an argument about the mechanism, not an exception to the rule. Substitute softdog and the argument collapses.

The fence path matrix

The deliverable that makes a fencing design reviewable is not “we use IPMI”. It is a table with one row per failure mode:

Failure	Who fences whom	Over what path	Is that path up?
One node loses power	Nobody - it is already off	BMC confirms power state	Yes, if the BMC has standby power
One node’s kernel stalls	Survivors fence it	BMC on the management network	Yes - unrelated to the stall
Cluster interconnect fails, nodes healthy	Survivor majority fences the minority	BMC on the management network	Yes - separate network from the interconnect
The management network fails	Level 2 takes over	PDU on its own network, or SBD over storage	Yes - independent of management
Inter-site link fails	Neither side can reach the other	Nothing	No - self-fencing or booth is required
Storage fabric fails	Survivors fence the isolated node	BMC	Yes, but SBD would not work here

The value is in the last column. A row that says “no” is a failure mode with no fencing, which means a failure mode where the cluster will refuse to recover - and where somebody will be tempted to disable STONITH at three in the morning.

That is not a reason to hide the row. It is the reason to have the table: each “no” is either accepted deliberately, or fixed by adding a method with a different dependency. The inter-site row is the one that most often needs SBD self-fencing or booth tickets, as linux-quorum-concepts sets out.

Anti-patterns

The fence device shares a network with the interconnect. Violates independence. The most common defect in real clusters and the one that produces the worst outage.
The fence device shares a power feed with its target. Losing the feed takes the node and its BMC together. The node really is off, but you cannot confirm it, so the fence fails and Pacemaker will not recover the resources. Correct, and an outage.
Storage fencing alone on a node holding a service IP. A SCSI reservation stops the writes. It does not stop the node answering clients on the VIP, so users are served by a node that cannot reach its data.
One fence device per cluster instead of per node. A single device is a single point of failure for the mechanism that exists to handle failures.
Fence agents that were never run by hand. Configure, never test, discover the parameters were wrong during the first real incident.
A fence topology registered with the arguments reversed. Covered in linux-stonith-and-data-integrity: the target comes before the device id, and getting it backwards registers a topology for a node that does not exist, silently.

Evaluating a platform this course does not cover

Ask these, in order, and stop when one has no answer:

What can stop a machine here without its cooperation - a power API, a hypervisor stop, a storage reservation, a watchdog device?
Does that control path share infrastructure with the cluster interconnect or with the storage the cluster uses?
Can I read the resulting state back, over a path independent of the one that issued the request?
When two nodes issue conflicting requests, does something arbitrate, or do both succeed?
Is there an existing fence agent, and does <agent> -o metadata list the parameters I think it takes?
Can I test it while the failure it handles is in progress?

If question 1 has no answer, the platform cannot host a cluster with shared writable state. That is a legitimate finding, and it is far better delivered during design than during an incident. Replication with a deliberate manual promotion is the honest alternative.