Anatomy of a bad fence: two post-mortems · Proxmox VE

Fencing is the least-loved part of a cluster. It is the mechanism that reboots your servers, and the only evidence it worked is an absence: the corruption that did not happen.

So the first fencing event most operators experience is one they consider a malfunction, and the reflex it produces - “turn that off” - is the single most dangerous response available. This lesson works through two incidents to explain why.

Both are composites of common real failures, with logs reconstructed to show what the evidence actually looks like. The mechanism they demonstrate is described in the Linux course at why fencing exists and STONITH and data integrity; this lesson is the Proxmox-specific forensics.

Where the evidence lives

Three logs, on different machines, telling different parts of the story.

Source	Runs on	Tells you
`journalctl -u pve-ha-crm`	The master node	What the cluster decided and why
`journalctl -u pve-ha-lrm`	Every node	What each node was asked to do and whether it complied
`journalctl -u corosync`	Every node	Membership changes - the input to every decision
`journalctl -k` on the fenced node	The victim, after reboot	Whether the watchdog fired, and whether anything preceded it

Read-only / Safecollect the evidence before it rotates— Read-only. Run this within hours of the event - the default journal retention on a busy node is shorter than the time it takes to schedule a post-mortem.

set -euo pipefail
SINCE='2026-08-11 03:00'
UNTIL='2026-08-11 04:00'
OUT=/root/fence-postmortem-$(date +%F)
mkdir -p "$OUT"

for NODE in pve-01 pve-02 pve-03; do
for UNIT in pve-ha-crm pve-ha-lrm corosync; do
  ssh -o BatchMode=yes "root@$NODE" \
    "journalctl -u $UNIT --since '$SINCE' --until '$UNTIL' --no-pager" \
    > "$OUT/$NODE-$UNIT.log" 2>/dev/null || true
done
ssh -o BatchMode=yes "root@$NODE" \
  "journalctl -k --since '$SINCE' --until '$UNTIL' --no-pager" \
  > "$OUT/$NODE-kernel.log" 2>/dev/null || true
done

ls -la "$OUT"

The fenced node’s own journal is the piece most often lost. It rebooted; if the journal was not persistent, everything it knew about its final minute is gone. Making the journal persistent on cluster nodes - Storage=persistent in journald.conf - costs a little disk and is the difference between a post-mortem and a guess.

Post-mortem 1: the node that was fenced for being busy

What was observed. At 03:41, pve-03 rebooted. Four VMs restarted on other nodes. pve-03 came back healthy in three minutes, with no hardware errors, no kernel panic and clean IPMI logs. Monitoring showed the node had been responsive to ICMP throughout.

What the CRM master recorded:

Read-only / Safepve-02 (CRM master): the decision

# journalctl -u pve-ha-crm --since '03:38' --until '03:45' --no-pager

03:39:51 pve-ha-crm[2411]: node 'pve-03': state changed from 'online' => 'unknown'
03:40:51 pve-ha-crm[2411]: service 'vm:140': state changed from 'started' to 'fence'
03:40:51 pve-ha-crm[2411]: service 'vm:141': state changed from 'started' to 'fence'
03:41:31 pve-ha-crm[2411]: node 'pve-03': state changed from 'unknown' => 'fence'
03:42:11 pve-ha-crm[2411]: successfully acquired lock 'ha_agent_pve-03_lock'
03:42:11 pve-ha-crm[2411]: fencing: acknowledged - got agent lock for node 'pve-03'
03:42:11 pve-ha-crm[2411]: node 'pve-03': state changed from 'fence' => 'unknown'
03:42:11 pve-ha-crm[2411]: service 'vm:140': state changed from 'fence' to 'recovery'
03:42:12 pve-ha-crm[2411]: recover service 'vm:140' from fenced node 'pve-03' to node 'pve-01'

Illustrative output

What corosync recorded on a surviving node:

Read-only / Safepve-01: membership, which is the real input

# journalctl -u corosync --since '03:38' --until '03:45' --no-pager

03:39:47 corosync[1802]: [KNET  ] link: host: 3 link: 0 is down
03:39:47 corosync[1802]: [KNET  ] host: host: 3 has no active links
03:39:49 corosync[1802]: [TOTEM ] Token has not been received in 3225 ms
03:39:51 corosync[1802]: [TOTEM ] A processor failed, forming new configuration
03:39:53 corosync[1802]: [QUORUM] Members[2]: 1 2
03:42:47 corosync[1802]: [KNET  ] rx: host: 3 link: 0 is up
03:43:02 corosync[1802]: [QUORUM] Members[3]: 1 2 3

Illustrative output

What the victim’s kernel log showed after reboot:

Read-only / Safepve-03: the last thing it knew

# journalctl -k -b -1 --since '03:38' --no-pager | tail -6

03:39:12 pve-03 kernel: bond0: (slave eno1): link status down, disabling slave
03:39:12 pve-03 kernel: bond0: now running without any active interface!
03:39:12 pve-03 kernel: vmbr0: port 1(bond0) entered disabled state
03:39:45 pve-03 kernel: bond0: (slave eno2): link status up, enabling slave
03:39:45 pve-03 kernel: bond0: now running without any active interface!
03:41:58 pve-03 kernel: watchdog: watchdog0: watchdog did not stop!

Illustrative output

The reconstruction. A switch firmware update took a link down. The bond had both members on the same switch, so the bond had no active interface for 33 seconds. Corosync lost the node at 03:39:47; the token timeout expired at 03:39:49; the cluster reformed without it. pve-03 had lost quorum, so its LRM could not renew its agent lock, so its watchdog was no longer being reset, and at 03:41:58 the watchdog fired.

Was this a malfunction? No. Every component did exactly what it was designed to do. The node genuinely could not communicate with the cluster, and a node that cannot communicate cannot be trusted not to be running guests.

Was the design correct? No. The failure was a single switch taking out every corosync path simultaneously, which is a design defect, not a fencing defect.

The response that turns this into a real outage

The meeting after this incident will contain a proposal to disable fencing, or to raise the token timeout to sixty seconds so “this cannot happen again”.

Both are worse than the incident.

Disabling fencing does not stop the node being isolated. It stops the cluster being sure the node has stopped - so the next time it happens, the CRM either refuses to recover the guests at all (the guests stay down, with no automatic recovery, which is the outcome HA existed to prevent) or, if misconfigured further, starts them elsewhere while the original node is still running them. That second case puts two live guests on one disk image, and the result is not a longer outage, it is a restore from backup.

Raising the token timeout to sixty seconds means every genuine node failure now takes at least a minute longer to detect, on every future incident, to compensate for a network that should be fixed. You have traded a permanent degradation of your recovery time for a workaround to a cabling problem.

The correct response is in the kernel log: bond0: now running without any active interface. Fix that.

The design changes that actually apply:

Corosync links on physically separate paths. Not two ports in one bond to one switch - a second ring on a different switch, ideally a different NIC. Corosync supports up to eight links and handles the failover itself; this is what corosync link tuning covers, and it is the single highest-value change on this list.
Corosync on a network that backups and migrations do not share. Token loss under load is the other common false-positive source.
Node maintenance mode before switch work. If the switch update had been preceded by ha-manager crm-command node-maintenance enable pve-03, the guests would have migrated off first and the isolation would have cost nothing.

Cluster-wide riskverify every corosync link before touching network hardware— Read-only, but run it before every network change on a cluster. A node showing one link is one switch away from being fenced.

set -euo pipefail

corosync-cfgtool -s

# The same question from the cluster's own view.
pvecm status | sed -n '/Membership information/,$p'

# Which physical interfaces carry those links.
grep -E 'ring[0-9]+_addr|linknumber' /etc/pve/corosync.conf

Post-mortem 2: the fence that nearly did not happen

What was observed. At 21:14, pve-01 stopped responding. Guests did not recover for eleven minutes. When they finally did, an operator had already logged into pve-01’s IPMI and found the node powered on with a responsive console.

What the CRM recorded:

Read-only / Safepve-02 (CRM master): the wait

# journalctl -u pve-ha-crm --since '21:14' --until '21:30' --no-pager

21:14:22 pve-ha-crm[2088]: node 'pve-01': state changed from 'online' => 'unknown'
21:15:22 pve-ha-crm[2088]: service 'vm:100': state changed from 'started' to 'fence'
21:16:02 pve-ha-crm[2088]: node 'pve-01': state changed from 'unknown' => 'fence'
21:16:02 pve-ha-crm[2088]: fencing: waiting for agent lock 'ha_agent_pve-01_lock'
21:17:02 pve-ha-crm[2088]: fencing: waiting for agent lock 'ha_agent_pve-01_lock'
21:20:02 pve-ha-crm[2088]: fencing: waiting for agent lock 'ha_agent_pve-01_lock'
21:25:14 pve-ha-crm[2088]: successfully acquired lock 'ha_agent_pve-01_lock'
21:25:14 pve-ha-crm[2088]: fencing: acknowledged - got agent lock for node 'pve-01'
21:25:15 pve-ha-crm[2088]: recover service 'vm:100' from fenced node 'pve-01' to node 'pve-03'

Illustrative output

What pve-01 looked like after recovery:

Read-only / Safepve-01: what was configured, and what was not

# systemctl status watchdog-mux --no-pager | head -5
ls -l /dev/watchdog*
cat /etc/default/pve-ha-manager

● watchdog-mux.service - Proxmox VE watchdog multiplexer
   Loaded: loaded (/lib/systemd/system/watchdog-mux.service; enabled)
   Active: active (running) since Mon 2026-08-10 09:12:44 CEST
ls: cannot access '/dev/watchdog*': No such file or directory
WATCHDOG_MODULE=ipmi_watchdog

Illustrative output

The reconstruction. WATCHDOG_MODULE=ipmi_watchdog was set during build from a runbook written for a different server generation. On this hardware the module name is different, so it never loaded, so /dev/watchdog never existed

and watchdog-mux started successfully anyway, reporting active (running) in every health check anyone had written.

The node had a storage-driver hang. Its kernel was alive enough to hold the agent lock’s lease alive intermittently but not enough to do useful work. With no watchdog, nothing reset it. The CRM waited eleven minutes for a lock that was only released when the hang cleared on its own.

Why this is the more dangerous incident. The first post-mortem was a correct fence for a bad reason and cost three minutes. This one was an eleven-minute unavailability with no automatic recovery, and it was one scheduler difference away from being worse: had the wedged node resumed executing while still believing it owned the guests, and had the CRM already started them elsewhere, both would have been writing to the same images.

watchdog-mux running is not evidence of a watchdog

This is the check that fools everyone, and it fools monitoring systems too.

systemctl status watchdog-mux reports active (running) on a node with no watchdog device at all. The service starts, finds nothing to multiplex, and sits there. Every “is the service up” check passes. Every dashboard is green. The node has no self-fencing capability and nothing anywhere says so.

The check that actually fails:

ls -l /dev/watchdog || echo 'NO WATCHDOG DEVICE - THIS NODE CANNOT SELF-FENCE'
lsmod | grep -iE 'wdt|watchdog|softdog' || echo 'no watchdog module loaded'

Why the first diagnosis is wrong: after an incident like this everyone concentrates on the storage driver that hung, because that is the visible fault. It is a real fault and it should be fixed. But the storage hang was survivable - a fence would have turned it into a three-minute outage. The eleven minutes came from the missing watchdog, and nobody looks there because watchdog-mux said it was fine.

Put test -c /dev/watchdog in node health monitoring on every cluster you run. It is one line and it is the only thing standing between a hung node and a long outage.

Why the CRM waits for a lock rather than checking whether the node is alive

There is an obvious-looking alternative design: before recovering guests, ping the failed node, or query its IPMI, and confirm it is down. Proxmox does not do this, and the reason is worth understanding because it explains the whole shape of the mechanism.

Every such check answers “was the node reachable from here, just now”. None of them answers “is the node running the guest”. A node can be unreachable from the CRM and perfectly able to write to shared storage over a different path - which is precisely the network-partition case, and precisely the case where getting it wrong destroys data.

So the CRM asks a question that has a safe answer instead. Each LRM holds an agent lock in pmxcfs with a lease. Holding that lock requires quorum. So:

A node with quorum renews its lock and keeps running its guests. Fine.
A node that loses quorum cannot renew. Its LRM sees this, stops its guests, and releases the lock cooperatively. Fast path.
A node that is wedged does nothing, including not renewing. The lease expires. The CRM acquires the lock.

In every branch, “the CRM holds the agent lock” implies “no LRM on that node is in a position to be running guests”. That is a statement about the distributed state machine rather than about network reachability, and it is true regardless of what path the failed node might still have to storage.

The watchdog closes the last gap. Without it, a node could be wedged long enough for its lease to expire and then resume - waking up believing it still owns guests the CRM has already restarted elsewhere. The watchdog guarantees that a node which has not renewed its lease within the timeout has already reset. That is why the watchdog timer must be shorter than the lock timeout, and why a missing watchdog does not merely slow recovery down - it removes the guarantee the whole design rests on.

Fence or not: reading the three cases apart

Evidence	Diagnosis
Corosync link down, node’s kernel log ends at watchdog, node healthy on return	False-positive fence. Fix the network path, not the fencing.
Node fenced, hardware or kernel error in IPMI or the pre-reboot journal	Correct fence. The mechanism did its job; fix the fault.
CRM logs `waiting for agent lock` repeatedly, recovery delayed by minutes	Fencing did not happen. Check for a watchdog device on the victim.
Guests recovered but the original node was never rebooted	Investigate immediately. Verify no guest ran in two places.
Node fenced during a backup or migration window, repeatedly	Corosync sharing a saturated network. Separate it.

What happens if you need to work on a node without it being fenced?

Use the mechanisms that exist for it, rather than disabling anything.

For work on one node - a firmware update, a reboot, a NIC replacement - maintenance mode migrates its HA resources away and marks it unavailable for placement, while leaving the HA stack fully armed everywhere else:

NODE=pve-03
ha-manager crm-command node-maintenance enable "$NODE"
ha-manager status
# ... do the work, reboot as needed ...
ha-manager crm-command node-maintenance disable "$NODE"

Services that were on the node migrate back when maintenance mode ends. The LRM’s watchdog stays active until the migrations complete, so there is no window in which the node is both running guests and unfenceable.

For work that affects the whole cluster - a corosync network change, a switch replacement carrying every link - there is a documented disarm:

ha-manager crm-command disarm-ha freeze
# ... the disruptive work ...
ha-manager crm-command arm-ha
ha-manager status

disarm-ha releases the watchdogs cluster-wide and takes a resource mode: freeze leaves services running and untouched, ignore releases HA’s claim on them. While disarmed there is no automatic failover anywhere, so this is a window you open deliberately, keep short, and close explicitly.

The distinction that matters: both of these are reversible, visible cluster states that ha-manager status reports. Neither is a config file edit that someone forgets. That is the whole reason to prefer them over any improvised alternative - the cluster knows it is disarmed, and so does the next person to run a status command.

The post-mortem write-up

A fencing post-mortem that does not answer these five questions has not finished:

What did corosync see, and when? Membership is the input to everything else. If corosync did not lose the node, the incident is not a fencing incident.
Was the node genuinely unable to participate? Fenced-and-healthy means a network defect. Fenced-and-broken means the mechanism worked.
Did the watchdog fire, or did the CRM wait? These are opposite failure modes with opposite fixes, and the CRM log distinguishes them in one line.
Could any guest have run in two places? If the answer is not a confident no, treat the affected images as suspect and verify them.
What would have prevented it without weakening the guarantee? If the proposed remedy is a longer timeout or less fencing, the analysis is not finished.

Knowledge check

Why this matters

Fencing is the part of a cluster whose value is invisible and whose costs are not. Every fence is a server rebooting and a service restarting; the corruption it prevented leaves no trace. So the pressure after a fencing incident is always in one direction - do less of this - and that pressure is almost always wrong.

The reframe that helps is to stop asking whether the fence was warranted and start asking which of three things happened. A node fenced while healthy is a network incident. A node fenced while broken is the system working. A node that should have been fenced and was not is the serious one, and it is the only one of the three that people routinely fail to recognise, because its symptom is a slow recovery rather than an abrupt one.

That third case is why test -c /dev/watchdog belongs in every cluster’s node health checks. The cost of adding it is one line. The cost of not having it is an eleven-minute outage that nobody can explain, followed by a post-mortem that blames the storage driver and leaves the actual defect in place for next time.