high riskcluster affecting~90 min

Respond to a fencing event and a self-fenced node

1 · Prerequisites

Confirm every item is in place before any state change.

HA is configured on this cluster, so a self-fence is a possible behaviour rather than an unexplained crash
Out-of-band console access to the fenced node, because the useful evidence is in the previous boot and the node may fence again
The corosync link topology is documented: which interfaces, which VLANs, whether links are shared with storage or migration traffic
A monitoring history exists for the window, or its absence is acknowledged as a limit on the investigation
The service owners for the guests that were running on the fenced node can be reached

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· ha-manager status shows the current state of every HA service and which node now owns each one
· pvecm status shows whether the cluster is quorate now, and how many nodes are members
· uptime -s on the fenced node shows a boot time consistent with the reported event
· The guests that were on the fenced node are confirmed running on exactly one node each - not zero, and not two
· journalctl --list-boots shows the boot boundary, so the previous boot can be read
· corosync-cfgtool -s on every node shows all links up right now

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm service state first: for every guest that was on the fenced node, establish it is running on exactly one node
2Confirm the fenced node is fully back and stable, or is deliberately kept out
3If the node is fencing repeatedly, put it into HA maintenance mode to stop the cycle without disabling the watchdog
4Read the last messages of the previous boot on the fenced node - the fence is usually visible as an abrupt end
5Read corosync on the surviving nodes for the same window: token loss, link down, retransmits
6Classify the cause: network, node hang, storage stall, resource exhaustion, or a deliberate action
7Correlate against switch, storage and hypervisor metrics for the same minute
8Fix the underlying cause; a fence is a symptom and the watchdog is not the fault
9Return the node to service gradually, starting with one guest that can be lost
10Verify HA state is clean and no service is left in an error or stale state
11Record the timeline, the cause, and the data impact of any guest that was killed rather than shut down

4 · Verification

Confirm the procedure actually fixed the problem.

✓ha-manager status shows every service started, none in error, fence or stopped state
✓Each guest that was on the fenced node is running on exactly one node - verified by checking every node, not by trusting the GUI
✓pvecm status is quorate on every node with expected votes equal to the node count
✓corosync-cfgtool -s shows every configured link up on every node, with no retransmit growth over a five-minute observation
✓The fenced node has stayed up for longer than the interval at which it was previously fencing
✓journalctl on the fenced node shows a clean boot with no repeated watchdog or corosync errors
✓The guests that were killed by the fence have been checked for filesystem consistency at the guest level, not merely for booting

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A fence cannot be rolled back. The node rebooted; the guests were killed and restarted elsewhere. This procedure is about stabilising, not reversing
↶If HA maintenance mode was enabled to stop a fence loop, disabling it again returns the node to normal HA participation - do that only after the cause is fixed
↶If guests were manually started to speed recovery, they must be reconciled with HA state or HA may try to start them a second time
↶Do NOT disable the watchdog or remove HA from the node as a way of stopping repeated fencing. That removes the protection, not the fault, and the next event becomes two nodes running the same guest
↶Any temporary corosync timing change made to stop a flap must be recorded and reverted once the network fault is fixed, because loosened timings hide the next fault too

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if any guest is found running on two nodes at once - that is active data corruption, not a recovery
· Escalate to the network team if corosync shows token loss or link failure, because a fence caused by the network will repeat on any node
· Escalate to the hardware owner if the node hung rather than lost the network - a hang with no journal evidence is usually firmware, memory or a storage controller
· Escalate to the cluster owner before any change to corosync timings or HA configuration
· Escalate to the service owners with the guest-level impact: these guests were killed, not shut down, and may need application-level consistency checks

Verified against Proxmox VE 9.2.4.

A node rebooted without anyone asking it to, and the guests it was running came up somewhere else. That is not a crash - that is the cluster working.

Proxmox HA uses self-fencing: a node that is running HA-managed guests arms a hardware or software watchdog. The local resource manager pets that watchdog only while the node holds quorum and can see the cluster. Lose quorum, or hang badly enough that the LRM stops running, and the watchdog is not petted, and roughly a minute later the node resets itself. Only then is the CRM allowed to start those guests elsewhere.

The reset is the precondition for recovery, not the failure. Which sets the shape of this response: verify the recovery, find what made the node unable to participate, and resist the very strong temptation to remove the watchdog.

When to use this runbook

A node rebooted with no operator action and no kernel panic in the journal.
ha-manager status shows or showed a service in fence state.
Guests moved between nodes without anyone migrating them.
A node is rebooting repeatedly on roughly a one-minute cycle.

What actually happened, in order

t+0s     node loses quorum, or the LRM stops running
t+~10s   LRM stops petting the watchdog
t+~60s   watchdog expires -> the node hard-resets. Guests are killed, not shut down
         (the exact interval depends on the watchdog and HA timing in use)
t+~120s  CRM confirms the node is fenced and may safely recover its services
t+~150s  guests start on surviving nodes

Two consequences worth stating plainly. The guests were killed, so their filesystems are in crash-consistent state and their applications did not flush. And there is a gap of a couple of minutes between the fault and the recovery, by design - the CRM must not start a guest until it is certain the old copy is dead.

Step 1: Service state first, investigation second

Read-only / Safewhere is everything running

ha-manager status
pvecm status
pvesh get /cluster/resources --type vm --output-format yaml | grep -E 'vmid|node|status'

Read-only / Safethe check that matters most: not running twice

# Run on EVERY node, including the one that fenced.
hostname -s
ps -eo pid,cmd | grep -oP '(?<=-id )[0-9]+' | sort -n | tr '\n' ' '
echo
pct list 2>/dev/null | awk 'NR>1 {print $1, $2}'

Step 2: Is it going to fence again?

Read-only / Safehow many times, how recently

journalctl --list-boots | tail -10
uptime -s
last -x reboot | head -10

A node that has rebooted three times in ten minutes is in a fence loop. Stop the loop before investigating, or you will lose your shell mid-command repeatedly.

Service impact possiblebreak the loop without removing the protection

NODE=pve02

# Moves HA services off and stops HA arming the watchdog on this node.
ha-manager crm-command node-maintenance enable "$NODE"
sleep 30
ha-manager status

Maintenance mode is the correct lever here. It tells the cluster this node should not carry HA services, so the LRM has nothing to protect and does not arm the watchdog. The protection is intact everywhere else, and it comes back on this node when you disable maintenance mode.

Step 3: Read the previous boot

The fence leaves a characteristic signature: the journal stops mid-sentence.

Read-only / Safethe last thing the node said before it reset

journalctl -b -1 --no-pager | tail -80
journalctl -b -1 -u corosync --no-pager | tail -40
journalctl -b -1 -u pve-ha-lrm --no-pager | tail -40
journalctl -b -1 -u pve-cluster --no-pager | tail -40
journalctl -b -1 -p err --no-pager | tail -40

What to look for, in order of how often it is the answer:

Evidence in `-b -1`	Cause class
`Token has not been received in NNN ms` then `link down`	Corosync network. Most common by a wide margin
LRM logs `lost lock 'ha_agent_NODE_lock'`	Node lost quorum or pmxcfs access
Journal simply stops with nothing unusual	The node hung. Nothing got written because nothing was running
Kernel messages about a blocked task or an I/O timeout	Storage stall - the LRM could not run because it was in uninterruptible sleep
OOM killer messages	Memory exhaustion starved the HA services
`pmxcfs` errors before the stop	Cluster filesystem problem; see the pmxcfs runbook

Read-only / Safethe same window, from the survivors

SINCE='2026-08-12 02:30'
UNTIL='2026-08-12 02:45'

journalctl -u corosync --since "$SINCE" --until "$UNTIL" --no-pager | tail -60
journalctl -u pve-ha-crm --since "$SINCE" --until "$UNTIL" --no-pager | tail -40

The surviving nodes saw the event from the outside. If they logged token loss for one node only, the fault is that node’s link. If several nodes logged it at once, the fault is the shared network - and it will happen again, to whichever node is unlucky next time.

Step 4: Examine the corosync path

Read-only / Safelinks, and whether they are shared with anything

corosync-cfgtool -s
corosync-cfgtool -n
grep -E 'ring[0-9]_addr|name:|transport|token' /etc/pve/corosync.conf

# Is the corosync interface also carrying storage or migration traffic?
ip -br addr
cat /etc/pve/datacenter.cfg

Read-only / Safewatch for latency and loss on the corosync path

PEER=192.0.2.12

ping -c 200 -i 0.2 -q "$PEER"
# Corosync is sensitive to jitter, not just to loss. Look at mdev,
# and at any packet loss at all.

ping -c 20 -M do -s 8972 "$PEER" || echo 'jumbo MTU path is broken'

Read-only / Safewhat else was happening at that minute

# Task history around the event
grep -h "$(date -d '2026-08-12 02:30' +%Y-%m-%d)" /var/log/pve/tasks/index 2>/dev/null | tail -30

# Backup jobs
cat /etc/pve/jobs.cfg 2>/dev/null

Step 5: If the node hung rather than lost the network

A hang leaves no journal evidence, which is itself the evidence. Check the hardware view instead:

Read-only / Safehardware and firmware

# IPMI system event log - the node's own account of the reset
ipmitool sel list | tail -30
ipmitool sel elist | grep -iE 'watchdog|reset|ecc|memory|power' | tail -20

dmesg -T | grep -iE 'mce|hardware error|edac|nmi' | tail -20
journalctl -b -1 -k --no-pager | grep -iE 'blocked for more than|task .* blocked|io error' | tail -20

An IPMI SEL entry for a watchdog timeout confirms the hardware watchdog fired. Memory or MCE errors point at the node itself and the answer is hardware, not configuration.

Step 6: Fix the cause, then return the node

Service impact possiblereturn to service gradually

NODE=pve02
VMID=9001

# Only after the cause is understood and fixed.
ha-manager crm-command node-maintenance disable "$NODE"
sleep 20
ha-manager status

qm migrate "$VMID" "$NODE" --online
sleep 30
pvesh get /nodes/"$NODE"/qemu/"$VMID"/status/current --output-format yaml

Leave it carrying one low-value guest for longer than the interval at which it was previously fencing. If it was fencing every ten minutes, “it has been fine for two minutes” proves nothing.

Step 7: The guests were killed - check them

Read-only / Safecrash consistency is not consistency

VMID=104

qm status "$VMID"
qm agent "$VMID" ping

# Inside the guest, or via the agent:
#   journalctl -b -1 -e        (did it log a clean shutdown? it did not)
#   systemctl --failed
#   filesystem check status for any non-journalled filesystem
#   database: check the engine's own recovery log

A VM that was fenced was powered off mid-write. Journalling filesystems usually recover; databases usually recover through their own write-ahead logs; applications that buffered in memory did not. Ask the service owner to check at the application level, and record which guests were affected.

Rollback

There is none for the fence itself. The relevant table is what to undo from your response:

Action taken	How to undo
`node-maintenance enable`	`node-maintenance disable`, once the cause is fixed
Guest started manually during the outage	Reconcile with `ha-manager status`; a service HA also started is the double-run risk
Corosync timings loosened to stop a flap	Revert once the network fault is fixed, and record that it was temporary
Watchdog disabled (do not do this)	Re-enable immediately and treat the interval as unprotected

Common patterns

Symptom	Likely cause	Resolution
Fence at 02:00 every night	Backup traffic saturating the corosync link	Separate corosync onto its own network, or add a second link
One node fences repeatedly, others never	That node’s NIC, cable, switch port or bond	Move the port; check the bond members individually
All nodes log token loss simultaneously	Shared switch, spanning-tree event, or MTU mismatch	Network team. This will recur on any node
Node resets with an empty journal	Hardware hang or storage stall	IPMI SEL, memory diagnostics, storage controller firmware
Fence right after a large live migration	Migration traffic on the corosync path	Set a dedicated migration network in `datacenter.cfg`
Guest in `error` state after recovery	CRM could not start it on any node	`ha-manager set SERVICE --state started` after fixing the reason
Node fences whenever a specific VM starts	Resource exhaustion, or an interface flap caused by a passthrough device	Investigate that guest specifically

Escalation

Escalate when:

Any guest is found running on two nodes.
Corosync shows token loss on more than one node.
The node hung with no journal evidence.
Any change to corosync timings or HA configuration is being considered.
Guests need application-level consistency checks after being killed.

Why this matters

The pressure in this incident points in exactly the wrong direction. A node that keeps resetting itself feels like the problem, and there is a one-line command that makes the resetting stop. Taking it trades a visible, recoverable disruption for an invisible, unrecoverable one: without fencing, HA cannot know a node is dead, and the failure mode becomes two hypervisors writing the same disk image.

The framing that keeps you honest is that the watchdog is a consequence, not a cause. Something made the node unable to participate in the cluster for a full minute. The reset is what the design does about that, and it did its job. The investigation is about the minute before it.

And in practice that minute is nearly always the network. Corosync needs small packets to arrive on time, and it shares a link with a backup job or a live migration far more often than anyone intends. The timestamps usually tell you before the logs do - check what else was running at that exact minute before you check anything else.