HA and fencing incidents · Proxmox VE

HA incidents have a characteristic shape: something moved, or something rebooted, and no human did either. Part XII explains how the HA stack is meant to work. This lesson is about reading what it actually did.

The single most useful framing to bring: the HA stack almost never fails. It does what it is specified to do, in response to something else that failed. The diagnosis is therefore rarely “what is wrong with HA” and nearly always “what did HA see”.

Read the state first

Read-only / Safethe three views of HA state— Read-only. ha-manager status is the summary; adding --verbose includes the full manager status. The two files are the raw state the CRM and each node LRM publish, and they are worth reading directly when the summary is ambiguous.

ha-manager status

ha-manager status --verbose

ha-manager config

cat /etc/pve/ha/manager_status

cat "/etc/pve/nodes/$(hostname)/lrm_status"

Read-only / Safe

$ ha-manager status

quorum OK
master pve-a1 (active, Tue Aug 12 03:14:02 2026)
lrm pve-a1 (active, Tue Aug 12 03:14:05 2026)
lrm pve-a2 (active, Tue Aug 12 03:14:03 2026)
lrm pve-a3 (idle, Tue Aug 12 03:14:04 2026)
service vm:141 (pve-a3, started)
service vm:142 (pve-a1, error)
service ct:205 (pve-a2, started)

The requested states

These are what you asked for, set with ha-manager set <sid> --state <state>:

State	Meaning
`started`	Keep it running; restart on failure, relocate if needed
`stopped`	Keep it stopped, but still relocate it on node failure
`disabled`	Stop it and do not relocate. This is the state you use for error recovery
`ignored`	Take it out of HA entirely; API calls bypass the HA stack

The CRM service states

These are what the cluster resource manager currently believes, and they are the vocabulary of any HA incident:

State	What it tells you
`stopped`	LRM has confirmed it is stopped
`request_stop`	CRM has asked; it is waiting for the LRM to confirm
`starting`	Start requested, awaiting LRM confirmation
`started`	Running
`fence`	Waiting for a node to be fenced — the node lost quorum and its services cannot be recovered until it is known to be dead
`recovery`	Trying to place the service on an available node
`freeze`	State deliberately left unchanged, during a node reboot or an LRM restart
`migrate`	Live migration in progress
`relocate`	Stop-and-start on another node in progress
`error`	Repeatedly failed; disabled until a human intervenes

fence and recovery are the two states that explain an outage

If you catch an incident live, these two are the ones that tell you what is happening and roughly how long it will take.

fence means the CRM will not touch the service yet, because the node it was on has not been confirmed dead. Recovering a service onto another node while the original might still be running it is precisely the split-brain the whole mechanism exists to prevent, so it waits. The wait is bounded by the watchdog: the documentation states a node whose LRM or CRM process is not scheduled or is killed will reboot after the watchdog has timed out, which happens after 60 seconds.

recovery means fencing is confirmed and the CRM is now placing the service. From here it is a normal start on a new node, subject to storage being reachable there.

The documented end-to-end figure is worth quoting to whoever is asking how long: ha-manager has typical error detection and failover times of about two minutes.

The node that rebooted with nothing in its logs

The classic watchdog reset, and its signature is an absence rather than a presence.

Read-only / Safedid this node shut down, or was it reset?— Read-only. A clean shutdown leaves a trail: systemd stopping units, a final journal entry, a recorded reason. A watchdog reset leaves the previous boot ending mid-sentence. Compare the last entries of the previous boot against the first of the current one.

journalctl --list-boots | tail -5

journalctl -b -1 -n 60 -o short-precise

journalctl -b -1 -u pve-ha-lrm -u pve-ha-crm -u corosync -o short-precise | tail -60

journalctl -b -0 -n 20 -o short-precise

Read-only / Safe

$ journalctl -b -1 -n 8 -o short-precise

Aug 12 03:11:58.204 pve-a2 corosync[1841]: [KNET  ] link: host: 1 link: 0 is down
Aug 12 03:11:58.205 pve-a2 corosync[1841]: [KNET  ] link: host: 3 link: 0 is down
Aug 12 03:12:03.912 pve-a2 corosync[1841]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 12 03:12:03.913 pve-a2 pve-ha-lrm[2210]: lost lock 'ha_agent_pve-a2_lock - cluster had no quorum
Aug 12 03:12:14.001 pve-a2 pmxcfs[1799]: [status] notice: node lost quorum
Aug 12 03:12:44.117 pve-a2 pve-ha-lrm[2210]: status change active => lost_agent_lock

The trigger is above the fence, not at it

Everything in that log after the first line is the HA stack behaving correctly. The interesting line is the first one: the corosync links went down.

This is the single most common misdiagnosis in Proxmox operations. The node rebooted, so the investigation focuses on the node — its hardware, its kernel, its memory. But the node rebooted because it was told to by a watchdog, which was armed because it was running HA services, which lost their lock because the node lost quorum, which happened because the network stopped delivering corosync traffic.

Diagnose upward from the fence:

Did the node lose quorum? journalctl -b -1 -u corosync on the fenced node, and the same window on nodes that stayed up.
Why? Link down, retransmits, a switch event, an MTU change, a bond failover, a firewall rule, a spanning-tree reconvergence, a backup saturating a shared link.
Was it one node or several? One node points at that node’s link or NIC. Several point at shared infrastructure, and shared infrastructure taking out several nodes at once is a design finding as much as an incident.

The remedy is almost never in the HA configuration. It is a redundant corosync link on separate hardware, or a link that is not shared with bulk traffic.

Disabling HA to stop the fencing

A cluster fenced two nodes in a week. Under pressure, the team removed the HA resources so it would stop happening.

It did stop happening. The corosync instability continued, unmonitored and unremarked, because nothing was reacting to it any more. Six weeks later a network change caused a longer partition, guests on both sides of it kept running against the same shared storage, and the resulting filesystem corruption took two days to recover from backup.

Fencing is not the problem. Fencing is the mechanism that guarantees a guest is not running in two places at once, and turning it off does not fix the instability — it removes the protection against the consequence.

If the fencing genuinely must stop while a network fault is investigated, the honest move is to set the affected services to ignored, record that HA protection is suspended, and keep the corosync instability as an open, monitored incident with an owner. Anything else is a decision to run unprotected and forget you did.

Recovering a service from `error`

The documented sequence, in order, and the order matters:

Bring the resource back into a safe and consistent state — for example, kill its process if the service could not be stopped.
Disable the resource to remove the error flag.
Fix the error that led to the failures.
Request that the service starts again once all errors are fixed.

Service impact possiblerecover a service stuck in error— Affects a guest. Step one is inspection; the disable clears the error flag and stops HA from acting; the final set returns it to HA management. Do not skip straight to --state started, which re-enters the same failure loop.

SID="vm:142"

ha-manager status
qm status 142

ha-manager set "$SID" --state disabled

qm start 142
qm stop 142

ha-manager set "$SID" --state started
ha-manager status

Why error exists, and why re-enabling immediately is wrong

A service reaches error after exhausting its restart and relocate budget — max_restart attempts on the current node (default 1) and then max_relocate attempts elsewhere (default 1). The state exists to stop the loop.

Setting it straight back to started without fixing anything restarts the loop: the same start failure, the same relocations, the same error state, plus more noise in the task log and, on a storage-related fault, more load on the storage that is already unhappy.

The step that is genuinely skipped most often is step one. A guest that “could not be stopped” may still have a QEMU process running, and starting it again elsewhere while that process holds its disks open is a real path to corruption on any storage that does not enforce exclusivity. Confirm the guest is actually down — qm status, and the absence of its process on the node it was on — before you re-enable it.

Under the hood

What is the lock, and why is the watchdog tied to it?

Each node’s LRM holds an exclusive cluster lock — its “agent lock” — taken through pmxcfs. Holding it is what entitles that node to act on HA services. The CRM holds a cluster-wide manager lock, and acquiring a failed node’s agent lock is what entitles the CRM to recover that node’s services elsewhere.

Because the locks live in pmxcfs, they require quorum. A node in a minority partition cannot renew its agent lock, and that is the event the log above records as lost lock and then status change active => lost_agent_lock.

Now the watchdog. Fencing has to guarantee that a node whose services are being recovered elsewhere is definitely not still running them. There is no way to ask it, because by definition you cannot talk to it. So the guarantee is made by the node itself: while an LRM holds services, it arms a watchdog and must keep petting it. Losing the agent lock means it stops, and the watchdog then resets the node — the documentation states this happens after the watchdog has timed out, which happens after 60 seconds, and the same applies if the LRM or CRM process is not scheduled or is killed.

Two consequences follow that matter operationally.

A fenced node reboots without flushing anything. It is a hardware reset, not a shutdown. Guest filesystems see it as a power loss, and anything unwritten is gone. That is why a fencing event is a data-integrity event for the guests involved, not merely an availability one.

Whether the watchdog is armed depends on whether services are held. A node with no HA services has nothing to arm, which is why one node in a cluster can fence while another in the same partition simply goes read-only and waits. ha-manager status shows an LRM as idle when it holds no services and active when it does, and that distinction predicts which nodes are at risk in a partition.

HA incident checklist

Establish what happened

ha-manager status — current state, and which services are where.
journalctl -b -1 on any node that rebooted — did it shut down or stop mid-sentence?
journalctl -u corosync --since <window> on every node, not just the one that rebooted. The healthy side’s view of the partition is half the evidence.
/etc/pve/ha/manager_status for the CRM’s record of state transitions.
pvecm status and corosync-cfgtool -s for the current link picture.

Establish why

Was there a network change, a switch event, a firmware update, a backup window, or a storage event in the same minutes? Correlate before theorising.
One node or several? One points at that node; several point at shared infrastructure.
Check for retransmits in the days before: journalctl -u corosync --since "-7d" | grep -i retransmit. Fencing events usually have weeks of warning.

Recover

Guests that were fenced have had a power-loss event — check their filesystems and applications rather than assuming a clean restart.
Services in error: the four-step sequence, in order.
Do not disable HA as a remedy. Suspend it explicitly with ignored if you must, with an owner and an end date.

Prevent

A second corosync link, on separate physical infrastructure. This is the fix for the large majority of fencing incidents.
Alert on corosync retransmits, not only on fencing.

What happens if HA recovers a guest whose storage the new node cannot reach?

This is the second-commonest HA surprise, and the mechanism is worth following because it explains a service that bounces between nodes.

The CRM places the service on a node it considers eligible. Eligibility is about HA rules and node availability.

The LRM on that node tries to start the guest and fails, because the guest’s disks are on a storage that is node-restricted, not shared, or currently inactive there.

The budget is consumed. max_restart attempts locally, then max_relocate attempts elsewhere — both defaulting to 1 — and then the service lands in error. On a cluster where several nodes cannot reach the storage, you see the guest appear briefly on two nodes and then stop, which reads as instability rather than as a placement problem.

The diagnosis is in the LRM’s start attempt, not in the HA state. The task log for the failed start names the storage, and pvesm status on that node shows it inactive or absent.

The fixes, in order of preference:

Make the storage genuinely available everywhere the guest may run. This is the real answer, and it is usually why shared storage is a prerequisite for HA in the first place.
Constrain placement to nodes that can run the guest, using the HA rules and affinity mechanisms from Part XII, so the CRM never chooses an impossible node.
If neither is achievable, the guest should not be under HA. A service that HA cannot actually recover is worse than one that is honestly unprotected, because the design document says it is covered.

Key takeaways

HA rarely fails; it reacts. Diagnose upward from the fencing event to the quorum loss to the network cause.
Requested states are started, stopped, disabled, ignored. CRM service states include fence (waiting for a node to be confirmed dead), recovery (placing the service), freeze, migrate, relocate and error.
Documented timings: a node whose LRM or CRM is not scheduled reboots after the watchdog times out, which happens after 60 seconds, and typical error detection plus failover is about two minutes.
A watchdog reset is identified by absence: the previous boot ends mid-sentence with no shutdown sequence. journalctl -b -1 is the command.
The trigger is above the fence. Read corosync on every node, and look for retransmits in the preceding week — fencing events usually have warning.
Recover from error in the documented order: make it safe, --state disabled, fix the cause, then --state started. Confirm the guest is really down first.
A fenced node reboots without flushing. Treat it as a power-loss event for its guests, not just an availability event.
Never disable HA to stop fencing. Suspend specific services with ignored, record it, and keep the underlying instability as an open incident.

Knowledge check

Why this matters

The hardest thing about HA incidents is that the system did nothing wrong, and everything visible is the system doing something dramatic.

A node rebooted. A guest moved. A service refuses to start. All three are consequences, all three are loud, and all three point away from the cause. The cause is usually a network that dropped corosync traffic for eleven seconds, which produces no alert, no ticket, and no visible symptom at the time.

That asymmetry is why the retransmit check belongs in your monitoring and in your incident checklist. Corosync tells you weeks in advance that its link is marginal, in a log line nobody reads, and then one night the margin runs out and a node reboots itself to protect your data.

Read the fence as a receipt for a decision made correctly, and go and find what it was reacting to.