HA architecture: CRM, LRM, watchdog, fencing · Proxmox VE

Why this matters in production

HA in Proxmox is not magic. It is a state machine implemented by two daemons plus a watchdog. Understanding the state machine is essential for diagnosing HA incidents.

The two daemons

Daemon	Role	Where it runs
`pve-ha-crm`	Cluster Resource Manager — makes cluster-wide decisions	One node at a time (master)
`pve-ha-lrm`	Local Resource Manager — executes actions on the local node	Every node

sequenceDiagram
  participant Master as CRM (master node)
  participant LRM1 as LRM (node 1)
  participant LRM2 as LRM (node 2)
  participant LRM3 as LRM (node 3)
  Master->>LRM1: start vm:100
  LRM1-->>Master: started
  Note over Master,LRM2: Node 1 fails
  Master->>LRM1: stop vm:100 (no reply)
  Master->>LRM1: fence node 1
  Master->>LRM2: start vm:100
  LRM2-->>Master: started

The CRM (Cluster Resource Manager)

The CRM:

Runs on one node at a time (the master, holding the manager lock).
Tracks the desired state of every HA-managed resource.
Sends commands to LRMs.
Decides which node should run which resource.
Manages fencing.

The CRM uses pmxcfs for distributed locking. The master lock is held by one CRM at a time; if the master fails, another CRM takes over.

The LRM (Local Resource Manager)

The LRM runs on every node. It:

Reads the requested state for its resources from manager_status.
Executes the corresponding qm or pct commands.
Reports results back to the CRM.

LRMs also hold an “agent lock” in pmxcfs. If the lock is lost, the LRM shuts down its resources (to prevent split-brain).

Fencing — preventing split-brain

Fencing ensures that a failed node can never come back online and run the same VMs as a surviving node. Without fencing, both nodes might believe they own the resource.

Proxmox uses self-fencing via watchdog:

The CRM detects that a node has not sent heartbeats.
The CRM waits for the agent lock on that node to be released (a healthy LRM releases the lock when it loses quorum).
If the lock is not released, the CRM assumes the node is wedged.
The watchdog on the wedged node triggers a reboot.

The watchdog is hardware-based if available (IPMI watchdog, iTCO_wdt, etc.) or software-based (Linux softdog) as a fallback.

cat /etc/default/pve-ha-manager && ls -la /dev/watchdog

ha-manager status

The agent lock, and why the CRM never asks whether a node is alive

There is an obvious-looking design the CRM deliberately does not use: before recovering guests, ping the failed node or query its IPMI and confirm it is down. Understanding why Proxmox avoids that explains the whole shape of the mechanism, including its timings.

Any such check answers “was that node reachable from here, just now”. None of them answers “is that node writing to the shared storage”. A node can be unreachable from the CRM and perfectly able to reach its disks by another path - which is exactly the network-partition case, and exactly the case where a wrong answer destroys data.

So the CRM asks a question that has a safe answer instead. Every LRM holds an agent lock in pmxcfs, on a lease. Holding that lock requires quorum. That gives three branches and one invariant:

A node with quorum renews its lock and keeps running its guests.
A node that loses quorum cannot renew it. Its LRM notices, stops its guests, and releases the lock cooperatively. This is the fast path.
A node that is wedged does nothing at all, including not renewing. The lease expires and the CRM acquires the lock.

In every branch, “the CRM holds the agent lock” implies “no LRM on that node is in a position to be running guests”. That is a statement about the distributed state machine, not about network reachability, and it stays true regardless of what path the failed node might still have to storage.

The watchdog closes the last gap. Without it, a node could be wedged long enough for its lease to expire and then resume - waking up believing it still owns guests the CRM has already restarted elsewhere. The watchdog guarantees that a node which has not renewed within the timeout has already reset.

Two consequences follow, and both are load-bearing:

The watchdog timer must be shorter than the lock timeout. That ordering is the safety property. Reverse it and the cluster can start a guest on a survivor while the original node is still running it.

A missing watchdog does not merely slow recovery. It removes the guarantee the design rests on, which is why anatomy of a bad fence treats an absent /dev/watchdog as the more serious of its two incidents.

The recovery delay is a feature, and shortening it is how you get two writers

Between “the node stopped answering” and “the guest starts elsewhere” there is a wait of roughly a minute. It is long enough that people ask whether it can be tuned down, usually while quoting an RTO at someone.

What fills it is the lock lease, and the lease must be longer than the worst-case interval in which a node could be alive but silent: a long garbage collection, a storage stall, a network reconvergence, an overloaded scheduler. It must also be longer than the watchdog timer, for the reason above.

The delay is not inefficiency. It is the interval during which the cluster establishes a fact it is about to bet your data on. If your RTO cannot absorb it, the answer is a faster-booting guest or a warm standby - not a shorter fencing window.

Recovery sequence

sequenceDiagram
  participant N1 as Failed node
  participant N2 as Surviving node
  participant W as Watchdog
  Note over N1: Node stops responding
  N2->>N2: CRM waits for LRM lock release
  N2->>W: Lock not released, trigger watchdog on N1
  W->>N1: Reboot
  N1->>N1: Booting...
  Note over N1: Node loses lock, resources stopped
  N2->>N2: CRM selects new host for affected VMs
  N2->>N2: LRM starts VMs on new host

Interactive HA simulator

Interactive · HA Behaviour Simulator

Click a node to mark it as failed. Watch how the cluster resource manager (CRM) steals the HA-managed VMs and restarts them on a surviving node. Note the watchdog state — if quorum is lost, the watchdog triggers a self-reset on the failed node.

Cluster nodes

HA-managed VMs (current placement)

VMID	Name	Preferred	Current node	State
101	web-01	pve-01	pve-01	started
102	db-01	pve-02	pve-02	started
103	app-01	pve-03	pve-03	started
104	util-01	pve-01	—	stopped

Cluster Resource Manager

CRM master: pve-01
Watchdog: armed
Quorate: yes

All nodes healthy. HA is idle; the CRM watchdog is armed but not triggering.

The simulator shows what happens to VMs as nodes fail and the CRM/LRM react. Note:

The CRM “steals” services from failed nodes.
VMs restart on survivors.
The watchdog state changes from armed to disarming when a node fails.

Watchdog configuration

Configuration change— Configure the hardware watchdog.

cat > /etc/default/pve-ha-manager <<EOF
WATCHDOG_MODULE=iTCO_wdt
EOF
systemctl restart watchdog-mux

Server boards commonly expose an IPMI watchdog; Intel chipsets expose iTCO_wdt. Confirm which module your hardware actually provides before setting the value - naming a module that does not load leaves the node with no watchdog at all, and every service-level health check still passes.

Read-only / Safefind out which watchdog this hardware has, before configuring one— Read-only. Run it on each node type in the estate; the answer differs by server generation, which is how a runbook written for the previous generation leaves a cluster unfenceable.

set -euo pipefail

# What watchdog drivers does this kernel offer for this hardware?
ls /sys/class/watchdog/ 2>/dev/null || echo 'no watchdog class entries'
for W in /sys/class/watchdog/watchdog*; do
[ -e "$W" ] || continue
printf '%s: ' "$(basename "$W")"
cat "$W/identity" 2>/dev/null || echo unknown
done

# What is loaded now, and what is configured?
lsmod | grep -iE 'wdt|softdog|watchdog' || echo 'no watchdog module loaded'
cat /etc/default/pve-ha-manager
systemctl is-active watchdog-mux

watchdog-mux is running and there is no watchdog

systemctl status watchdog-mux reports active (running) on a node with no watchdog device at all. The service starts, finds nothing to multiplex, and sits there. Every service-level check passes; every dashboard is green; the node cannot self-fence and nothing says so.

This is the single most common latent HA defect, because the check everyone writes is the one that cannot detect it.

The check that can fail:

test -c /dev/watchdog \
  && echo 'watchdog present' \
  || echo 'NO WATCHDOG DEVICE - THIS NODE CANNOT SELF-FENCE'

Put that line in node health monitoring on every cluster you run. Its cost is one line; the cost of not having it is an unexplained multi-minute outage when a node wedges, followed by a post-mortem that blames whatever driver hung and leaves the real defect in place.

What happens if the CRM master node itself fails?

Another node’s CRM takes over, and the transition is one of the better-designed parts of the stack - but there is a window, and knowing its shape stops you misreading a normal event as an incident.

The master is whichever CRM holds the manager lock in pmxcfs. It is not a configured role and there is no election to configure. When the master’s node fails, its lock lease expires and another CRM acquires it, at which point that node begins making cluster-wide decisions.

Three things follow:

Recovery is delayed by roughly one lease interval, on top of the fencing wait, because the new master has to acquire the lock before it can act on anything. A node failure that takes out the master is measurably slower to recover from than one that does not.
The LRMs kept running throughout. Guests on healthy nodes are not affected by a master change; the LRMs continue executing the last state they were given. A master transition is not a service interruption.
A master that moves repeatedly is a symptom, not a feature. It means the lock is being lost, which means quorum or pmxcfs is unstable, which is a corosync problem. Chase it in corosync link tuning, not in the HA configuration.

ha-manager status | grep -i master
journalctl -u pve-ha-crm --since '24 hours ago' --no-pager \
  | grep -ci 'manager status update\|status change' || true

Production considerations

Common mistakes

Not configuring a hardware watchdog.
Running clusters at >80% utilisation (no HA failover room).
Misconfigured shared storage (HA cannot restart VMs if storage is gone).

Key takeaways

CRM makes cluster-wide decisions; LRM executes locally.
Watchdog prevents split-brain via self-fencing.
Capacity headroom is mandatory.

Knowledge check

Why this matters

Most operators meet HA as a checkbox and a set of commands. What this lesson adds is the reason the mechanism is shaped the way it is, and that reason turns out to be the answer to nearly every “can we make this faster” conversation you will have about it.

The stack is built around one hard problem: deciding, from a node that cannot talk to another node, whether it is safe to start that node’s guests. There is no reliable direct answer to that question, so Proxmox replaces it with a question that does have one - who holds the agent lock - and then uses the watchdog to make the lock’s meaning airtight.

Once you hold that, the rest stops looking arbitrary. The minute of delay is the lease. The requirement for a hardware watchdog is the difference between a guarantee and a hope. The insistence on shared storage is what makes recovery possible at all. And the reason you must never shorten the CRM’s wait below the watchdog timer is that it converts a system designed around certainty into one that guesses - about a decision whose wrong answer is two writers on one disk.