When HA cannot save you · Proxmox VE

Why this matters in production

HA is not magic. There are concrete scenarios where HA fails to recover a workload, and the operator needs to know what to do instead. This lesson enumerates them.

Failure mode 1: No capacity for failover

If a node fails and the surviving nodes cannot host the failed node’s VMs (because of RAM, vCPU, or storage limits), HA places the VMs in error state.

flowchart LR
  A[Node A fails] --> B[CRM tries to recover VMs]
  B --> C[Survivors have capacity?]
  C -->|yes| D[VMs restart on survivors]
  C -->|no| E[VMs in error state]

The arithmetic, so it is not a feeling

“We have headroom” is a belief until someone computes it. The computation is short, and the number it produces is the one that belongs in the capacity review.

Read-only / Safecan the cluster survive losing its largest node— Read-only. Sums configured memory per node and checks whether the remaining nodes could hold the largest node's guests. Configured memory is the right input here because that is what HA must be able to allocate.

set -euo pipefail

TOTAL_ALLOC=0
declare -A NODE_ALLOC

for NODE_DIR in /etc/pve/nodes/*; do
NODE=$(basename "$NODE_DIR")
SUM=0
for CONF in "$NODE_DIR"/qemu-server/*.conf "$NODE_DIR"/lxc/*.conf; do
  [ -e "$CONF" ] || continue
  MEM=$(awk '/^memory:/{print $2; exit}' "$CONF")
  SUM=$(( SUM + ${MEM:-512} ))
done
NODE_ALLOC["$NODE"]=$SUM
TOTAL_ALLOC=$(( TOTAL_ALLOC + SUM ))
printf '%-12s %6d MiB allocated to guests\n' "$NODE" "$SUM"
done

# Physical memory per node, for comparison.
pvesh get /cluster/resources --type node --output-format yaml \
| grep -E 'node:|maxmem:|mem:'

printf '\ntotal allocated across cluster: %d MiB\n' "$TOTAL_ALLOC"
echo 'Headroom test: (sum of all nodes physical) - (largest node physical)'
echo '               must exceed TOTAL_ALLOC, with room for the host itself.'

Three refinements that turn the number from arithmetic into a plan:

Use configured, not used, memory. A VM with memory: 32768 using 4 GiB still needs 32 GiB allocatable on the target unless ballooning is in play. HA places on what it must allocate.
Subtract the host’s own footprint. Ceph OSDs in particular are not small; a node running OSDs has materially less memory available for guests than its physical total.
Do it for the largest node, not the average. N+1 means surviving the loss of whichever node hurts most, and that is rarely the one you pictured.

The same reasoning applies to CPU, and to storage IOPS. Memory is the one that produces a hard failure - error state, guest does not start - which is why it is the one to compute first.

Failure mode 2: Shared storage is gone

HA requires the VM’s disk to be reachable from a surviving node. If shared storage is unreachable, HA cannot restart the VM.

Scenario	HA behaviour
NFS server down	HA cannot start VMs
Ceph cluster down	HA cannot start VMs
iSCSI SAN down	HA cannot start VMs

Recovery: restore the storage first, then manually start the VMs.

Failure mode 3: Network partition

If a network partition isolates the failed node from the survivors AND the QDevice (if any), the CRM may not be able to fence the node. HA waits.

Recovery requires restoring network connectivity. No automatic recovery.

What happens if the partition splits a six-node cluster three-and-three?

Neither half is quorate, and neither half does anything. That is the correct outcome and it is worth understanding why it feels wrong.

Quorum requires a strict majority. Three of six is not a majority, so both partitions lose quorum simultaneously. In each half:

/etc/pve goes read-only, so no configuration can change.
Every LRM loses its ability to renew its agent lock and stops its HA resources.
No CRM can acquire the manager lock, so nothing is recovered anywhere.

Six nodes are up, healthy, with working storage, and every HA guest is stopped. The instinct in the room is that the cluster has failed catastrophically and that pvecm expected 3 on one side would fix it.

It would, and it would also mean that if the other side does the same - or if the partition heals a moment later - two clusters have each been running the same guests against the same storage. That is the outcome the whole design exists to prevent, and both halves are equally entitled to believe they are the survivors.

The real answers, in order:

Fix the network. The partition is the fault; quorum is doing its job.
Have an odd number of votes. An even-sized cluster can split evenly; an odd one cannot. This is why a QDevice matters on even-node clusters as much as on two-node ones - see quorum and QDevice.
If you must act before the network is fixed, act on exactly one side and only after physically confirming the other side is powered off. The full treatment, including why pvecm expected is a last resort rather than a first response, is in quorum recovery.

Failure mode 4: HA itself fails

The HA manager (CRM) runs on a single node. If that node has a software issue, HA becomes unavailable. The next node’s CRM takes over after a timeout.

Failure mode 5: Cascading failures

If a node fails and the additional load on survivors causes them to fail too, HA cannot recover. This is the worst-case scenario.

flowchart LR
  A[Node A fails] --> B[Survivors overloaded]
  B --> C[Survivor B fails]
  C --> D[Survivor C overwhelmed]
  D --> E[Cascade]

Prevention: keep cluster utilisation well below the threshold that triggers a cascade. N+1 headroom that includes the failed node’s full workload is mandatory.

Failure mode 6: HA rules prevent recovery

Strict HA rules can prevent recovery. If a VM’s strict rule lists only one node, and that node fails, the VM enters error state.

Recovery: relax the rule, manually move the VM, or rebuild.

Failure mode 7: Watchdog misconfiguration

If the watchdog is not configured (or misconfigured), self-fencing may not happen. A wedged node may come back online and confuse the cluster.

Verify the watchdog:

ls -la /dev/watchdog && cat /etc/default/pve-ha-manager

Note what that check does not rely on: systemctl status watchdog-mux reports active (running) on a node with no watchdog device at all, which is why the device test is the one that can fail. The full incident is worked through in anatomy of a bad fence.

Failure mode 8: everything HA was never for

Worth stating separately because it is the largest category by incident count, and because HA’s name invites the confusion.

Event	Does HA help?	What does
Someone deletes a database table	No	Backup with retention
Ransomware encrypts the guests	No	Immutable offsite backup
A bad deploy breaks the application	No	Application rollback
Guest filesystem corruption	No - it restarts the corrupt guest	Restore
A guest OOMs repeatedly	No - it restarts it, repeatedly	Right-sizing
Site loss	No	DR
Certificate expiry across the estate	No	Monitoring
A single node’s hardware dies	Yes	This is the one

HA addresses exactly one failure: a host stops. Everything else on that list is someone else’s job, and an estate that has bought HA and believes it has bought resilience is one deletion away from finding out.

Why HA restarting a broken guest makes things worse, and what to do about it

There is a failure shape that HA turns from an outage into a slow-motion incident: a guest that starts, runs briefly, and dies.

HA sees the guest stop, and its job is to make requested state match actual state. So it restarts it. The guest dies again. HA restarts it - up to max_restart times on that node, then relocates it up to max_relocate times

and each attempt takes storage bandwidth, CPU, and time, on a cluster that may already be busy for the reason the guest is dying.

Meanwhile monitoring sees a service that is up, then down, then up. Alerting that requires two consecutive failures never fires. The service is unusable throughout and no page is generated.

Eventually the counters exhaust and the resource lands in error, which is the system working correctly: the CRM has stopped and asked for a person. But by then several minutes of thrash have happened.

Two things reduce it:

Keep max_restart and max_relocate low - their defaults of 1 each are right for most workloads. Raising them lengthens the thrash without changing the outcome.
Alert on the HA state, not only on the service. error is a discrete, unambiguous condition and a far better alert source than a flapping port check:

ha-manager status | awk '/^service/ && /error/ {print; found=1} END {exit !found}'

That exits non-zero when nothing is in error, so it inverts cleanly into a monitoring check that fires only on the real condition.

Production considerations

Common mistakes

Believing HA is “automatic” and never testing.
Sizing at >80% utilisation.
Forgetting the watchdog configuration.

Key takeaways

HA fails when survivors lack capacity.
HA fails when shared storage is unreachable.
HA fails when the watchdog is misconfigured.
Plan runbooks for these scenarios.

Knowledge check

Why this matters

The value of enumerating what HA cannot do is not pessimism. It is that every item on the list is a gap somebody has to own, and gaps that are not named do not get owned.

Two of them are worth more than the rest. Capacity is the failure that turns HA from a safety net into a decoration, and it is invisible because a cluster at 92% utilisation looks exactly like one at 55% until a node dies. It is also the only item here that is pure arithmetic - twenty minutes with the script on this page and you know, rather than believe.

Shared storage is the other, and it is worse because the failure is correlated: the same network event that partitions a node frequently degrades storage for everyone, so HA is asked to recover in the worst conditions the cluster has seen that week rather than in test conditions.

Everything else on the list resolves the same way. Write the failure, write the mechanism that covers it, write when it was last tested. The rows with no mechanism are the honest output of the exercise, and they are worth more than the rows that are already fine.