Skip to main content
RunBook Academy

← All break/fix scenarios in Proxmox VE

advancedCluster issues~20 min

Cluster node is unreachable but no HA recovery happens

Reported symptoms

  • ●A cluster node is unreachable from the GUI
  • ●HA VMs running on that node are still running (in theory)
  • ●Other nodes do NOT recover the VMs
  • ●pvecm status on a remaining node shows the missing node

Evidence

  • · pvecm status (run from a remaining node)
  • · pvesh get /cluster/resources --type vm shows the VMs still listed as on the missing node
  • · ha-manager status shows the affected VMs in "started" state on the missing node
  • · corosync on remaining nodes has lost quorum (but did not lose it)
Diagnosis and resolutionclick to reveal

Root cause

The missing node has not been fenced. The HA manager requires fencing confirmation before recovering a VM, to avoid split-brain where the "missing" node is actually still serving traffic and the "recovery" node starts a duplicate instance. Without a working fence mechanism (IPMI, iLO, iDRAC, or watchdog), the HA manager stays in a recovery-blocked state.

Remediation

1. Confirm the node is truly unreachable (not just network glitch): `ping -c 3 <missing-node-ip>` from multiple cluster nodes `ssh root@<missing-node>` (try directly) 2. Check the HA status: `ha-manager status` Look for VMs in "fence", "recovery", or "freeze" state 3. If the node is truly down and you must recover, manually fence: `ha-manager fence <node>` (if a fence mechanism is configured) 4. If no fence is available, manually relocate the VMs: - For each affected VM, from a working node: `qm migrate <vmid> <target-node> --with-local-disks` - Or: `ha-manager relocate vm:<vmid> --node <target>` 5. Once the missing node is confirmed dead, remove from cluster: `pvecm delnode <missing-node>` (on a remaining node) 6. Restore quorum: cluster should stabilise with 2 of 3 (or use qdevice if you have one).

Verification

- All VMs are running on reachable nodes - Cluster quorum is restored - No "started" state on the missing node - All VMs respond to ping / SSH

Prevention

- Always configure fencing (IPMI, iLO, iDRAC) for production clusters - Use qdevice for 2-node setups - Document the manual fencing procedure - Test fence behaviour in a lab quarterly