high riskcluster affecting~30 min
Recover quorum after losing a single node
1 · Prerequisites
Confirm every item is in place before any state change.
- A 3-node PVE cluster that has lost one node permanently
- Corosync link still working between the two surviving nodes
- QDevice configured (recommended but optional)
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · pvecm status reports the missing node as "node is offline"
- · The two survivors can reach each other on the corosync link
- · No network partition between survivors
- · Verify the missing node really is gone: ping, IPMI, physical check
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Confirm the missing node is permanently offline (hardware failure, not network glitch)
- 2On a surviving node, remove the dead node from corosync: pvecm delnode <hostname>
- 3Verify quorum restored: pvecm status should show 2 nodes, 2 votes, quorate
- 4Re-add the replacement node to the cluster (see add-node-to-cluster runbook)
- 5Run ha-manager status to confirm HA resources recovered
- 6Check that any VMs that were on the missing node have been recovered by HA on a survivor
- 7Update DNS, monitoring, and documentation to reflect the new cluster topology
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓pvecm status shows 2 active nodes, quorate
- ✓All running VMs are accounted for
- ✓No HA errors in journalctl -u corosync --since "1 hour ago"
- ✓New node joins and rebalances without issue
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If the new node fails to join, do NOT lower expected votes to 1; investigate why the join fails
- ↶If HA cannot recover resources, manually migrate VMs: qm migrate <vmid> <target-node>
6 · Escalation
When the runbook isn't enough, contact:
- · Cluster admin if quorum cannot be restored
- · Network admin if corosync link is unreachable
Recover quorum after losing a single node
When a node dies permanently, the cluster keeps running with reduced vote count. The cluster may or may not be quorate depending on configuration. This runbook covers the recovery path.
When this runbook applies
- A node has been offline for longer than corosync’s token timeout
- The cluster is degraded but not split-brain
- You have determined the node will not come back
Do not use this runbook for transient network failures — those typically self-heal once the link is restored.