Quorum loss recovery

Why this matters in production

Quorum loss is a stressful event. Operators panic. The wrong action (pvecm expected 1) can permanently corrupt the cluster. This lesson teaches the safe path.

Diagnosing quorum loss

pvecm status

The output shows:

• Quorum information: how many votes are needed (quorum value).
• Votequorum information: how many votes are present.
• Nodes’ states: M (member), x (lost), d (dead, not in cluster), N (not member).

If Votequorum: 1 and quorum requires 2, the cluster has lost quorum.

# pvecm status

Quorum information
------------------
Date:             Wed Aug 12 02:41:19 2026
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.2c8
Quorate:          No

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      2
Quorum:           3 Activity blocked
Flags:

Membership information
----------------------
  Nodeid      Votes Name
0x00000001          1 10.0.0.11 (local)
0x00000002          1 10.0.0.12

set -euo pipefail

# 1. Which partition am I in, and how many votes does it hold?
pvecm status

# 2. What can corosync see, per node and per link? This distinguishes
#    a dead node from a dead link.
corosync-cfgtool -n
corosync-cfgtool -s

# 3. When did it happen, and what did corosync say at the time?
journalctl -u corosync --since '1 hour ago' --no-pager | tail -40

# 4. Is the configuration the same everywhere it can be checked?
#    A Config Version mismatch is a cause rather than a symptom.
grep -E 'config_version' /etc/corosync/corosync.conf

# 5. If a QDevice is configured, is it alive and voting?
pvecm status | grep -i qdevice

# 6. Reassurance, and it matters: are the guests still running?
qm list
pct list

Causes of quorum loss

The safe recovery path

Step 1: Identify the cause

Use the cluster network diagnostics from the previous lesson. Do not skip this step.

Step 2: Restore the missing node(s)

If a node is down, fix it:

• Restart the node.
• Fix the network.
• Replace failed hardware.

Wait for the node to rejoin the cluster naturally.

Step 3: If a node is permanently gone

If a node cannot return (hardware destroyed, decommissioned):

pvecm delnode pve-04

Quorum should be restored as long as the remaining nodes have enough votes.

When you must force

In rare cases, the cluster has lost quorum and cannot recover naturally:

• All nodes are alive, but a network partition prevents them from agreeing.
• After a QDevice failure, you must bring the cluster back together manually.

In this case, you can force a node to be “the cluster” temporarily:

pvecm expected 1

After forcing:

1. Restore the missing nodes.
2. Wait for the cluster to reform.
3. Restore the expected votes.

set -euo pipefail

# What does the cluster currently believe, and what should it believe?
pvecm status | grep -E 'Expected votes|Highest expected|Total votes|Quorate'
grep -c '^ *node {' /etc/pve/corosync.conf   # nodes actually configured

# Once every node has returned and the membership is complete, restore the
# expected vote count to match the real node count.
EXPECTED=3
pvecm expected "$EXPECTED"

# Verify, on every node. They must all agree.
pvecm status | grep -E 'Expected votes|Quorate'

The anti-pattern: pvecm expected 1 as a quick fix

Operators sometimes reach for pvecm expected 1 to make a problem go away. This is almost always wrong. The only safe use is:

• All other nodes are truly unreachable (verified via OOB).
• No data is in danger of being written elsewhere.

If you are tempted to use this command, take a breath and verify each remaining node’s state via OOB first.

A break/fix exercise

Production considerations

Common mistakes

• Intervening before diagnosing. Nothing is degrading; the guests are up.
• pvecm expected 1 without OOB verification of every other node.
• Treating unreachable as dead. They look identical and are not.
• Restarting corosync everywhere at once in an attempt to force a reform, risking a fence storm on an HA-armed cluster.
• Removing a node that is alive but partitioned, which removes a working member that never finds out.
• Leaving expected votes lowered after recovery, so the cluster will accept quorum it should not have.
• Restarting pveproxy and pvedaemon because the GUI is returning errors. They are downstream of the problem.
• Failing to communicate, when the honest status — “everything is running, nothing can be changed, we are waiting for a node” — is one people accept easily.

Key takeaways

• An inquorate cluster is frozen, not broken. Diagnose before acting.
• Activity blocked in pvecm status is corosync stating that writes are refused; the membership list tells you which partition you are in.
• The question that governs every safe action is whether the missing node is dead or merely unreachable, and only out-of-band management answers it.
• A QDevice arbitrates and grants its vote to exactly one partition; pvecm expected asserts and consults nothing.
• Restoring the expected-vote count is the second half of the procedure, not cleanup. A lowered count is silently unsafe until corosync restarts.
• A dead QDevice on a two-node cluster produces no symptom at all until the next node failure.

Knowledge check