What /etc/pve does when quorum is lost · Proxmox VE

There is one sentence in this lesson that is worth more than the rest of it:

pmxcfs serves reads without quorum and rejects every write.

Every confusing thing about an inquorate Proxmox cluster is a consequence of that asymmetry. The web interface loads, because loading it is a read. The guest list is populated, because that is a read. The VMs are running, because they were already running and a running QEMU process does not re-read /etc/pve. And then you click Start on a stopped guest, or edit a description, or try to migrate something, and it fails with an error that does not mention quorum at all.

The operator conclusion at that point is almost always “the GUI is broken”. It is not. The cluster is telling you, in the only way it has, that it does not currently have the right to change its own mind about anything.

The rule, stated precisely

/etc/pve is not a directory on a disk. It is a FUSE mount point backed by pmxcfs, which the Proxmox documentation describes as “a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync”.

Replicated in real time is the operative phrase. A write to /etc/pve is not a local file write that is later synchronised; it is a cluster-wide ordered transaction. Ordering a transaction across a set of nodes requires knowing which nodes are in the set, and that is exactly what quorum establishes. With no quorum there is no agreed membership, so there is no way to order a write, so the write is refused.

Reads have no such requirement. The node already holds a complete copy of the configuration in memory. Serving it costs nothing and risks nothing.

Operation	Inquorate node	Why
Reading any file under `/etc/pve`	Works	Served from the local in-memory copy
Web interface loads, guest list populates	Works	Reads only
Already-running VMs and containers keep running	Works	QEMU and LXC do not consult `/etc/pve` after start
Guest disk and network I/O	Works	Nothing in the data path touches pmxcfs
SSH to the node, `journalctl`, `top`, `zpool status`	Works	Not cluster state
Writing any file under `/etc/pve`	Fails	No agreed ordering
`qm set`, `pct set`, creating or destroying a guest	Fails	Writes the guest config file
Starting a stopped guest	Fails	Needs the cluster lock, which is a pmxcfs write
Live migration in either direction	Fails	Writes the config on the target node
Adding or editing storage, users, firewall rules, HA resources	Fails	All of these live in `/etc/pve`
HA recovery of a failed guest	Does not happen	The LRM cannot act without quorum
Backups to PBS	Usually fails	`vzdump` takes a cluster-wide lock

Read-only / Safethe cheapest possible confirmation— Permission denied on a filesystem you are root on is the signature. It is not a permissions problem and no amount of chmod will change it. If this fails, the node is inquorate until proven otherwise.

# touch /etc/pve/quorum-probe

touch: cannot touch '/etc/pve/quorum-probe': Permission denied

Illustrative output

The error message never says the word quorum

This is the whole reason the lesson exists. Every layer above pmxcfs translates the refused write into its own vocabulary, and none of those translations mention quorum:

The GUI shows unable to open file '/etc/pve/nodes/pve-02/qemu-server/100.conf.tmp.1234' - Permission denied (500).
qm start reports cluster not ready - no quorum? (500) — the closest to honest, and even that is phrased as a question.
A backup job fails with a lock timeout.
An API client gets a bare HTTP 500.
pvesm status may show storages as inactive, sending you off to debug storage.

So the first diagnosis is nearly always wrong, and the wrong diagnoses are expensive: restarting pveproxy, restarting pvedaemon, remounting storage, regenerating certificates. None of them help, all of them add moving parts to an incident, and one of them — restarting pve-cluster — can make things worse if corosync.conf on that node is the thing that is broken.

Before touching anything else, run pvecm status and read the Quorate line. It costs two seconds and it decides which of two completely different incidents you are in.

Reading the evidence

Read-only / Safepvecm status on a node that has lost quorum— Three lines carry the diagnosis: Quorate: No, Total votes: 1 against Expected votes: 3, and the phrase Activity blocked next to the quorum figure. The Membership list showing only the local node tells you this partition contains one node.

# pvecm status

Cluster information
-------------------
Name:             prod-cluster
Config Version:   7
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed Aug 12 02:41:19 2026
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2.1a4
Quorate:          No

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      1
Quorum:           2 Activity blocked
Flags:

Membership information
----------------------
  Nodeid      Votes Name
0x00000002          1 10.0.0.12 (local)

Illustrative output

Activity blocked is the string to remember. Corosync prints it when the current partition holds fewer votes than the quorum threshold, and it is the authoritative statement that writes are being refused.

Why the read path and the write path diverge

pmxcfs is a single userspace daemon that does three things at once:

It exposes a FUSE filesystem at /etc/pve.
It keeps the entire configuration tree in memory. The documentation is explicit that “a copy of the data resides in RAM”, which is why there is a hard size ceiling — currently 128 MiB for the whole of /etc/pve.
It persists that tree to a SQLite database at /var/lib/pve-cluster/config.db, and replicates changes to peers through corosync’s closed process group (CPG) messaging.

A read() against the FUSE mount is answered from the in-memory tree. No network, no consensus, no failure mode beyond the daemon being dead.

A write() is submitted to corosync as a CPG message, delivered in the same order to every member of the group, and only then applied to memory and to config.db. That guarantee — every node applies every change in the same order — is what makes /etc/pve a coherent cluster-wide filesystem rather than three directories that usually agree.

When the node is not part of a quorate partition, corosync will not deliver the message, so the write has nowhere to go. pmxcfs returns EPERM rather than queueing it, and returning an error is the correct behaviour: a queued write would be a promise the cluster cannot keep, and applying it locally would be the definition of split-brain.

The 128 MiB ceiling deserves a footnote of its own. It is generous for configuration, and it is not generous if somebody starts using /etc/pve as a convenient replicated share for ISO checksums, scripts or backup logs. That is a real thing operators do, because /etc/pve/ is the only directory on the box that is automatically the same everywhere. Filling it degrades the whole cluster, not one node.

What is still safe to do while you are inquorate

An inquorate cluster is a frozen cluster, not a broken one. The workloads are running. Nothing is corrupting. You have time, and the correct use of that time is diagnosis, not intervention.

Read-only / Safethe inquorate-cluster triage sequence— Every command here is read-only and safe to run on any node in any partition. Run all of it before you consider changing anything.

set -euo pipefail

# 1. Confirm the diagnosis and see which partition you are in.
pvecm status

# 2. Which peers can this node actually see at the corosync layer?
#    -n lists nodes with per-link status; -s shows this node's link health.
corosync-cfgtool -n
corosync-cfgtool -s

# 3. What did corosync say as it happened? The membership change is logged.
journalctl -u corosync --since '30 min ago' --no-pager

# 4. Is pmxcfs itself healthy, or is the daemon the problem rather than quorum?
systemctl status pve-cluster --no-pager

# 5. Are the guests actually still running, whatever the GUI says?
qm list
pct list

Point 5 is worth dwelling on, because it is the reassurance that lets you slow down. qm list reads the config from /etc/pve and the run state from the local system. If it shows your guests as running, they are running, they are serving traffic, and the business impact so far is “no changes can be made” rather than “the service is down”.

What happens to a guest that was mid-migration when quorum was lost?

This is the one case where an inquorate cluster is genuinely messy, and it is worth knowing the shape of it before you meet it.

A live migration writes the guest configuration on the target node and removes it from the source. If quorum is lost between those two steps, the write cannot complete, and the migration task fails with a lock or permission error.

What you can end up looking at:

The guest config still present on the source node, with the guest stopped there. This is the common case and it is recoverable: once quorum returns, start the guest on the source. Nothing was lost, because the memory transfer had not been committed.
A stale lock in the config file — a lock: migrate line that nobody is holding any more. Once the cluster is quorate again, qm unlock $VMID clears it. Do not clear a lock while inquorate and do not clear one while the migration task might still be alive; check ps and the task log first.
Disk state that is ahead of config state on a shared backend. If the target had already begun writing, the shared disk may contain writes the source does not know about. This is why you never start the same guest on two nodes to “see which one works”.

The rule that keeps you out of trouble: fix quorum first, then reconcile guests. Reconciling guests from inside a partition is guessing.

`pmxcfs -l`, and what it actually costs

pmxcfs accepts three options. Only one of them matters operationally:

Option	Documented as
`-d`, `--debug`	“Turn on debug messages”
`-f`, `--foreground`	“Do not daemonize server”
`-l`, `--local`	“Force local mode (ignore corosync.conf, force quorum)”

Read that description carefully, because it is doing more than it looks like. Local mode does not ask for quorum and does not wait for it. It ignores corosync.conf entirely and asserts quorum unilaterally, which makes /etc/pve writable on this node with no coordination with any other node whatsoever.

That is occasionally the only tool that works. There are three situations where it is the right answer:

corosync.conf itself is broken. It lives inside /etc/pve, which is read-only without quorum, which requires a working corosync.conf. That circular dependency is exactly what local mode exists to break.
A single surviving node after a genuine disaster, where the others are verifiably destroyed and you need to edit configuration to bring services back.
Recovering configuration from a node you are about to reinstall.

And there is one situation where it destroys data: two nodes in local mode at the same time, on either side of a network partition. Both are writable, both accept changes, and neither knows about the other. There is no merge algorithm in pmxcfs. When corosync comes back, one version of the truth wins and the other is gone.

Cluster-wide risklast resort: bring pmxcfs up in local mode on ONE node— Cluster-wide risk. Only after every other node is verified powered off or fenced via out-of-band management - not pinged, not assumed. Doing this on two nodes of a partitioned cluster produces divergent configuration that cannot be merged.

set -euo pipefail

# PRECONDITION, and it is not optional: every other node is confirmed OFF
# through iDRAC / iLO / IPMI, or physically. A node you cannot reach is not
# a node that is off.

systemctl stop pve-cluster

# Foreground and debug, so you can see what it is doing and stop it with
# Ctrl-C rather than leaving a daemon in local mode by accident.
pmxcfs -l -f -d

# In a second session: /etc/pve is now writable on this node only.
# Make the single change you came here to make - typically repairing
# corosync.conf - and nothing else.

# Then stop the foreground process (Ctrl-C) and return to normal operation:
systemctl start pve-cluster
pvecm status

Local mode and pvecm expected are not the same tool and not interchangeable

Both make an inquorate node writable. They do it at different layers and they fail differently, and conflating them is how people reach for the wrong one.

pvecm expected 1 lowers the expected vote count inside the running corosync instance. Corosync is still running, still listening, still evaluating membership — so if the partitioned nodes reappear, they form a membership with a node that has been told the cluster is one node large, and every side believes it is authoritative.

pmxcfs -l bypasses corosync entirely for the filesystem layer. Corosync may not even be running. There is no membership evaluation at all.

Neither is a first response to an outage, and the technical-accuracy standard for this course names pvecm expected 1-as-first-response specifically as an anti-pattern. The first response is pvecm status, then finding out why the votes are missing, then restoring the missing nodes. These two commands are what you use when you have already established that the missing nodes are never coming back.

Verifying that you are actually recovered

A green GUI is not verification. The GUI was green throughout the outage. Verification means demonstrating a write, and demonstrating it on the node you care about.

Read-only / Safeprove the cluster is writable again— Run this on every node, not just the one you were working on. A cluster where two of three nodes can write and one cannot is a partition you have not finished diagnosing.

set -euo pipefail

# 1. Quorate, with the vote count you expect for a healthy cluster.
pvecm status | grep -E 'Quorate|Total votes|Expected votes'

# 2. Every node present and every link up.
corosync-cfgtool -n

# 3. The actual test: a real write to /etc/pve, then clean up after it.
#    A temporary file in the cluster root is replicated like anything else.
PROBE=/etc/pve/.quorum-probe-$(hostname -s)
date -Is > "$PROBE"
cat "$PROBE"
rm -f "$PROBE"

# 4. Confirm the write reached a peer. Run this on a DIFFERENT node while
#    the probe file still exists to see replication rather than local state.

# 5. If HA is configured, the manager should be back to a normal state
#    rather than sitting in a wait or error state.
ha-manager status

Step 3 is the one that can fail, which is what makes it verification. If date -Is > "$PROBE" returns Permission denied, you are still inquorate regardless of what any dashboard says.

Common mistakes

Restarting services to fix a read-only /etc/pve. pveproxy, pvedaemon and pvestatd are downstream of the problem. Restarting them changes nothing and lengthens the incident.
Concluding the storage is broken because pvesm status shows inactive storages. Storage definitions live in /etc/pve/storage.cfg; an inquorate node can still read them, but pvestatd updates and some activation paths need a writable cluster.
Running pmxcfs -l on more than one node. There is no recovery from this other than choosing a winner and discarding the other node’s changes.
Treating “the VMs are still up” as evidence that nothing is wrong. It is evidence that nothing is wrong yet. Nothing will fail over, nothing will restart, and no backup will run until quorum returns.
Using /etc/pve as a general-purpose replicated share. The 128 MiB ceiling is for the entire cluster configuration, and exhausting it is a cluster-wide fault.

Key takeaways

Reads succeed without quorum; writes are refused. Everything else follows.
The error messages never say “quorum”. pvecm status and the Activity blocked string are the fast diagnosis.
Running guests are unaffected, which buys you time to diagnose properly instead of intervening.
pmxcfs -l ignores corosync.conf and forces quorum on one node. It is for a broken corosync.conf or a genuinely dead cluster, never for an active partition.
Verify recovery with a write that can fail, on every node.

Knowledge check

Why this matters

Quorum loss is not a rare event. A switch reboots, a bond flaps, somebody adds a firewall rule, a backup saturates the link corosync shares with everything else, and one node or all of them lose the ability to agree.

What makes it expensive is not the outage — the guests keep running — but the diagnosis. The symptom set points away from the cause: the GUI is up, the error says permission denied, the storage looks inactive. An operator who has not internalised reads succeed, writes fail will spend twenty minutes restarting daemons and remounting filesystems on a cluster that was never broken in any of those ways, and the genuinely dangerous move — forcing quorum on a node while its peers are alive on the other side of a partition — starts to look attractive somewhere around minute thirty.

Knowing the rule turns a thirty-minute panic into a two-second pvecm status and a decision about which incident you are actually in.