critical riskcluster affecting~120 min

Recover a cluster from a corrupt /etc/pve (pmxcfs)

1 · Prerequisites

Confirm every item is in place before any state change.

Root shell access to the affected node, ideally via out-of-band console rather than SSH
It is established how many nodes are affected - one node, or the whole cluster - because the recovery differs completely
The state of quorum on the unaffected nodes is known
Free disk space on / on the affected node, verified, because a full root filesystem is a common cause and makes every repair fail
A recent copy of the cluster configuration exists somewhere outside the cluster, or its absence is acknowledged as a constraint on this recovery
Guest workload state is known: which guests are running, on which nodes, and whether they are currently serving

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· systemctl status pve-cluster shows whether the service is running, failed, or restart-looping
· mount | grep /etc/pve shows whether the FUSE filesystem is mounted at all
· ls /etc/pve returns either the expected tree, an empty directory, or an I/O error - and which one it is decides the path
· df -h / shows free space; a full root filesystem must be cleared before any repair is attempted
· journalctl -u pve-cluster -b names the actual failure, and database disk image is malformed is a different fault from a mount failure
· pvecm status on an unaffected node reports whether the rest of the cluster is quorate
· Running guests are confirmed still running - pmxcfs being down does not stop a running VM

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Do not restart services blindly. Read the journal first and classify the fault: not mounted, mounted but read-only, or backing database corrupt
2Confirm running guests are unaffected, and stop anything that would try to start or migrate a guest during the repair
3Take a copy of the backing database and its write-ahead log before touching either
4Establish whether the rest of the cluster is quorate and holds a good copy of the configuration
5Run an integrity check against the copied database, not the live one
6If the cluster is healthy and only this node is damaged: stop pve-cluster, move the local database aside, and let the node resync from the cluster
7If every node is damaged: restore config.db from backup on one node, start it, and resync the others from it
8If no backup exists: start pmxcfs in local mode on the least damaged node, extract what can be read, and rebuild the configuration deliberately
9Verify the recovered configuration against the known guest inventory before allowing anything to start guests
10Restart the cluster stack in order: corosync, then pve-cluster, then the PVE daemons
11Re-verify quorum, storage, HA and guest configuration counts on every node
12Record what was lost between the last good state and the recovery

4 · Verification

Confirm the procedure actually fixed the problem.

✓systemctl is-active pve-cluster returns active on every node, with no restart loop in journalctl
✓mount shows /etc/pve mounted, and touch /etc/pve/.rwtest succeeds then is removed - proving it is writable, not merely present
✓pvecm status reports Quorate: Yes with expected votes equal to the node count
✓The guest config count in /etc/pve/nodes matches the inventory recorded before the incident, node by node
✓pvesm status lists every storage as active on every node
✓ha-manager status runs without error and shows services on known nodes
✓A trivial configuration write succeeds and is visible on a second node within seconds, proving replication works
✓sqlite3 on the live config.db reports integrity_check ok on every node

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The copy of config.db and config.db-wal taken before any change is the rollback. Without it there is none
↶Moving the local config.db aside is safe while the cluster is quorate, because the cluster holds the authoritative copy - and is catastrophic if it is not, because that node may hold the last copy
↶Writes made while pmxcfs is in local mode are NOT replicated. If the node later rejoins, those writes may be discarded or may conflict - local mode is an extraction tool, not a way to keep working
↶Restoring config.db from backup reverts the entire cluster configuration to the backup time: guests created since then disappear from the configuration, though their disks remain on storage
↶If a repair attempt makes things worse, stop and restore the copied database rather than trying a second repair on top of the first
↶Never run a SQLite repair against the live database. Work on a copy and swap it in

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if every node is affected - that is a cluster-wide configuration loss, not a node repair
· Escalate before restoring config.db from a backup, because it reverts cluster-wide configuration and someone must own that decision
· Escalate before starting pmxcfs in local mode on a node that is still part of a live cluster
· Escalate to the storage owner if guest disks exist on storage but no configuration references them after recovery
· Escalate to Proxmox support if the SQLite database is corrupt on more than one node simultaneously, because that usually indicates a shared underlying fault such as storage or memory

Verified against Proxmox VE 9.2.4.

/etc/pve is not a directory. It is a FUSE filesystem served by pve-cluster, backed by a SQLite database at /var/lib/pve-cluster/config.db, and replicated between nodes over corosync. Everything that defines the cluster lives there: guest configs, storage definitions, HA rules, user accounts, the firewall.

When it breaks, the symptom is alarming and the immediate impact is usually smaller than it looks. Running guests keep running. QEMU does not consult /etc/pve to keep serving. What stops is everything administrative: starting, stopping, migrating, backing up, HA acting, the GUI being useful.

That gap - loud symptom, contained impact - is the space in which you get to be careful. Use it.

When to use this runbook

/etc/pve is empty, or listing it returns an I/O error.
pve-cluster fails to start or restart-loops.
The journal reports database disk image is malformed or a SQLite error.
/etc/pve is mounted but read-only, and the cause is not simply lost quorum.

If /etc/pve is read-only and the node is inquorate, that is not corruption - that is pmxcfs working exactly as designed. Restore quorum and it becomes writable again. See recover-quorum-after-node-loss.

Classify the fault before touching anything

Read-only / Safewhat is actually wrong

systemctl status pve-cluster --no-pager
journalctl -u pve-cluster -b --no-pager | tail -60

mount | grep '/etc/pve' || echo 'pmxcfs is NOT mounted'
ls -la /etc/pve 2>&1 | head
df -h /
pvecm status 2>&1

What you see	What it means	Where to go
`/etc/pve` read-only, `pvecm status` not quorate	Normal behaviour, not corruption	Restore quorum. Stop here
`pve-cluster` inactive, mount missing, journal shows a clean stop	Service was stopped	Start it and read the journal
Journal: `database disk image is malformed`	SQLite corruption on this node	Step 2 onwards
Journal: `unable to create fuse mount`	FUSE or a stale mount	Unmount the stale mount, then start
`/etc/pve` present but empty, node recently joined	Never synced	Rejoin from clean
Root filesystem 100% full	Everything downstream of this fails	Clear space first. Nothing else will work
Corruption on more than one node	A shared cause: storage, RAM, power	Stop. Escalate. Do not repair in parallel

Step 1: Confirm the guests are still fine

Read-only / Saferunning guests do not depend on /etc/pve

ps -eo pid,cmd | grep -E '[k]vm -id' | awk '{print $1, $4}'
pgrep -a lxc-start | head

# Anything that would try to change guest state during the repair
systemctl status pve-ha-lrm pve-ha-crm --no-pager | grep -E 'Active|Loaded'

If HA is active, consider putting the node into maintenance so the CRM does not try to relocate services onto or away from a node whose cluster filesystem is broken. If the node is already inquorate, HA has already stopped acting on it.

Step 2: Copy the database before anything else

Read-only / Safepreserve the evidence and the rollback

STAMP=$(date +%Y%m%d-%H%M%S)
DEST="/root/pmxcfs-rescue-$STAMP"
mkdir -p "$DEST"

systemctl stop pve-cluster
sleep 3

cp -a /var/lib/pve-cluster/config.db "$DEST/" 2>/dev/null
cp -a /var/lib/pve-cluster/config.db-wal "$DEST/" 2>/dev/null
cp -a /var/lib/pve-cluster/config.db-shm "$DEST/" 2>/dev/null

ls -la "$DEST"
sha256sum "$DEST"/config.db*

Copy the write-ahead log too. A config.db copied without its -wal is missing the most recent transactions, which is often exactly the window you care about.

Step 3: Check integrity on the copy

Read-only / Safetest the copy, never the original

STAMP=20260812-0300
DEST="/root/pmxcfs-rescue-$STAMP"

apt install -y sqlite3
sqlite3 "$DEST/config.db" 'PRAGMA integrity_check;'
sqlite3 "$DEST/config.db" 'SELECT count(*) FROM tree;'
sqlite3 "$DEST/config.db" 'SELECT name FROM tree WHERE parent = 0 LIMIT 20;'

ok means the database is fine and the fault is elsewhere - the mount, FUSE, permissions, or disk space. Anything else confirms corruption and points you at Step 4 or Step 5.

Read-only / Safeextract what is readable, even from a damaged database

STAMP=20260812-0300
DEST="/root/pmxcfs-rescue-$STAMP"

sqlite3 "$DEST/config.db" .dump > "$DEST/config-dump.sql" 2> "$DEST/dump-errors.txt"
wc -l "$DEST/config-dump.sql"
head -5 "$DEST/dump-errors.txt"

grep -c 'qemu-server' "$DEST/config-dump.sql"

Even a .dump that errors partway usually yields most of the tree. That output is how you reconstruct guest configs if every other path fails.

Step 4: One node damaged, cluster healthy - resync

This is the good case, and it is the common one. The cluster holds the authoritative configuration; this node’s local copy is disposable.

Read-only / Safeconfirm the cluster is authoritative first

# From a HEALTHY node, not the damaged one:
pvecm status
ls /etc/pve/nodes/
find /etc/pve/nodes -name '*.conf' | wc -l
sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check;'

That count is the number you will verify against after the resync. Write it down.

Destructiveon the damaged node: discard the local copy and rejoin the sync

STAMP=20260812-0300

systemctl stop pve-cluster
systemctl stop corosync

mv /var/lib/pve-cluster/config.db "/root/pmxcfs-rescue-$STAMP/config.db.discarded"
rm -f /var/lib/pve-cluster/config.db-wal /var/lib/pve-cluster/config.db-shm

systemctl start corosync
sleep 5
systemctl start pve-cluster
sleep 10

systemctl is-active pve-cluster
pvecm status
find /etc/pve/nodes -name '*.conf' | wc -l

The node rebuilds its local database from the quorate cluster. The config count must match the number recorded from the healthy node.

Step 5: Every node damaged - restore or rebuild

Destructiverestore config.db from backup on ONE node

BACKUP=/mnt/offsite/pve/config.db-20260810
STAMP=20260812-0300

systemctl stop pve-cluster
cp -a "$BACKUP" /var/lib/pve-cluster/config.db
rm -f /var/lib/pve-cluster/config.db-wal /var/lib/pve-cluster/config.db-shm
chown root:root /var/lib/pve-cluster/config.db
chmod 0600 /var/lib/pve-cluster/config.db

systemctl start pve-cluster
sleep 10
ls /etc/pve/nodes/
find /etc/pve/nodes -name '*.conf' | wc -l

Then resync the other nodes from this one using the Step 4 procedure.

If there is no backup: local mode as an extraction tool

Destructivepmxcfs local mode - read out, do not work in

systemctl stop pve-cluster
pmxcfs -l

ls -R /etc/pve/nodes/ | head -50
cp -a /etc/pve/nodes /root/pmxcfs-local-extract/
cp -a /etc/pve/storage.cfg /etc/pve/user.cfg /root/pmxcfs-local-extract/ 2>/dev/null

# Then stop local mode again
pkill -f 'pmxcfs -l'

Local mode makes /etc/pve writable without quorum, and nothing it writes is replicated

pmxcfs -l starts the cluster filesystem in local mode: no corosync, no quorum requirement, /etc/pve writable. It is the standard tool for getting configuration out of a broken node.

It is also the standard way people make an incident worse, because /etc/pve looks normal again and the temptation is to carry on administering. Changes made in local mode exist only on that node. When the node rejoins a cluster, they can be silently overwritten by the cluster state - or, worse, partially merged.

Use it to read and copy. Do not use it to create guests, change storage, or “just fix” the HA config. And never run it on a node whose peers are still live and quorate: two nodes both writing an unreplicated /etc/pve is exactly the split-brain the design prevents.

Step 6: Bring the stack up in order

Service impact possiblecorosync, then pmxcfs, then the daemons

systemctl start corosync
sleep 5
corosync-cfgtool -s

systemctl start pve-cluster
sleep 10
systemctl is-active pve-cluster

systemctl restart pvedaemon pveproxy pvestatd
systemctl restart pve-ha-lrm pve-ha-crm

systemctl --failed --no-pager

Order matters: pmxcfs needs corosync to sync, and the PVE daemons need /etc/pve to exist before they will behave.

Step 7: Verify, with checks that can fail

Read-only / Safemounted, writable, and replicating

mount | grep '/etc/pve'

# Writable, not merely present
touch /etc/pve/.rwtest && echo 'writable' && rm -f /etc/pve/.rwtest

pvecm status
find /etc/pve/nodes -name '*.conf' | wc -l
pvesm status
ha-manager status
sqlite3 /var/lib/pve-cluster/config.db 'PRAGMA integrity_check;'

Read-only / Safeprove replication actually works

# On node A
echo "recovery-check-$(date +%s)" > /etc/pve/.replication-test
cat /etc/pve/.replication-test

# On node B, within a few seconds
#   cat /etc/pve/.replication-test
# Then clean up on either node:
#   rm -f /etc/pve/.replication-test

A mounted /etc/pve that does not replicate is a node that will diverge from the cluster. That check is the one that distinguishes “pmxcfs started” from “pmxcfs recovered”.

Step 8: Reconcile against the inventory

Read-only / Safedoes the configuration match reality

# Configs the cluster believes in
find /etc/pve/nodes -name '*.conf' | sed 's#.*/##; s/\.conf$//' | sort > /tmp/configured.txt
wc -l /tmp/configured.txt

# Guests actually running right now
ps -eo cmd | grep -oP '(?<=-id )[0-9]+' | sort > /tmp/running.txt

comm -13 /tmp/configured.txt /tmp/running.txt
# Anything printed is RUNNING with NO CONFIG - the urgent category

A guest running with no configuration will disappear the moment it stops. Reconstruct its config from the extracted dump, or from PBS, before anything reboots.

Then check the other direction: volumes on storage with no config referencing them.

Read-only / Safeorphaned disks

for S in $(pvesm status | awk 'NR>1 {print $1}'); do
echo "== $S"
pvesm list "$S" | awk 'NR>1 {print $1}'
done | tee /tmp/volumes.txt

grep -c 'vm-' /tmp/volumes.txt

Rollback

What you did	How to undo it
Copied the database	Nothing to undo
Moved the local `config.db` aside	Stop `pve-cluster`, move the copy back, start
Restored from backup	Stop `pve-cluster`, put the Step 2 copy back, start. You are back to broken, but broken is a known state
Wrote something in local mode	Not undoable cleanly. Note exactly what was written before rejoining
Ran a SQLite repair on the live database	Only recoverable from the Step 2 copy

Common patterns

Symptom	Likely cause	Resolution
`/etc/pve` read-only	Inquorate node - normal behaviour	Restore quorum, not the database
`database disk image is malformed`	Storage fault, unclean power loss, or full disk	Resync from cluster if healthy; else restore
`pve-cluster` restart loop	Corrupt database, or `/` full	`df -h /` first, then the journal
Empty `/etc/pve` after a join	Never synced	Remove the node and rejoin from clean
Corruption on several nodes at once	Shared cause: RAM, storage backplane, power event	Stop. Escalate. Repairing in parallel hides the cause
GUI shows nodes grey after recovery	Certificates or `pveproxy` state	`pvecm updatecerts --force`, restart `pveproxy`
Guests running that have no config	Configuration reverted to an older backup	Rebuild configs from the dump or PBS before anything stops

Escalation

Escalate when:

More than one node shows database corruption.
No backup of config.db exists and the cluster is not quorate.
Guests are running with no configuration.
A restore from backup is being considered - that is a cluster-wide decision.
Local mode is being considered on a node whose peers are live.

Why this matters

The instinctive response to a broken /etc/pve is to restart things until it comes back, and that instinct is directly opposed to what the situation rewards. Running guests are unaffected. The service is up. You have time, and the only truly irreversible mistake available is to destroy the last good copy of the configuration while trying to fix a node.

So the sequence is: copy first, classify second, repair third. The copy costs thirty seconds and is the rollback for everything that follows.

The concept worth carrying out of here is that /etc/pve is a database with a filesystem interface, replicated by consensus. That explains the read-only behaviour when quorum is lost - it is a safety property, not a fault - and it explains why local mode is so dangerous: it is the switch that turns the consensus off while leaving the interface looking entirely normal. A node in local mode feels fixed, and it is quietly writing configuration that no other node will ever see.