Rebuilding a destroyed cluster: /etc/pve, node identity and the platform itself

Every guest is in PBS. The datastore is replicated offsite. Restore has been drilled. And then the site is gone, and you are standing in front of new hardware with a working backup server and no idea what your storage was called.

A guest backup contains a guest. It does not contain the platform: which storages existed and how they authenticated, which users and tokens had which permissions, what the bridges and bonds were, which VMIDs mapped to which customers, what the firewall rules were, or the keys without which the encrypted backups you are so pleased about cannot be read.

That material is small - a few hundred kilobytes - and losing it turns a one-day recovery into a one-week reconstruction from memory.

What is identity and what is configuration

/etc/pve is a distributed filesystem backed by a SQLite database. Its contents divide into three categories that need different treatment.

The third row is the one that ends recoveries. priv/storage/<storeid>.enc holds the PBS encryption key for that storage. Without it, an encrypted datastore is a large collection of chunks that nobody can decrypt - and the backup of that key is, by default, only on the cluster you just lost.

Capturing the platform

Beyond /etc/pve, a node has host-level state that is not in the cluster filesystem and that a rebuild needs.

set -euo pipefail
STAGE="/root/platform-backup-$(hostname -s)-$(date +%F)"
mkdir -p "$STAGE"
chmod 700 "$STAGE"

# The cluster filesystem: configuration, identity and secrets.
tar czf "$STAGE/etc-pve.tar.gz" -C / etc/pve

# The pmxcfs database itself - the authoritative copy of the above.
cp /var/lib/pve-cluster/config.db "$STAGE/config.db"

# Host state that lives outside /etc/pve.
tar czf "$STAGE/etc-host.tar.gz" -C / \
etc/network/interfaces etc/hosts etc/hostname etc/resolv.conf \
etc/corosync etc/ssh etc/apt etc/systemd/system \
etc/multipath.conf etc/multipath 2>/dev/null || true

# Facts you will want and cannot reconstruct: disk identity and layout.
lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,SERIAL,MODEL > "$STAGE/lsblk.txt"
blkid                                            > "$STAGE/blkid.txt"
zpool status                                     > "$STAGE/zpool.txt" 2>/dev/null || true
pvs; vgs; lvs                                    > "$STAGE/lvm.txt" 2>/dev/null || true
pveversion -v                                    > "$STAGE/pveversion.txt"

ls -la "$STAGE"

Then send it somewhere that survives the site, encrypted, using the standalone client covered in standalone clients:

set -euo pipefail
STAGE="/root/platform-backup-$(hostname -s)-$(date +%F)"
export PBS_REPOSITORY='platform@pbs@pbs-dr.example.com:platform'
export PBS_PASSWORD_FILE=/etc/pve/priv/platform-backup.pw

proxmox-backup-client backup \
"platform.pxar:$STAGE" \
--backup-id "platform-$(hostname -s)" \
--keyfile /etc/proxmox-backup/platform.key

proxmox-backup-client snapshot list --backup-id "platform-$(hostname -s)"

# The staging copy holds cluster secrets. Do not leave it lying about.
shred -u "$STAGE"/* 2>/dev/null || rm -rf "$STAGE"

Scenario A: one node’s hardware died, the cluster is fine

The simplest case, and the one where people over-engineer.

The cluster still has quorum. The dead node’s configuration is already replicated on every survivor, because that is what pmxcfs does. You do not need a backup of /etc/pve at all.

set -euo pipefail
DEAD=pve-03

# Confirm it is gone and the cluster is quorate without it.
pvecm status
pvecm nodes

# Ceph first, if the node ran OSDs or a monitor - destroy those before delnode.
# See the Ceph day-2 lesson; this is not a step to improvise.

pvecm delnode "$DEAD"

# The stale node directory is left behind deliberately. Remove it once you
# have taken anything you wanted from it.
ls /etc/pve/nodes/
# rm -r "/etc/pve/nodes/$DEAD"

The replacement is a fresh install that joins with pvecm add. The documentation is explicit that reinstalling is the recommended way to bring a removed node back, because the old install still holds cluster keys and shared state that will conflict.

Scenario B: total cluster loss, hardware replaced

Every node gone. You have platform backups and guest backups offsite.

The order matters, and the instinct - restore guests first, because that is what people are asking for - produces a cluster you then have to rebuild underneath running workloads.

1. Install PVE fresh on the replacement nodes. Match the version to what you had: pveversion.txt from the platform backup is why you captured it.
2. Restore host networking from etc-host.tar.gz - interfaces, hosts, hostname. Reboot and confirm the nodes reach each other and the offsite PBS. Do not proceed until they do.
3. Create a new cluster with pvecm create and join the others. This is a new cluster with new keys. Do not attempt to resurrect the old cluster identity; there is nothing to resurrect it into.
4. Restore platform configuration selectively. Storage definitions, user and ACL config, firewall and SDN, HA rules. File by file, from the archive, into the new cluster - not by dropping in the old config.db, which would bring back the old cluster’s membership and node identities.
5. Restore the encryption keys into /etc/pve/priv/storage/ so the PBS storage can actually be read.
6. Attach the PBS datastore and confirm you can list snapshots.
7. Restore guests, in the dependency order from RPO, RTO and dependency modelling.
8. Re-enable HA last, after storage and networking are verified. HA on a half-built cluster fights you.

set -euo pipefail
ARCHIVE=/root/restore/etc-pve.tar.gz
WORK=/root/restore/extracted
mkdir -p "$WORK"

tar xzf "$ARCHIVE" -C "$WORK"
OLD="$WORK/etc/pve"

# Configuration that transfers cleanly to a new cluster.
cp "$OLD/storage.cfg"     /etc/pve/storage.cfg
cp "$OLD/user.cfg"        /etc/pve/user.cfg
cp "$OLD/datacenter.cfg"  /etc/pve/datacenter.cfg
cp -r "$OLD/firewall"     /etc/pve/ 2>/dev/null || true
cp -r "$OLD/sdn"          /etc/pve/ 2>/dev/null || true

# Secrets - restore with care, and check the modes afterwards.
mkdir -p /etc/pve/priv/storage
cp "$OLD"/priv/storage/*  /etc/pve/priv/storage/ 2>/dev/null || true
cp "$OLD/priv/token.cfg"  /etc/pve/priv/token.cfg 2>/dev/null || true

# Deliberately NOT copied: corosync.conf, authkey.*, nodes/ - these are the
# old cluster's identity and belong to the cluster that no longer exists.

pvesm status
pveum user list

Scenario C: rebuilding around one survivor

The awkward middle case: two nodes of three destroyed, one intact with its storage. The survivor has one vote of three and its /etc/pve is read-only, so nothing can be changed - including the cluster configuration that would fix it.

set -euo pipefail

pvecm status

# ONLY after confirming the other nodes are physically gone or powered off.
pvecm expected 1

pvecm status | grep -E 'Quorate|Expected votes'

From there the survivor is writable, and the rebuild proceeds:

1. pvecm delnode the destroyed nodes, so their membership does not follow you into the rebuilt cluster.
2. Confirm pvecm status shows one node, expected votes 1.
3. Build the replacement nodes fresh and pvecm add them.
4. Restore expected votes implicitly as nodes join - each addition raises the count, and quorum returns to a real majority.
5. Verify HA and storage before re-enabling HA resources.

The survivor keeps its guests running throughout. That is the point of doing it this way rather than rebuilding from backups: the surviving node’s workloads never stopped.

Verifying that a rebuild is possible

The only honest verification is a restore, and it is cheap because the payload is small.

set -euo pipefail
export PBS_REPOSITORY='platform@pbs@pbs-dr.example.com:platform'
SNAP='host/platform-pve-01/2026-08-11T02:00:00Z'
DEST=/srv/scratch/platform-verify

mkdir -p "$DEST"
proxmox-backup-client restore "$SNAP" platform.pxar "$DEST" \
--keyfile /etc/proxmox-backup/platform.key

# Everything a rebuild needs must be present and readable.
for F in etc-pve.tar.gz config.db etc-host.tar.gz lsblk.txt pveversion.txt; do
test -s "$DEST/$F" && printf 'OK   %s\n' "$F" || printf 'MISSING %s\n' "$F"
done

# The single most valuable check: are the storage encryption keys in there?
tar tzf "$DEST/etc-pve.tar.gz" | grep 'priv/storage/' \
|| echo 'NO STORAGE KEYS IN BACKUP - encrypted datastores are unrecoverable'

That last check is the one that matters most and the one nobody writes. A platform backup without the encryption keys is a platform backup for a cluster whose backups cannot be read.

Knowledge check