Disaster recovery for the backup server itself · Proxmox VE

Draw the dependency graph of your recovery plan. Every guest restore, every file-level recovery, every ransomware response and the platform rebuild from the previous lesson all have an arrow pointing at one box: the backup server.

Now ask what happens to that graph if the box is gone. On most estates the answer is that every arrow terminates in nothing, and this is rarely written down anywhere, because the backup server is the thing that protects other things and the question of what protects it does not naturally arise.

What is actually lost

Not everything, and knowing the split determines the recovery.

Component	Where it lives	Lost with the host?
Chunk store and indexes	The datastore filesystem	No if the storage survives
Datastore definitions	`/etc/proxmox-backup/datastore.cfg`	Yes
Users, ACLs, API tokens	`/etc/proxmox-backup/user.cfg`, `acl.cfg`	Yes
Remotes and sync jobs	`remote.cfg`, `sync.cfg`	Yes
Prune, GC and verify schedules	`prune.cfg`, `verification.cfg`	Yes
Server certificates and fingerprint	`/etc/proxmox-backup/proxy.{pem,key}`	Yes
Tape configuration, if used	`tape.cfg`, `media-pool.cfg`	Yes
Encryption keys	Client-side; PVE holds them in `/etc/pve/priv/storage/`	Not PBS’s to lose, and the worst thing to lose

The pattern: the data is on the disks, the ability to use it is in /etc/proxmox-backup/. That directory is small. Backing it up is the single highest-value action in this lesson, and it takes one line.

Read-only / Safecapture the PBS server configuration— Read-only collection. The output contains credentials and certificate private keys - encrypt it and store it away from this host.

set -euo pipefail
STAGE="/root/pbs-config-$(hostname -s)-$(date +%F)"
mkdir -p "$STAGE"; chmod 700 "$STAGE"

tar czf "$STAGE/etc-proxmox-backup.tar.gz" -C / etc/proxmox-backup

# Host-level state a rebuild needs.
tar czf "$STAGE/etc-host.tar.gz" -C / \
etc/network/interfaces etc/hosts etc/hostname etc/ssh 2>/dev/null || true

# How the datastore was assembled - not reconstructible from the chunks.
lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,SERIAL > "$STAGE/lsblk.txt"
zpool status                               > "$STAGE/zpool.txt" 2>/dev/null || true
findmnt -no TARGET,SOURCE,FSTYPE           > "$STAGE/mounts.txt"
dpkg -l 'proxmox-backup*' 'pbs-*'          > "$STAGE/versions.txt"
proxmox-backup-manager cert info           > "$STAGE/cert.txt"

ls -la "$STAGE"

PBS-to-PBS sync as the DR mechanism

The datastore itself is protected by replicating it to a second PBS server, and PBS supports sync in both directions. The choice between them is a security decision, not a networking one.

Pull: the DR server fetches

Configuration changepull sync - configured and run on the DR server— Creates a remote and a sync job on the DR side. The DR server initiates every connection; the primary needs no credentials for the DR server.

set -euo pipefail

# On the PRIMARY, get the fingerprint the DR server must trust.
# proxmox-backup-manager cert info | grep -i fingerprint

# On the DR server:
proxmox-backup-manager remote create pbs-primary \
--host pbs1.example.com \
--auth-id sync@pbs \
--password 'REPLACE_ME' \
--fingerprint 'REPLACE_WITH_PRIMARY_FINGERPRINT'

proxmox-backup-manager sync-job create dr-pull \
--remote pbs-primary \
--remote-store main \
--store dr-main \
--schedule 'hourly' \
--remove-vanished false

proxmox-backup-manager sync-job list

Push: the primary sends

Configuration changepush sync - configured and run on the primary— Creates a remote and a push-direction sync job on the primary. The primary holds credentials that can write to the DR server.

set -euo pipefail

proxmox-backup-manager remote create pbs-dr \
--host pbs-dr.example.com \
--auth-id 'push-main@pbs' \
--password 'REPLACE_ME' \
--fingerprint 'REPLACE_WITH_DR_FINGERPRINT'

proxmox-backup-manager sync-job create dr-push \
--remote pbs-dr \
--remote-store dr-main \
--store main \
--sync-direction push \
--schedule 'hourly'

proxmox-backup-manager sync-job list

Pull is the safer default, and the reason is ransomware

The two directions differ in which machine holds credentials for the other, and that decides what an attacker who owns the primary can reach.

Pull. The DR server has credentials for the primary. The primary has none for the DR server and, if the network is arranged properly, cannot even connect to it. An attacker with full control of the primary can destroy the primary’s datastore and has no path to the copy.

Push. The primary has credentials that can write to the DR server. An attacker who owns the primary owns those credentials. Whether they can destroy the DR copy depends entirely on what permissions the push user has - and if that user was granted Remote.DatastorePrune so remove-vanished would work, the answer is that they can.

This is the same argument as the 3-2-1 discussion in ransomware and 3-2-1, applied to the sync mechanism itself. Push has real uses - a primary behind NAT with no inbound path, a DR site that must not initiate outbound connections - and where you use it, give the push user the minimum permission set and never remove-vanished.

The documentation adds an operational warning that applies to push in particular: use a dedicated remote configuration and a dedicated user for each push job, because pushed content is owned by the configured remote user, and several jobs sharing one user with remove-vanished enabled will delete each other’s snapshots.

Why a synced datastore is immediately usable, and what verification is really for

A PBS sync is not a file copy of the chunk store. The target requests the snapshot’s index, works out which chunk digests it does not already hold, and fetches only those. Chunks it already has - because they are shared with an earlier snapshot, or with a completely different guest - are not transferred at all.

That has a consequence people find surprising the first time: a DR datastore receiving its second guest often transfers a small fraction of that guest’s size, because most of the chunks arrived with the first. Deduplication is global to the datastore and applies across the sync.

It also means the DR datastore is not a copy in any interesting sense. It is an independently assembled chunk store that happens to contain the same content, addressed by the same digests. There is no “restore the backup server first” step - a DR PBS with synced snapshots is immediately a restore source, which is the property the whole architecture rests on.

Content addressing is also why verification matters and what it does. Every chunk’s name is the hash of its content, so a verify job can recompute the hash and detect bit rot, a bad disk, or a truncated transfer, with no reference copy needed. The digest is the checksum.

Run verification on the DR datastore, on its own schedule, and do not assume the primary’s verification covers it. They are separate stores on separate disks with separate failure modes, and the DR copy is the one that will be read under pressure by someone with no time to discover a bad chunk.

Key escrow: the part with no support path

If your backups are encrypted - and they should be, particularly offsite - then the key is a component of the recovery system, and it is the only one with no fallback whatsoever.

PBS provides a master-key mechanism for exactly this, and it is the piece most estates skip.

Configuration changecreate a master key pair and enrol it— Generates an RSA key pair. The private key must be removed from this host and stored offline - it is the escrow, and leaving it here defeats the purpose.

set -euo pipefail

# Creates master-public.pem and master-private.pem in the current directory.
proxmox-backup-client key create-master-key

# Tell the client to wrap each new backup key with the master public key.
proxmox-backup-client key import-master-pubkey ./master-public.pem

ls -l master-public.pem master-private.pem

Read-only / Safeproduce a paper backup of a key— Read-only. Writes a text and QR representation of the key to a file. Print it, store it off-site, and delete the file.

set -euo pipefail

proxmox-backup-client key paperkey \
--output-format text /etc/proxmox-backup/encryption-key.json \
> /root/paperkey.txt

wc -l /root/paperkey.txt
# print it, then:
# shred -u /root/paperkey.txt

Data-loss riskrecover a backup key using the master private key— The recovery path the escrow exists for. Requires the offline master private key. Run on a machine that will hold the recovered key only as long as the restore takes.

set -euo pipefail

proxmox-backup-client key import-with-master-key /root/recovered.key \
--master-keyfile /media/escrow/master-private.pem \
--encrypted-keyfile /root/rsa-encrypted.key

ls -l /root/recovered.key

A workable escrow design, stated as requirements rather than products:

Requirement	Why
The master private key exists on media that is not any running system	So a compromise of the estate does not include it
At least two copies, at two addresses	One safe, one fire
A paper key as the last resort	Media fails silently over years; paper does not
Two people needed to retrieve it	So one person’s departure or compromise is not decisive
An annual test that reads a backup using only escrowed material	Because untested escrow is a belief, not a control

That last row is the one that finds the problem. The commonest escrow defect is not a lost key; it is a key that was rotated on the systems and never re-escrowed, so the safe holds a key that decrypts nothing made in the last two years.

Rebuilding a PBS host

Datastore disks survived - a dead motherboard, a failed boot device:

Install PBS fresh, matching the previous version from versions.txt.
Restore host networking, hostname and /etc/hosts.
Attach the datastore storage; confirm the filesystem mounts and the chunk store directory is intact.
Restore /etc/proxmox-backup/ from the config archive.
Restart proxmox-backup-proxy and proxmox-backup.
Verify before announcing it is back.

Configuration changere-adopt an existing datastore on a rebuilt host— Creates the datastore definition pointing at an existing chunk store. PBS scans and adopts the existing content; it does not overwrite it.

set -euo pipefail
DSPATH=/mnt/datastore/main

# The chunk store must be there before you define anything.
test -d "$DSPATH/.chunks" || { echo 'no chunk store at that path'; exit 1; }

proxmox-backup-manager datastore create main "$DSPATH"
proxmox-backup-manager datastore list

# Prove the content is real, not merely present.
proxmox-backup-client snapshot list --repository 'root@pam@localhost:main' | head

Then start a verification of the whole datastore - from the web interface, or by scheduling a verify job - and wait for it to finish before telling anyone the backup server is back. A datastore that lists snapshots has an intact index; only a verify pass recomputes the chunk digests and proves the data behind them survived whatever killed the host.

Datastore disks lost as well - the DR server is now the primary:

Point the PVE cluster’s storage definition at the DR PBS server. Its fingerprint differs, so the storage entry must be updated, and this is where the printed fingerprint from the platform backup earns its keep.
Restore what you need directly from the DR datastore. It is a complete restore source; there is no rehydration step.
Build the replacement primary, and reverse the sync so the rebuilt server pulls from the DR server.
Reverse it back only once the new primary has a verified, complete datastore.

Step 3 is the one that gets skipped under pressure, and skipping it means running with a single copy for however long the rebuild takes - which is precisely the condition that just cost you a server.

What happens if the PBS server is compromised rather than destroyed?

A harder problem than loss, because the sync mechanism that protects you against destruction can propagate a compromise.

Datastore contents are not silently rewritable. The chunk store is content-addressed: a chunk’s name is the hash of its content, so a chunk modified in place no longer matches its name and verification detects it. An attacker cannot quietly alter the contents of a backup.

Deletion is entirely available. Snapshots can be removed, prune schedules can be set to nothing, and garbage collection can be run to reclaim the space. None of that requires defeating the content addressing.

Sync will carry the deletion to any target with remove-vanished enabled, which is the concrete argument made above.

So the response differs from the destruction case:

Isolate before investigating. Disable sync jobs in both directions immediately, so the compromise cannot propagate while you work.
Verify the DR copy on the DR server, using its own verify job, not the primary’s records of it.
Establish when. The DR server’s snapshot list is the evidence: compare what it holds against what the primary holds, and the divergence dates the deletion.
Rebuild the primary rather than cleaning it. A compromised backup server held credentials for everything it backs up.
Rotate everything it knew - API tokens on the cluster, sync credentials, and the encryption keys if there is any chance they were readable.

Point 5 is the one people underestimate. A PBS server holds credentials to read every guest in the estate. Its compromise is not a backup incident, it is an estate-wide credential incident, and treating it as the former is how the attacker keeps access after the rebuild.

Verification

Read-only / Safequarterly: prove the DR backup server can stand alone— Read-only against the DR datastore. Run every step against the DR server explicitly, never against the primary - the point is to test the path you will actually use.

set -euo pipefail
export PBS_REPOSITORY='restore@pbs@pbs-dr.example.com:dr-main'

# 1. It is reachable and its fingerprint is what your documentation says.
proxmox-backup-manager cert info | grep -i fingerprint

# 2. It holds recent snapshots, not just old ones.
proxmox-backup-client snapshot list | tail -10

# 3. Its own verify job ran recently and passed - separate disks, separate
#    failure modes from the primary, so the primary's results say nothing.
proxmox-backup-manager task list --limit 20

# 4. A restore actually works using only DR-side credentials and the
#    escrowed key. This is the step that makes the other three meaningful.
proxmox-backup-client restore \
'vm/9001/2026-08-11T02:00:00Z' drive-scsi0.img.fidx /srv/scratch/dr-test.raw \
--keyfile /media/escrow/recovered.key

Step 4 is the whole exercise. Steps 1 to 3 confirm the DR server exists and its data is intact; only step 4 confirms that you can get data out of it, with the credentials and the key you would actually have, on a day when the primary is not there to help.

Knowledge check

Why this matters

The backup server is the one component whose failure invalidates every other recovery plan you have, and it is routinely the one component with no recovery plan of its own. The reason is structural rather than careless: backup programmes are designed by asking “what would we need if we lost X” for each X in the estate, and the backup server is never one of the Xs, because it is the answer to the question rather than a subject of it.

Two things follow from taking it seriously.

The first is cheap. Configuration is small, sync is built in, and a DR replica is immediately a restore source because content addressing means there is no rehydration step. An afternoon of work removes the single point of failure entirely.

The second is the one that requires an actual decision. Encryption without escrow converts a data-loss risk into a certainty, and escrow requires people and process rather than configuration - two custodians, two addresses, an annual test, a rule that key rotation includes re-escrow. That is unglamorous organisational work, and it is the difference between an estate whose offsite backups can be read by whoever survives and one where they are a large collection of random-looking bytes.