medium riskservice affecting~150 min

Prove a PBS backup is genuinely restorable, not merely green

1 · Prerequisites

Confirm every item is in place before any state change.

An isolated network exists for the restored guest: a bridge with no uplink, or a firewall rule set that provably blocks it from reaching production
Spare capacity exists on a node and a storage for a full copy of the guest being tested
A free VMID range is reserved for restore testing, so a test restore can never overwrite a production guest
The encryption key is available if the backups are encrypted, and its location is somewhere other than the cluster being protected
The service owner has provided an application-level check that would detect a bad restore, not merely a booting one
The agreed RTO and RPO for this guest are known, because the test measures against them

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· proxmox-backup-client or pvesm list shows the snapshots that exist for the guest, and the newest one is as recent as the schedule implies
· The PBS verify job status for the chosen snapshot is known, and whether it has ever been verified at all
· The target node has enough free memory and the target storage enough free space for a full restore
· The isolated bridge exists and is confirmed to have no path to production, tested rather than assumed
· The chosen test VMID is unused on every node
· The encryption key, if any, is confirmed usable by decrypting a snapshot listing before the restore begins

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Choose the guest and the snapshot to test, favouring the newest and a deliberately older one
2Confirm the snapshot exists and record its timestamp, size and verification state
3Confirm you can read the snapshot metadata, which proves the encryption key works before a long restore begins
4Prepare an isolated network so the restored guest cannot contact production
5Note the start time - the restore duration is the measurement this test exists to produce
6Restore to an unused test VMID on the test storage
7Attach the restored guest only to the isolated bridge, and confirm it has no other network interface
8Start the guest and confirm it reaches its operating system
9Run the application-level check the service owner provided
10Check data currency inside the guest: the newest record, file or log entry, compared against the snapshot time
11Record the measured RTO and compare it against the agreed target
12Destroy the test guest and free the storage
13Record the result, including anything that had to be worked around

4 · Verification

Confirm the procedure actually fixed the problem.

✓The restore completed with a non-zero-size disk and no errors in the task log
✓The restored guest boots to its operating system, confirmed on the console or via the guest agent
✓The application-level check passes inside the restored guest, and it is a check capable of failing on stale or corrupt data
✓Data currency inside the guest matches the snapshot timestamp - a restore that is a week older than the snapshot claims is a finding
✓The restored guest has no network path to production, verified by attempting to reach a production address and failing
✓The measured restore duration is recorded and compared against the agreed RTO
✓The test guest and its volumes are removed afterwards, confirmed by listing the storage

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶This procedure creates a new guest at an unused VMID and changes nothing about production, so there is nothing to roll back if it is followed correctly
↶The one destructive mistake available is restoring over a production VMID. Reserving a test VMID range is what prevents it, and the check that the target VMID is unused is not optional
↶If a restored guest is accidentally attached to a production network, shut it down immediately rather than reconfiguring it while it runs
↶If the test restore consumed enough storage to threaten production, delete it first and investigate afterwards
↶Leaving the test guest in place after the test is not harmless: it consumes capacity and will eventually be started by someone who does not know what it is

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if a restore fails or produces a guest that does not boot, because the backup that was believed good is not
· Escalate if the encryption key cannot decrypt the snapshot, since every backup under that key is unreadable and that is a standing data-loss condition
· Escalate to the service owner if the measured RTO exceeds the agreed target, because the recovery plan is based on a number that is wrong
· Escalate if data inside the restored guest is materially older than the snapshot timestamp, which indicates the backup captured a stale or quiesced-incorrectly state
· Escalate if a restored guest reached a production network at any point during the test

Verified against Proxmox Backup Server 4.2.5 and Proxmox VE 9.2.4.

A PBS verify job reads the stored chunks and checks their checksums against what the index says they should be. That is a genuinely valuable thing to do, and it answers exactly one question: has the data on the datastore rotted since it was written?

It does not answer any of the questions you have when the building is on fire:

Will the guest boot from this?
Is the data inside it consistent, or was it captured mid-transaction?
Can I decrypt it, if my key lived on the cluster that just died?
How long does the restore take?
Is what got backed up actually what matters?

Only a restore answers those. This runbook is how you do one on a Tuesday instead of during an incident.

When to use this runbook

Scheduled restore testing - quarterly at minimum for anything critical.
After a change to the backup configuration, schedule or datastore.
Before relying on a new backup target for the first time.
As part of a DR exercise.
After any incident where a backup was needed and something was harder than expected.

What each layer actually proves

Check	Proves	Does not prove
Backup job succeeded	Data was written to the datastore	That it is readable, complete, or useful
Verify job green	Chunks match their checksums	That a guest can be built from them
`snapshot list` works	Metadata is readable; the key works for metadata	That the data chunks restore
Restore completes	The image can be reconstructed	That the guest boots
Guest boots	The disk is structurally sound	That the application works or the data is current
Application check passes	The service can run on this data	That the data is as recent as you think
Data currency check	The backup captured what you expect	Nothing further. This is the end of the ladder

Every step of that ladder is skipped by “the backup dashboard is green”.

Step 1: Choose what to test

Read-only / Safewhat snapshots exist, and how old

STORE=pbs-main
VMID=104

pvesm list "$STORE" | grep "vm/$VMID"
pvesm list "$STORE" | grep "vm/$VMID" | tail -3

Read-only / Safefrom the PBS side, with verification state

export PBS_REPOSITORY='backup@pbs@192.0.2.50:main'

proxmox-backup-client snapshot list --repository "$PBS_REPOSITORY"
proxmox-backup-client snapshot list --repository "$PBS_REPOSITORY" --output-format json \
| head -c 2000
echo

Test two snapshots: the newest, and one deliberately older - a week or a month back. The older one exercises pruning and long-term retention, which is where “the backup exists” and “the backup is restorable” most often diverge.

Step 2: Prove the encryption key before the long part

Read-only / Safecan you decrypt at all

export PBS_REPOSITORY='backup@pbs@192.0.2.50:main'
KEYFILE=/etc/pve/priv/storage/pbs-main.enc

ls -la "$KEYFILE"
proxmox-backup-client snapshot list --repository "$PBS_REPOSITORY" --keyfile "$KEYFILE"

SNAP='vm/104/2026-08-11T23:00:00Z'
proxmox-backup-client catalog dump "$SNAP" --repository "$PBS_REPOSITORY" --keyfile "$KEYFILE" \
2>/dev/null | head -20

If the encryption key exists only on the cluster it protects, the backups are not a recovery plan

Client-side encrypted PBS backups are unreadable without the key. There is no escrow, no recovery mechanism and no vendor who can help.

The test is one question: if this whole cluster were destroyed right now, could you decrypt these backups? If the honest answer involves a file on the cluster, then the backups protect you from a guest being deleted and not from anything larger.

Verify this deliberately, as part of every restore test:

The key is in the password manager or secret store, not only in /etc/pve/priv/.
Someone other than you can retrieve it.
It has been tested by decrypting something recently - which is what the command above just did.

Doing this check early also saves an hour: discovering the key problem after a two-hour restore is discovering it in the worst order.

Step 3: Prepare isolation

Read-only / Safean isolated bridge with no uplink

grep -A5 'vmbr9' /etc/network/interfaces
ip -br link show vmbr9
bridge link show | grep vmbr9

An isolation bridge is a Linux bridge with no bridge-ports - nothing physical attached, so nothing on it can leave the host.

Configuration changecreate one if it does not exist

cat >> /etc/network/interfaces <<'EOF'

auto vmbr9
iface vmbr9 inet manual
  bridge-ports none
  bridge-stp off
  bridge-fd 0
#   restore-test isolation bridge - deliberately has no uplink
EOF

ifreload -a
ip -br link show vmbr9

Step 4: Reserve a safe VMID and start the clock

Read-only / Safethe target VMID must be unused everywhere

TESTVMID=9104

ls /etc/pve/nodes/*/qemu-server/"$TESTVMID".conf 2>/dev/null && echo 'IN USE - CHOOSE ANOTHER'
ls /etc/pve/nodes/*/lxc/"$TESTVMID".conf 2>/dev/null && echo 'IN USE - CHOOSE ANOTHER'
pvesh get /cluster/resources --type vm --output-format json | grep -c "\"vmid\":$TESTVMID"

Reserve a range - 9000-9999 is a common convention - and never restore a test into an ID below it. This check is the only thing standing between a restore test and overwriting the production guest you were testing.

Read-only / Safestart the clock: RTO is the number this test produces

date +%s > /tmp/restore-test-start
date

Step 5: Restore

Service impact possiblerestore to the test VMID, not the original

STORE=pbs-main
TESTVMID=9104
TARGET=local-zfs
SNAP='backup/vm/104/2026-08-11T23:00:00Z'

qmrestore "$STORE:$SNAP" "$TESTVMID" --storage "$TARGET" --unique 1
qm config "$TESTVMID"

--unique 1 generates new MAC addresses, which prevents an address clash with the still-running production guest. That matters even on an isolated bridge, because a duplicate MAC that leaks onto a real network is a genuinely confusing failure.

Configuration changeisolate every interface before starting

TESTVMID=9104

qm config "$TESTVMID" | grep '^net'

# Point every interface at the isolated bridge
qm set "$TESTVMID" --net0 virtio,bridge=vmbr9
qm set "$TESTVMID" --net1 virtio,bridge=vmbr9 2>/dev/null

qm config "$TESTVMID" | grep '^net'
# Every line must say vmbr9. A guest with two NICs has two chances to escape.

Step 6: Boot and check the ladder

Service impact possiblestart it

TESTVMID=9104

qm start "$TESTVMID"
sleep 90
qm status "$TESTVMID"
qm agent "$TESTVMID" ping 2>/dev/null || echo 'agent not responding yet'

Read-only / Safedid it really boot, or is QEMU merely running

TESTVMID=9104

qm agent "$TESTVMID" get-osinfo
qm agent "$TESTVMID" exec -- /usr/bin/uptime
qm agent "$TESTVMID" exec -- /bin/systemctl --failed --no-pager

If there is no guest agent, use the console. qm status reporting running proves QEMU started, and a guest sitting at an unbootable-disk error also reports running.

Read-only / Safeprove the isolation actually holds

TESTVMID=9104
PRODADDR=192.0.2.11

qm agent "$TESTVMID" exec -- /bin/ping -c 2 -W 2 "$PRODADDR"
# This MUST fail. If it succeeds, shut the guest down immediately.

Step 7: The application check

This is the step that distinguishes a restore test from a boot test, and it must come from the service owner - you cannot invent a meaningful check for someone else’s application.

Read-only / Saferun the owner's check inside the guest

TESTVMID=9104

# Examples of the shape - the real one comes from the service owner:
qm agent "$TESTVMID" exec -- /bin/systemctl is-active postgresql
qm agent "$TESTVMID" exec -- /usr/bin/psql -U postgres -c 'SELECT count(*) FROM orders;'
qm agent "$TESTVMID" exec -- /bin/ls -la /var/lib/app/data

Read-only / Safedata currency - the check people forget

TESTVMID=9104
SNAPTIME='2026-08-11T23:00:00Z'

echo "snapshot claims: $SNAPTIME"

# Newest record, file or log line inside the guest
qm agent "$TESTVMID" exec -- /bin/sh -c 'ls -lt --time-style=full-iso /var/log | head -5'
qm agent "$TESTVMID" exec -- /usr/bin/psql -U postgres -tc \
'SELECT max(created_at) FROM orders;'

Step 8: Record the RTO

Read-only / Safewhat the recovery actually costs

START=$(cat /tmp/restore-test-start)
NOW=$(date +%s)
echo "restore to verified-working: $(( (NOW - START) / 60 )) minutes"

TESTVMID=9104
qm config "$TESTVMID" | grep -E '^(scsi|virtio|sata)[0-9]'

Then do the arithmetic that matters: this guest took N minutes. In a real disaster you are restoring not one guest but all of them, over the same network, onto storage that is also rebuilding. If the DR plan assumes forty guests in four hours and one guest took twenty-five minutes, the plan is wrong by a factor nobody has noticed.

State that explicitly in the result. A measured single-guest RTO with the multiplication left undone is a number that will reassure people incorrectly.

Step 9: Clean up

Destructiveremove the test guest completely

TESTVMID=9104

qm stop "$TESTVMID"
sleep 10
qm destroy "$TESTVMID" --purge 1 --destroy-unreferenced-disks 1

ls /etc/pve/nodes/*/qemu-server/"$TESTVMID".conf 2>/dev/null || echo 'config removed'
pvesm list local-zfs | grep "vm-$TESTVMID-" || echo 'volumes removed'

Check the storage listing explicitly. A destroyed VM that left volumes behind is capacity nobody will reclaim, and next quarter’s test will find the storage a little fuller for no reason anyone can explain.

Step 10: Record the result

The output of this procedure is not “the restore worked”. It is:

Which guest, which snapshot, which datastore.
Measured restore duration, and the extrapolation to a full DR event.
Whether the encryption key was retrievable independently of the cluster.
Whether the application check passed, and who defined it.
Data currency: snapshot timestamp versus newest data found.
Anything that had to be worked around, however small - because during a real incident those become the delays.
What was not tested, stated plainly.

A test that found nothing and recorded nothing has not improved anything.

Rollback

Situation	Action
Restore to a wrong, production VMID	Stop immediately. This overwrote a live guest - treat as an incident and restore that guest properly
Test guest reached production	Shut it down at once. Do not reconfigure it while running. Then assess what it contacted
Restore filling the storage	Delete the test guest first; investigate after
Test complete	Destroy the guest and verify the volumes are gone

Common patterns

Symptom	Likely cause	Resolution
Verify green, restore fails	Verify checks chunks, not restorability	This is exactly why the test exists. Escalate
Cannot decrypt	Key missing or wrong	Standing data-loss condition. Escalate immediately
Guest boots to an unbootable-disk error	EFI disk or boot order not captured or not restored	Check `efidisk0`; confirm what the backup includes
Data older than the snapshot	Application not writing, or a volume excluded	Investigate what the job actually covers
Restore very slow	Datastore on spinning disks, or network limited	Real RTO input. Record it rather than dismissing it
Guest fails to start: no space	Test restore sized against used, not allocated	Check allocated size first
Restored guest joins production	Isolation not applied to every interface	Two layers of isolation, checked after boot
Test guest still present months later	Cleanup skipped	Destroy it; add cleanup to the close-out

Escalation

Escalate when:

A restore fails, or the guest does not boot.
The encryption key cannot decrypt.
Measured RTO exceeds the agreed target.
Data currency does not match the snapshot.
A restored guest touched a production network.

Why this matters

Backup verification answers a storage question - have these bytes rotted - and the organisation reads it as an answer to a recovery question. The gap between those two is where most backup failures live, and it is invisible precisely because the dashboard is green.

The single most valuable check in this runbook is the cheapest: compare the newest data inside the restored guest against the snapshot timestamp. It costs one query and it catches the whole class of failure where backups run perfectly against something that stopped being the thing you needed - an application that no longer writes there, a volume that was added and never included, a database whose files moved.

The second thing this produces is a number. RTO in a DR plan is usually an estimate someone made in a meeting; here it is measured, and it is measured for one guest under ideal conditions with everything else working. Multiplying that by the number of guests you would actually be restoring, over a network that is also rebuilding, is often the moment a recovery plan stops being plausible - and it is much better to learn that on a Tuesday.