Backup verification and integrity testing

A backup you haven’t restored is a hope, not a backup. The corollary: a backup you haven’t verified is the same. PBS has built-in verify jobs that check chunk integrity; this lesson shows how to use them, build restore drills into your operational rhythm, and prove to yourself (and your auditor) that the restore actually works.

PBS verify jobs

PBS verifies backup chunks against their checksums. If a chunk’s checksum doesn’t match the stored value, the chunk is corrupt (or has been tampered with). Verify jobs run on a schedule and report results.

Configuring a verify job

In the GUI: PBS → Datastore → Verify Jobs → Add.

Or via CLI:

# On the PBS host
proxmox-backup-manager verify-job create weekly-verify \
  --store main \
  --schedule 'Mon 03:00' \
  --ignore-verified 14   # Skip chunks verified in the last 14 days

The ignore-verified flag is important. Without it, every job re-verifies every chunk. With 10 TB of backups, that’s a lot of CPU and disk I/O. With ignore-verified 14, each chunk is verified at most every 14 days.

Reading verify results

# List verify jobs
proxmox-backup-manager verify-job list

# Show recent verify results
proxmox-backup-manager verify-job show weekly-verify

The output includes:

• Schedule and last-run status
• Number of chunks verified
• Number of chunks with errors
• Number of snapshots verified

A clean verify job shows:

• errors: 0
• verified_chunks: <number>
• verified_snapshots: <number>

A failing verify job shows:

• errors: <number>
• Specific snapshot IDs with corrupt chunks
• The affected chunks (you’ll need to investigate)

What to do when verify finds errors

When a verify job reports errors:

1. Don’t panic. A single corrupt chunk is a small problem; the rest of the backup is fine.
2. Identify the snapshot. Look up which snapshot the corrupt chunk belongs to.
3. Check the underlying storage. If PBS lives on ZFS, run a scrub. If on hardware RAID, check the controller logs.
4. Restore from an earlier snapshot. If the corrupt chunk can’t be repaired, the snapshot with the corruption is unrecoverable. Use the next-earlier snapshot.
5. Investigate the root cause. Is the storage failing? Is the network introducing corruption? Is the disk controller buggy?
6. Document the incident. Even small corruption events are valuable signal.

File-level restore drills

A file-level restore drill: pick a random file from a random backup, restore it, and verify it matches the original.

Manual drill

# Pick a VM (use 100 as example)
VMID=100
SNAP=$(proxmox-backup-manager snapshot list $VMID --repo main | tail -1 | awk '{print $1}')

# Pick a known file from inside the VM
# Use the guest agent to find it
ORIGINAL_HASH=$(qm guest exec $VMID sha256sum /etc/passwd)

# Restore the file
proxmox-backup-client restore $VMID/$SNAP /etc/passwd - \
  --repository main \
  > /tmp/restored-passwd

RESTORED_HASH=$(sha256sum /tmp/restored-passwd | awk '{print $1}')

# Verify
if [ "$ORIGINAL_HASH" = "$RESTORED_HASH" ]; then
  echo "PASS: $VMID/$SNAP /etc/passwd restored intact"
else
  echo "FAIL: hash mismatch"
fi

Automated drill

A scheduled cron job that performs a random restore drill:

#!/bin/bash
# /usr/local/bin/restore-drill.sh
set -euo pipefail

REPO=main
DAYS_BACK=7

# Pick a random VM with a recent backup
VMID=$(proxmox-backup-manager list --repo $REPO | shuf -n1 | awk '{print $1}')

# Pick a snapshot within the drill window
SNAP=$(proxmox-backup-manager snapshot list $VMID --repo $REPO | \
  awk -v cutoff="$(date -d "$DAYS_BACK days ago" +%s)" \
  '$2 >= cutoff {print $1; exit}')

if [ -z "$SNAP" ]; then
  echo "No snapshot in last $DAYS_BACK days for VM $VMID"
  exit 0
fi

# Files to test (should be present in every Linux VM)
FILES="/etc/passwd /etc/hostname /etc/os-release"

for f in $FILES; do
  # Get the hash from the running VM
  if qm guest exec $VMID sha256sum "$f" > /tmp/drill-current 2>/dev/null; then
    CURRENT=$(awk '{print $1}' /tmp/drill-current)
  else
    echo "VM $VMID not running, skipping"
    continue
  fi

  # Restore the file from PBS
  RESTORED=$(proxmox-backup-client restore $VMID/$SNAP "$f" - \
    --repository $REPO 2>/dev/null | sha256sum | awk '{print $1}')

  if [ "$CURRENT" = "$RESTORED" ]; then
    echo "PASS $VMID/$SNAP $f"
  else
    echo "FAIL $VMID/$SNAP $f current=$CURRENT restored=$RESTORED"
  fi
done

Run this weekly via cron. Each run validates three to five files from a random recent backup. Failures are immediate alerts.

VM-level restore drills

A VM-level restore drill is more involved. Pick a VM, restore it to a sandbox, boot it, verify it works.

#!/bin/bash
# /usr/local/bin/vm-restore-drill.sh

set -euo pipefail
REPO=main
VMID=$1   # The VM to drill on

# Get the latest snapshot
SNAP=$(proxmox-backup-manager snapshot list $VMID --repo $REPO | tail -1 | awk '{print $1}')
echo "Drill: restoring VM $VMID from snapshot $SNAP"

# Restore to a new VMID (use 900 + VMID to avoid conflicts)
DRILL_VMID=$((900 + VMID))
qmrestore "$REPO:vzdump-qemu-$VMID-$SNAP.*" $DRILL_VMID \
  --storage local-zfs

# Start the drill VM
qm start $DRILL_VMID

# Wait for boot
sleep 30

# Run validation commands via guest agent
STATUS=$(qm guest exec $DRILL_VMID systemctl is-system-running)
if [ "$STATUS" = "running" ]; then
  echo "PASS: VM $DRILL_VMID (from $VMID) boots and runs"
else
  echo "FAIL: VM $DRILL_VMID status: $STATUS"
fi

# Compare the restored VM to the source
# (could check application status, file hashes, etc.)

# Clean up
qm stop $DRILL_VMID
qm destroy $DRILL_VMID

Run this monthly on a non-critical VM. Verify the drill VM boots, the guest agent responds, and key applications are functional.

Detecting silent data corruption

Silent corruption — bit rot that occurs without storage errors — is detected by:

• ZFS checksums at the storage layer (every block has a SHA-256 checksum)
• PBS chunk checksums at the backup layer (every chunk has a SHA-256 checksum)
• Application-level checks (database integrity checks, file checksums in critical paths)

The chain works when all three layers are exercised regularly:

• ZFS scrubs weekly catch corruption at the storage layer
• PBS verify weekly catches corruption at the backup layer
• Application-level checks (Postgres checksums, etc.) catch corruption at the data layer

If a corrupt chunk is found in PBS, the restore from that snapshot fails. If the same chunk is also corrupt in ZFS, even a ZFS scrub won’t repair it. The application-level check is the last line of defence.

Recovery-time dashboard

Build a Grafana panel showing:

• Last successful verify job: date, chunks verified
• Last successful restore drill: date, VM
• PBS datastore free space: %
• Backup retention: actual vs target

The dashboard answers the question “how confident am I that my backups work?” with metrics, not feelings.

# Last verify job duration
last_over_time(pbs_verify_duration_seconds[7d])

# Verify errors in the last 30 days
sum_over_time(pbs_verify_errors_total[30d])

# Days since last successful restore drill
time() - last_over_time(vm_restore_drill_success_timestamp[90d])

Production considerations

• Verify jobs compete with backups. Both read the PBS datastore. Schedule verify when backups aren’t running (e.g., late evening).
• Restore drills need real hardware. A restore drill on the production cluster exercises the production environment but also uses production resources. Schedule during low-traffic windows or on a separate test cluster.
• PBS performance. A 10 TB PBS datastore with verify taking hours is normal. If verify takes days, your PBS is undersized.
• Restore time vs RTO. The restore drill tells you how long a real restore takes. Compare to your RTO. If restore > RTO, your RTO is unrealistic.

Common mistakes

• No verify jobs. Backups without verify are guesses.
• Verify in production hours. Schedule verify for late evening or weekends.
• Restore drills only on the test cluster. Production drills reveal real-world timing and resource constraints.
• Treating verify as binary. Verify errors are signal, not noise. Investigate every error.

Key takeaways

• PBS verify jobs are scheduled checksum audits. Run them weekly.
• Restore drills prove backups work. Run them monthly.
• Three layers of corruption detection: storage, backup, application.
• Track verify success and restore drill success as SLIs.

Knowledge check