Restore testing discipline - the regular validation routine

Intermediate⏱ ~10 minbash

What you'll learn

Run restore tests at the right cadence
Capture and act on findings
Use restore tests to improve the backup system
Document and audit the test results

Prerequisites

Restore matters more than backup - the discipline of restore

Verified against Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL 9.x · Rocky Linux 9.x · AlmaLinux 9.x · Linux kernel 6.1 LTS / 6.6 LTS · systemd 255+ · OpenSSH 8.7p1 (RHEL 9) / 9.6p1 (Ubuntu 24.04) · nftables 1.0.x · chrony 4.x · Pacemaker 2.1.x · Corosync 3.1.x · 2026-08-09

Not yet marked complete on this device.

Restore testing is the discipline that turns “I hope this works” into “I know this works”. This lesson covers the routine and what to look for.

Test cadence

Frequency	Scope	Outcome
Weekly	Spot-check: extract one file	Detects corruption early
Monthly	One service, one host	Tests the procedure for that service
Quarterly	All services, one host	Full host restore
Annually	All services, all hosts	Full DR drill

The cost of testing is hours per quarter; the cost of a failed real restore is data loss.

Run a test restore

1. Choose a non-production host as the target.
2. Choose a recent backup (last weekly or monthly).
3. Follow the restore runbook exactly.
4. Time each step.
5. Verify the restored service.
6. Compare to the expected state.
7. Document findings.

What to look for

Backup corruption: backup file is unreadable.
Incomplete backup: missing files or directories.
Wrong permissions: restored files have wrong ownership or mode.
Missing dependencies: e.g. the application needs libraries not in the backup.
Stale configuration: the backup has a config that does not match the current expected state.
Time: how long did the restore take? Is it within RTO?

Capture findings

A test restore that finds nothing is still valuable (confirmation). A test restore that finds something is gold.

RESTORE TEST REPORT
Date: 2026-08-09
Backup used: backup-2026-08-02
Target: test-host-01

Steps:
1. Retrieved key from vault: 2 min
2. SSH to backup host: 1 min
3. Listed backups: 30 sec
4. Chose backup-2026-08-02: 30 sec
5. Mounted target: 1 min
6. Extracted: 18 min
7. Moved data: 12 min
8. Set ownership: 2 min
9. Started PostgreSQL: 5 sec
10. Verified data: 3 min

Total: 38 min (within RTO of 4 hours)

Findings:
- Restore procedure worked as documented
- No corruption detected
- Time was within RTO
- Improvement: backup list was slow; investigate why

Use findings to improve

Each test restore produces findings:

Restoration issues: fix the runbook.
Time issues: improve the procedure.
Missing items: update the backup to include them.
Permission issues: fix the restore to set them.
Corruption: investigate the backup pipeline.

The improvement loop is the value of testing.

Document and audit

Store restore test reports in a known location. Review quarterly. The trend over time:

Are restores getting faster?
Are findings decreasing?
Are procedures being followed?

A flat or worsening trend is a signal that the backup system needs attention.