This lab runs a complete restore drill: from backup to service running. By the end you will have a validated restore procedure.
Objective
Restore a production application onto a clean host from backup alone, measure how long each step really takes, and prove the restored data is identical to the original — content and metadata. A drill that cannot fail has verified nothing, so this one has a validation gate that fails on ownership, permission, ACL or extended-attribute drift.
Architecture
One host, three roles:
- Source of truth — the pre-restore baseline captured from the live application host (or from a snapshot of it).
- Backup store — the repository holding the most recent backup, plus its encryption key in the vault.
- Target — a clean test host with the application package installed but no application data.
The drill only proves anything if the target is genuinely clean. A target with leftover data from an earlier drill will pass validation for reasons that have nothing to do with the backup.
Requirements
- A test host with the application installed and the service stopped.
- Read access to the backup repository and its encryption key in the vault. Retrieving the key is part of the drill — do not pre-stage it.
getfacl,getfattr(packageattr),sha256sum,systemctl,curl.- On RHEL-family targets,
ls -Zand a working SELinux policy. - A maintenance window on the target. The drill destroys and rebuilds the application data directory.
Scenario
The application myapp stores its data in /var/lib/myapp, runs as
the myapp user, and has an RTO of 4 hours. Backups run nightly to an
encrypted repository. Nobody has restored from it in production. Your
job is to find out, today, whether it works.
Tasks
Task 1: Capture the pre-restore baseline
Everything the validation gate compares against comes from here. Take it from the live host, before you touch the target.
# Content and layout, with mode and ownership
sudo find /var/lib/myapp -printf '%p %m %U:%G\n' | sort > /tmp/pre-restore.txt
# POSIX ACLs
sudo getfacl -R -p /var/lib/myapp > /tmp/pre-restore.acl
# Extended attributes (includes security.selinux on RHEL family)
sudo getfattr -R -d -m - /var/lib/myapp > /tmp/pre-restore.xattr
# Content hashes
sudo find /var/lib/myapp -type f -exec sha256sum {} + | sort > /tmp/pre-restore.sha256
# Copy all four off the host - they must survive the target being wiped
scp /tmp/pre-restore.* operator@jump-host:/srv/drills/Task 2: Prepare the target
sudo systemctl stop myapp
sudo mv /var/lib/myapp /var/lib/myapp.pre-drill
sudo mkdir -p /var/lib/myappKeep /var/lib/myapp.pre-drill until the drill is signed off. It is
your undo.
Task 3: Execute the restore, timing each step
Time each step by bracketing real work. A timestamp taken immediately after another timestamp measures nothing.
#!/usr/bin/env bash
set -euo pipefail
step() { # step <label> <command...>
local label="$1"; shift
local t0 t1
t0=$(date +%s)
"$@"
t1=$(date +%s)
printf '%-24s %6ss\n' "$label" "$((t1 - t0))" | tee -a /tmp/drill-timings.txt
}
DRILL_START=$(date +%s)
step 'vault key retrieval' vault kv get -field=key secret/myapp/backup
step 'repository mount' sudo mount /dev/sdb1 /mnt/backup
step 'data extract' sudo borg extract --numeric-ids /mnt/backup/repo::myapp-latest
step 'ownership + contexts' sudo restorecon -R /var/lib/myapp
step 'service start' sudo systemctl start myapp
step 'health check' curl -fsS http://localhost/health
printf '%-24s %6ss\n' 'TOTAL' "$(( $(date +%s) - DRILL_START ))" \
| tee -a /tmp/drill-timings.txtTask 4: Smoke-test the service
# Service running
systemctl status myapp
# Smoke test
curl -I http://localhost/health
# Data verification
psql -c "SELECT count(*) FROM users;"
# Compare with expected state
diff /etc/myapp/config.yml /backup/expected-config.ymlThese checks prove the service came up. They do not prove the restore was correct — a service will happily start on a data directory whose permissions are wrong in ways that only bite on the next write. The Validation section is what proves the restore.
Task 5: Document the drill
Fill the step timings from /tmp/drill-timings.txt and the metadata
results from the Validation section. Do not write numbers you did not
measure — an RTO figure that came from an estimate rather than a
stopwatch is the thing this drill exists to replace.
RESTORE DRILL REPORT
Date: 2026-08-09
Backup used: backup-2026-08-02
Target: test-host-01
RTO target: 4 hours
Actual time: 38 minutes <- TOTAL line from /tmp/drill-timings.txt
Steps (measured, not estimated):
1. Vault key retrieval: 2 min
2. Repository mount: 1 min
3. Data extract: 18 min
4. Ownership + contexts: 2 min
5. Service start: 5 sec
6. Health check: 3 min
Validation gate:
- Ownership/mode diff: PASS
- ACL diff: PASS
- xattr/SELinux diff: FAIL - 412 files restored as default_t
- Content hashes: PASS
- Service health: PASS
RESULT: FAIL (metadata)
Findings:
- Restore time well within RTO
- No content corruption detected
- Service starts cleanly, but the backup job omits --xattrs, so
SELinux contexts do not survive the round trip
Issues found:
- The backup list was slow; investigate
- The encryption key is in a different vault than the runbook
says; update the runbook
- The backup job runs tar without --acls --xattrs --selinux
Improvements to make:
1. Add --acls --xattrs --selinux to the backup job and re-drill
2. Update the runbook to point to the correct vault
3. Investigate slow backup list
4. Add a test for "find the vault" to the drillA drill that reports FAIL is a successful drill. It found a real defect in the backup job, in a maintenance window, on a test host. The alternative was finding it during a real restore.
Task 6: Update the runbook
Based on findings, update the runbook:
- Correct the vault reference.
- Add notes about backup list timing.
- Add the new “find the vault” test.
Task 7: Schedule the next drill
Quarterly cadence means four fields in the month column, not one.
# Restore drill: 02:00 on the 9th of Feb, May, Aug, Nov
0 2 9 2,5,8,11 * root /opt/scripts/restore-drill.shCheck the expression before you trust it. 0 2 9 11 * is minute 0,
hour 2, day-of-month 9, month 11 — that fires once a year, on 9
November. A schedule labelled quarterly that runs annually is worse
than no schedule, because the calendar entry stops anyone asking.
# Confirm the next few firings
sudo systemd-analyze calendar --iterations=4 '*-02,05,08,11-09 02:00:00'Validation
The drill passes only if all five gates pass. Run them on the target, against the baseline captured in Task 1.
# 1. Ownership and mode match the pre-restore baseline
sudo find /var/lib/myapp -printf '%p %m %U:%G\n' | sort > /tmp/post-restore.txt
diff /tmp/pre-restore.txt /tmp/post-restore.txt \
&& echo 'PASS: ownership/mode intact'
# 2. POSIX ACLs survived the round trip
sudo getfacl -R -p /var/lib/myapp > /tmp/post-restore.acl
diff /tmp/pre-restore.acl /tmp/post-restore.acl \
&& echo 'PASS: ACLs intact'
# 3. Extended attributes and SELinux contexts survived
sudo getfattr -R -d -m - /var/lib/myapp > /tmp/post-restore.xattr
diff /tmp/pre-restore.xattr /tmp/post-restore.xattr \
&& echo 'PASS: xattrs intact'
ls -Z /var/lib/myapp | head # RHEL family: must not be default_t
# 4. Content is complete and byte-identical, not merely present
sudo find /var/lib/myapp -type f -exec sha256sum {} + | sort > /tmp/post-restore.sha256
diff /tmp/pre-restore.sha256 /tmp/post-restore.sha256 \
&& echo 'PASS: content identical'
# 5. The service starts and serves
systemctl is-active myapp && curl -fsS http://localhost/health \
&& echo 'PASS: service healthy'Expected Outcome
At the end of the drill:
/var/lib/myappon the target is byte-identical to the baseline, with matching ownership, modes, ACLs and extended attributes./tmp/drill-timings.txtholds a per-step breakdown measured with a stopwatch, and the total is comfortably inside the 4-hour RTO.- The report records a PASS or FAIL per gate, with the FAILs traced to a specific backup-job option.
- The runbook has been corrected where the drill contradicted it.
- The next drill is scheduled with a cron expression you verified fires quarterly.
/var/lib/myapp.pre-drillhas been removed, once and only once the gates pass.