Runbook: Restore procedure - files, ownership and service state

This is the runbook the rest of the backup domain depends on. linux-lab-restore-drill tells you to “follow the runbook”; the disaster recovery checklist gates on “restore runbook is current” and “restore runbook has been tested within the last quarter”. This is that document.

Restore is the only part of a backup system that has a deadline. Work through the steps in order and do not skip the verification: a restore that produced the wrong ownership is not a successful restore, it is an outage that starts later.

Step 0: Start the clock

RESTORE_START=$(date -u +%FT%TZ)
echo "restore start ${RESTORE_START}, RTO 4h, target /srv/app" \
| sudo tee -a /var/log/restore-drill.log

Write the RTO down before you begin. It is the number that decides whether you keep going or escalate, and nobody estimates it accurately once the restore is already running.

Step 1: Validate the archive before touching production

Verify the source first. A corrupt archive discovered after you have destroyed the target turns a recoverable incident into data loss.

# Borg - structural check plus full data verification
borg check --verify-data ssh://backup@vault/./repo::app-2026-08-10

# Restic
restic check --read-data-subset=10%

# PostgreSQL custom-format dump - list the table of contents
pg_restore -l /backup/app-2026-08-10.dump | head

# Plain tar - read the whole archive without extracting
tar --acls --xattrs -tvf /backup/app-2026-08-10.tar.zst >/dev/null && echo OK

Step 2: Capture a metadata baseline

You cannot verify parity later if you did not measure before. This is the artefact Step 7 compares against.

sudo find /srv/app -printf '%p\t%u\t%g\t%m\n' | sort > /var/tmp/pre-owner.txt
sudo getfacl -R -p /srv/app                     > /var/tmp/pre-acl.txt
sudo getfattr -R -d -m - /srv/app 2>/dev/null   > /var/tmp/pre-xattr.txt
ls -Z -d /srv/app                               > /var/tmp/pre-context.txt
sudo du -sb /srv/app                            > /var/tmp/pre-size.txt

Step 3: Preserve the current state - never overwrite in place

sudo systemctl stop app.service
sudo systemctl is-active app.service     # expect: inactive

sudo mv /srv/app /srv/app.pre-restore-$(date +%F-%H%M)
sudo install -d -o root -g root -m 0755 /srv/app

mv on the same filesystem is atomic and instant, and it keeps a complete rollback path on disk. An extraction over a live directory is not reversible: it merges the archive into whatever is there, so a failed restore leaves a tree that is neither the old state nor the new one and you can no longer tell which files came from where.

Only delete the preserved copy after Step 8 has passed.

Step 4: Restore with metadata flags

The default flags of every restore tool discard the things that make a restore correct. Ownership, ACLs, extended attributes and hard links are all opt-in.

# From a mounted archive or a staging copy
sudo rsync -aAXH --numeric-ids --info=progress2 \
/mnt/restore/srv/app/ /srv/app/

# From a tar archive
sudo tar --acls --xattrs --numeric-owner --same-owner \
-xf /backup/app-2026-08-10.tar.zst -C /srv

# Borg direct extract (run from / and use the archive's relative path)
cd / && sudo borg extract --numeric-ids \
ssh://backup@vault/./repo::app-2026-08-10 srv/app

What each flag buys you:

--numeric-ids is the one people get wrong. Restoring a system whose app user is UID 1001 onto a host where app is UID 1005 will, without it, map by name and appear to work - until you restore onto a rescue image with no app user at all and every file lands owned by a stranger.

Step 5: Ownership, mode and context

Metadata is three separate things and restoring two of them is a very convincing failure.

sudo chown -R app:app /srv/app
sudo find /srv/app -type d -exec chmod 0750 {} +
sudo find /srv/app -type f -exec chmod 0640 {} +

# SELinux: relabel from policy, then confirm
sudo restorecon -R -v /srv/app
ls -Zd /srv/app

Step 6: Start the service and watch it

sudo systemctl start app.service
sudo journalctl -u app.service -f --since '1 min ago'

Watch the first sixty seconds. A service that starts and then exits at 45 seconds because it could not open one file counts as a failed restore, and systemctl is-active will have said active in between.

Step 7: Verify parity against the baseline

sudo find /srv/app -printf '%p\t%u\t%g\t%m\n' | sort > /var/tmp/post-owner.txt
sudo getfacl -R -p /srv/app                     > /var/tmp/post-acl.txt
sudo getfattr -R -d -m - /srv/app 2>/dev/null   > /var/tmp/post-xattr.txt

diff /var/tmp/pre-owner.txt /var/tmp/post-owner.txt
diff /var/tmp/pre-acl.txt   /var/tmp/post-acl.txt
diff /var/tmp/pre-xattr.txt /var/tmp/post-xattr.txt

Every diff must be empty, or every line of difference must be explained and written down. “Probably fine” is how a permissions incident is scheduled for next Tuesday.

Then check the data itself, not just the files:

# Database: does it agree with the recorded row counts?
psql -At -c 'SELECT count(*) FROM orders;'

# Files: does a checksum manifest still match?
sudo sha256sum -c /var/tmp/manifest.sha256 | grep -v ': OK$'

# Application: does its own self-test pass?
sudo -u app /usr/local/bin/app --self-check

Step 8: Record elapsed time against RTO

RESTORE_END=$(date -u +%FT%TZ)
echo "restore end ${RESTORE_END} start ${RESTORE_START} target /srv/app result PASS" \
| sudo tee -a /var/log/restore-drill.log

Record the number even when the restore succeeded, and especially when it took longer than the RTO. An RTO that has never been measured is a guess written on a slide. The measurement is the deliverable of every restore drill, and it is what justifies faster storage, a closer archive replica, or a smaller restore unit.

Only now remove the preserved copy:

sudo rm -rf /srv/app.pre-restore-*

Escalation

Escalate when:

• The archive fails its integrity check. Do not restore from it; find another copy.
• Elapsed time reaches half the RTO with the restore incomplete. Escalating at 50% leaves room to act; escalating at 100% does not.
• The restore would discard writes made after the backup point. That is the data owner’s decision, not yours.
• The restore is part of recovering from a compromise. The archive may contain the intrusion, and restoring it reinstates the attacker.