Restore a deleted configuration file under time pressure

Objective

Recovering one file is the most common restore anybody performs, and it is the one most often done badly, because it feels too small to deserve a procedure. This lab gives you the procedure and then puts a stopwatch on it.

The command that copies bytes out of a repository is one line. Everything that decides whether the recovery works is on either side of it: knowing what changed and when, choosing which copy to take, checking the copy before it goes anywhere near the live path, keeping the damaged version, and putting the new file in place with the metadata the service actually needs.

Architecture

Three recovery points and one damage event. Their order is the whole lab.

flowchart LR
    S1["snapshot 1\nconfig good"] --> S2["snapshot 2\nconfig good\nlog_level edited"]
    S2 --> D["deploy rewrites appd.conf\nupstream_url dropped\nowner root:root, mode 0600"]
    D --> S3["snapshot 3\nDAMAGE INSIDE"]
    S3 --> F["restart -> exit 78\ncannot read appd.conf"]
    S2 -.->|"newest recovery point\nthat predates the damage"| R["staging restore"]
    R --> I["install -o root -g rbdr-appd -m 0640"]

Snapshot 3 is the newest and it is the wrong one. Nothing about it looks wrong: it completed, it is listed, it is the most recent state of the directory. It is simply the most recent state of a directory that was already broken.

Requirements

• A disposable host with systemd and sudo. Everything created is prefixed rbdr- so Cleanup can be scoped and asserted.
• restic on PATH. The transcripts this course quotes were captured on:

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

• In C-simulation mode, run the service payload directly with sudo -u rbdr-appd /usr/local/bin/rbdr-appd wherever a task says systemctl restart. The exit status carries the same information; only the supervisor is missing.

Scenario

At 03:14 the on-call phone goes off. rbdr-appd is down. It was restarted forty minutes ago by an unrelated package upgrade and has not come back. Somebody deployed a configuration change yesterday evening.

You have three daily snapshots of /etc/rbdr-appd. You have a service that refuses to start. You are being asked how long this will take, which is the question you can only answer after you have chosen a recovery point.

Tasks

Task 1 — Record the pre-lab state, then arm the scenario

Cleanup compares against this file. Record it before anything exists.

LAB="$HOME/rbdr-lab-09"
CONF=/etc/rbdr-appd/appd.conf
export RESTIC_REPOSITORY="$LAB/rbdr-repo"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"
mkdir -p "$LAB/staging" "$LAB/evidence"

{
  date -Is
  ls -d /etc/rbdr-appd /usr/local/bin/rbdr-appd /etc/systemd/system/rbdr-appd.service 2>&1
  getent passwd rbdr-appd || echo "no rbdr-appd user"
  systemctl is-active rbdr-appd 2>&1
} | tee "$LAB/pre-state.txt"

Now build the service. It reads one file, refuses to start without it, and exits 78 (EX_CONFIG) when the configuration is unusable.

sudo install -d -m 0755 /etc/rbdr-appd
getent passwd rbdr-appd >/dev/null || sudo useradd --system --no-create-home rbdr-appd

sudo tee /usr/local/bin/rbdr-appd >/dev/null <<'PROG'
#!/bin/sh
CONF=/etc/rbdr-appd/appd.conf
[ -r "$CONF" ] || { echo "rbdr-appd: cannot read $CONF" >&2; exit 78; }
for KEY in listen_port upstream_url worker_threads; do
  grep -Eq "^$KEY=" "$CONF" || { echo "rbdr-appd: missing key $KEY" >&2; exit 78; }
done
echo "rbdr-appd: configuration accepted"
exec sleep infinity
PROG
sudo chmod 0755 /usr/local/bin/rbdr-appd

sudo tee /etc/systemd/system/rbdr-appd.service >/dev/null <<'UNIT'
[Unit]
Description=rbdr-appd lab service
[Service]
User=rbdr-appd
ExecStart=/usr/local/bin/rbdr-appd
UNIT
sudo systemctl daemon-reload

Then produce the history: a good file, one snapshot, a legitimate edit, a second snapshot, the damaging deploy, and a third snapshot that captures it.

printf '%s\n' '# rbdr-appd configuration' 'listen_port=8443' \
  'upstream_url=https://orders.internal.example/api' 'worker_threads=8' 'log_level=info' \
  | sudo tee "$CONF" >/dev/null
sudo chown root:rbdr-appd "$CONF"
sudo chmod 0640 "$CONF"
sudo systemctl start rbdr-appd

head -c 32 /dev/urandom | base64 > "$RESTIC_PASSWORD_FILE"
chmod 0600 "$RESTIC_PASSWORD_FILE"
sudo -E restic init
sudo -E restic backup --tag daily /etc/rbdr-appd

sleep 2
sudo sed -i 's/^log_level=info/log_level=debug/' "$CONF"
sudo -E restic backup --tag daily /etc/rbdr-appd

sleep 2
printf '%s\n' '# rendered by rbdr-deploy' 'listen_port=8443' 'worker_threads=8' 'log_level=debug' \
  | sudo tee "$CONF" >/dev/null
sudo chown root:root "$CONF"
sudo chmod 0600 "$CONF"

sleep 2
sudo -E restic backup --tag daily /etc/rbdr-appd
sudo systemctl restart rbdr-appd || true

The snapshots here are seconds apart rather than a day apart. The reasoning you are about to do is identical either way: it compares one timestamp against another.

Task 2 — Start the clock, then find what changed and when

T0=$(date +%s)
date -Is | tee "$LAB/timeline.txt"

systemctl --no-pager status rbdr-appd | head -12
sudo journalctl -u rbdr-appd --no-pager -n 5
stat -c '%n mtime=%y owner=%U:%G mode=%a' "$CONF"

Two facts come out of this and both matter. The journal gives the failure message and the exit status. The stat line gives the modification time — your damage time — and the current owner and mode, which are also part of what changed. A configuration file is bytes plus metadata, and a deploy that rewrites one usually rewrites both.

Task 3 — List the snapshots and pick one deliberately

sudo -E restic snapshots --tag daily
DAMAGED_AT=$(stat -c %Y "$CONF")
date -d "@$DAMAGED_AT" -Is

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

The listing is sorted oldest first, so the newest is at the bottom and the eye lands on it. Resist that. Read down the Time column and take the last row whose time is earlier than the damage time you just printed.

# Select the newest snapshot strictly older than the damaged file.
SNAP=$(sudo -E restic snapshots --tag daily --json \
  | jq -r --arg damaged "$(date -u -d "@$DAMAGED_AT" +%Y-%m-%dT%H:%M:%SZ)" \
      '[.[] | select(.time < $damaged)] | last | .short_id')
test -n "$SNAP" && test "$SNAP" != null
sudo -E restic snapshots "$SNAP"
echo "chosen recovery point: $SNAP" | tee -a "$LAB/timeline.txt"

Task 4 — Restore into staging, never onto the live path

STAGE="$LAB/staging/$SNAP"
sudo -E restic restore "$SNAP" --target "$STAGE"
sudo ls -l "$STAGE/etc/rbdr-appd/"
sudo stat -c '%n owner=%U:%G mode=%a' "$STAGE$CONF"

$ restic restore into a target directory

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

The restore prints the snapshot time it is unpacking. Read it. It is the cheapest available confirmation that you took the recovery point you meant to take, and it is the last moment before the file starts moving.

Do not assume the staged copy carries the original ownership. Check it. The stat above costs nothing and its answer decides whether Task 7 needs install or a plain copy.

Task 5 — Diff the restored file against the damaged one

sudo diff -u "$STAGE$CONF" "$CONF" | tee "$LAB/appd.conf.diff"
echo "diff exit code: ${PIPESTATUS[0]}"
sudo stat -c '%n owner=%U:%G mode=%a' "$STAGE$CONF" "$CONF"

$ diff -u staged current, then stat both

--- /root/rbdr-lab-09/staging/.../etc/rbdr-appd/appd.conf
+++ /etc/rbdr-appd/appd.conf
-# rbdr-appd configuration
+# rendered by rbdr-deploy
listen_port=8443
-upstream_url=https://orders.internal.example/api
worker_threads=8
log_level=debug

/root/rbdr-lab-09/staging/.../appd.conf owner=root:rbdr-appd mode=640
/etc/rbdr-appd/appd.conf owner=root:root mode=600

The diff is the artefact you attach to the incident record. It is also the check that catches the case where you picked the wrong snapshot: if the diff shows nothing, the snapshot you chose already contains the damage, and you go back to Task 3 with one fewer candidate.

Task 6 — Preserve the damaged file as evidence

EV="$LAB/evidence/appd.conf.damaged.$(date +%Y%m%dT%H%M%S)"
sudo cp -a "$CONF" "$EV"
sudo sha256sum "$EV" "$STAGE$CONF" | tee -a "$LAB/timeline.txt"

cp -a keeps the mode, ownership and timestamps. That metadata is most of the evidence: it says who wrote the file and when, and it is the input to the question everyone asks afterwards, which is how this got deployed at all.

Task 7 — Install it wrongly first, watch it fail, then install it correctly

Run the obvious command. It is the one most people reach for.

sudo cp "$STAGE$CONF" "$CONF"
sudo systemctl restart rbdr-appd || true
systemctl is-active rbdr-appd; echo "is-active exit code: $?"
sudo journalctl -u rbdr-appd --no-pager -n 3
stat -c '%n owner=%U:%G mode=%a' "$CONF"

$ restart after a plain cp, then read the journal

failed
is-active exit code: 3
rbdr-appd: cannot read /etc/rbdr-appd/appd.conf
/etc/rbdr-appd/appd.conf owner=root:root mode=600

cp onto a file that already exists writes through the existing inode, so it replaced the contents and left the deploy tool’s root:root 0600 exactly where it was. The file is now byte-for-byte correct and the service still cannot open it.

sudo install -o root -g rbdr-appd -m 0640 "$STAGE$CONF" "$CONF"
command -v restorecon >/dev/null && sudo restorecon -v "$CONF"
stat -c '%n owner=%U:%G mode=%a' "$CONF"

install sets ownership and mode as part of placing the file, which is why it is the right tool here and cp is not. On a host with SELinux in enforcing mode, a file that arrived from a staging directory under a home directory can carry the wrong label; restorecon puts it back to the policy default for that path.

Task 8 — Prove the service consumed the file, and stop the clock

sudo systemctl restart rbdr-appd
systemctl is-active rbdr-appd; echo "is-active exit code: $?"
sudo journalctl -u rbdr-appd --no-pager -n 3 | grep 'configuration accepted'
T1=$(date +%s)
printf 'Actual restore time: %s seconds\n' "$((T1 - T0))" | tee -a "$LAB/timeline.txt"

$ restart, check is-active, then grep the journal for the acceptance line

active
is-active exit code: 0
rbdr-appd: configuration accepted

is-active returning active says the process is running. The journal line says it parsed the file and found the keys it needs. Those are different claims, and a single-file restore is only finished when you have the second one.

Validation

Each line names the command, the string to expect and the exit code.

grep -c '^upstream_url=' /etc/rbdr-appd/appd.conf
sudo diff -q "$STAGE$CONF" /etc/rbdr-appd/appd.conf
stat -c '%U:%G %a' /etc/rbdr-appd/appd.conf
systemctl is-active rbdr-appd
sudo journalctl -u rbdr-appd --no-pager -n 20 | grep -c 'configuration accepted'
test -s "$LAB/appd.conf.diff"
ls "$LAB/evidence" | grep -c '^appd.conf.damaged.'
grep -c '^Actual restore time:' "$LAB/timeline.txt"

The third row is the one that fails in Task 7 and passes in Task 8, and it is the only row that changes between them. The diff, the row counts and the service status are all identical across both attempts.

Expected Outcome

The service is running on a configuration file that came out of a snapshot you chose on evidence, the damaged file is preserved with its metadata, and the diff between the two is on disk.

Record these two numbers in timeline.txt:

• Actual restore time: measured from T0 in Task 2 to T1 in Task 8. It includes the wrong-ownership attempt, because a real one would.
• Actual RPO observed: the interval between the snapshot you chose and the damage. Any legitimate configuration edit made inside that window — the log_level change, in this fixture — is gone, and you have to know that before you tell anyone the service is back to normal.

Troubleshooting

Cleanup

sudo systemctl stop rbdr-appd
sudo systemctl disable rbdr-appd 2>/dev/null || true
sudo systemctl reset-failed rbdr-appd 2>/dev/null || true
sudo rm -f /etc/systemd/system/rbdr-appd.service /usr/local/bin/rbdr-appd
sudo systemctl daemon-reload
sudo rm -rf /etc/rbdr-appd
sudo userdel rbdr-appd
sudo rm -rf "$LAB/staging" "$RESTIC_REPOSITORY" "$RESTIC_PASSWORD_FILE"

{
  date -Is
  ls -d /etc/rbdr-appd /usr/local/bin/rbdr-appd /etc/systemd/system/rbdr-appd.service 2>&1
  getent passwd rbdr-appd || echo "no rbdr-appd user"
  systemctl is-active rbdr-appd 2>&1
} | tee "$LAB/post-state.txt"

diff <(tail -n +2 "$LAB/pre-state.txt") <(tail -n +2 "$LAB/post-state.txt") \
  && echo "CLEAN: post-state matches the pre-state recorded in Task 1"

timeline.txt, appd.conf.diff and evidence/ are deliverables and are deliberately kept.

Production notes

• The staging path is not optional. It is what lets you diff, and the diff is what turns “I restored a file” into “I know what the restore changed”.
• Preserve the damaged version before you replace it. It is the only copy of the failure, and the post-incident review needs it more than you need the disk space.
• install rather than cp for anything a service reads. Ownership, mode and security context are part of the file, and a restore that returns only the bytes returns most of the file.
• Keep the restore command out of the runbook’s critical path and put the snapshot-selection rule in instead. The command never varies; the choice does, and it is where the time goes.
• Record the measured restore time every time you do this. Single-file recoveries are the only recovery most estates perform often enough to have real numbers for, and those numbers are what make an RTO estimate honest.

What You Learned

• The newest snapshot is a default, not a decision. When the damage is a bad write rather than a delete, the newest recovery point is the one most likely to contain it.
• You need the damage time before you need the repository. File mtime and the service journal give it to you in two commands.
• A file is bytes plus metadata. A byte-perfect restore installed with the wrong owner, mode or security context fails exactly like no restore at all, and it fails with a message that points at permissions rather than at you.
• Diff before you install. It validates the snapshot choice and produces the incident artefact in the same step.
• Restart is not verification. The service being active and the service having parsed the file are two different claims; check the second.