Restoring a single file under pressure

The previous lesson established that an archive carries only the metadata it was told to carry, and that a file with the right bytes can still be the wrong file. That stops being academic the moment somebody puts a ticket in front of you saying a configuration file is wrong and asking for yesterday’s version back. This is the most common recovery request in any estate and the one most often performed badly — small enough to feel as though it needs no procedure, urgent enough that nobody ever writes one. What follows is the procedure.

Three facts before you touch the repository

A single-file recovery begins with three questions, and none of them are about the backup system.

The first is which path, exactly. A ticket saying “the app config is broken” names a symptom, not a file. Applications read several files in a defined precedence, and the one the reporter has in mind is frequently not the one the service loaded. Resolve the request to an absolute path first, because every later step depends on it.

The second is what happened to it — truncated, overwritten with different content, deleted, or intact but with its ownership or permissions changed. These have different remedies and only one is a restore. A file whose mode changed does not need a copy pulled out of a repository; it needs its mode changed back, and a restore that also replaces the contents quietly discards whatever legitimate edits happened since.

The third question is the one that decides the whole operation: when did the damage occur. The recovery point is a consequence of that answer and of nothing else. Backup schedules capture whatever is on disk when they run. If a bad deploy rewrote the file at 14:10 on Tuesday and the nightly job ran at 02:00 on Wednesday, then Wednesday’s backup contains the damage, faithfully, and restoring from it changes nothing at all. The engineer who does that concludes the backup system is broken, when what actually happened is that the recovery point was chosen by recency instead of by evidence.

Evidence for the damage window comes from outside the backup tool: the file’s own modification time, the service log around the first failed read, deployment and configuration-management history, and the audit trail if the host keeps one. Modification time is the cheapest signal and the least trustworthy: a configuration-management run that rewrote the file with identical content still moved it. Treat mtime as a hypothesis and find a second source that agrees.

Only once you have a window do you list what is available to restore from.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Two snapshots one second apart is an artefact of the capture; in production those rows are a day apart and the question is which falls on the safe side of the damage window. The notice that timestamps are shown in local time matters more than it looks — incident timelines are assembled from logs in UTC, and an hour of offset is enough to pick the wrong row.

The other columns are part of the selection, not decoration. The listing carries a Host and a Paths column — 8211a08b55c3 and /work/prod in the capture — and a repository that receives more than one machine shows rows from all of them interleaved by time. A snapshot from a different host, or rooted at a different path, can hold a file with the same name and none of the same history. Narrow the list to the host and the path the ticket names before reading timestamps off it.

The target is a staging path, and the damaged file is evidence

Restoring into the location the file normally occupies destroys two things at once: the ability to compare the candidate against what is currently there, and the only remaining copy of what the failure actually looked like. Both are needed, and both are gone the instant the write lands.

So the first command of the recovery is not a restore. It is a copy of the damaged file into an evidence directory, with its metadata intact, and a record of the metadata in a form a human can read later.

TARGET=/etc/app/app.conf
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
STAGE=/var/tmp/recovery-$STAMP
mkdir -p "$STAGE/evidence"

# The damaged file is evidence, and it is also the fallback if the restore
# turns out to be worse than what is on disk.
cp -a "$TARGET" "$STAGE/evidence/app.conf.damaged"
stat -c '%n owner=%U:%G mode=%a mtime=%y' "$TARGET" | tee "$STAGE/evidence/app.conf.stat"

That stat line does more work than it looks. It records the ownership and mode the service was running with before the incident, which is the specification the replacement has to be restored to. Reconstructing that afterwards from memory, at speed, is how a recovery causes its second outage.

With the evidence preserved, the restore itself is a restore somewhere else. Both tools support that directly. restic takes a --target directory and is narrowed to particular paths with --include, so one file does not require materialising a whole snapshot; the documentation also covers restic dump for writing a file’s contents to standard output. Borg is narrowed by passing the paths to extract as arguments, and borg extract writes into the current working directory — so with Borg the staging decision is made by choosing where you stand.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

The restore reported seven files and directories; the checksum comparison answered for the three files inside them, and only the second statement is about bytes. Where a file lands beneath the target is also worth reading rather than assuming — a script that hard-codes a layout the tool did not produce creates a directory where you expected a file, and reports success doing it.

Compare, and treat each outcome as information

Restoring one path and then comparing it against the original is the step most single-file recoveries skip.

SNAP=3fe43af4
restic restore "$SNAP" --target "$STAGE" --include "$TARGET"

# Read the layout the tool produced rather than assuming one.
CANDIDATE=$(find "$STAGE" -type f -name app.conf)

if cmp -s "$CANDIDATE" "$TARGET"; then
  echo "identical - this recovery point already contains the damage"
else
  diff -u "$TARGET" "$CANDIDATE"
fi

Three outcomes, each telling you something different.

If the two files differ in the way the report described — the missing stanza is present again, the wrong value is right again — the recovery point is plausible and you can proceed.

If the two files are byte-identical, the chosen recovery point already contains the damage. This is the single most common way a single-file restore wastes an hour, and it is not a tool failure. It is the damage window telling you it opened earlier than you assumed. Go back to the snapshot list, take the next one further back, and repeat; the comparison is cheap precisely because you restored into staging.

If the two files differ in ways nobody described, stop. Something else changed the file legitimately between the recovery point and now, and putting the candidate in place will revert those changes along with the damage. That is a decision for whoever owns the change, not for the person running the restore.

A checksum manifest of known-good content is stronger than a diff against the current file, because it settles the question without reference to a file whose trustworthiness is in doubt. Borg makes the same point from the other side: the extraction says nothing useful, and the verification is a separate command.

$ borg extract /work/repo::day1

  >>> exit code: 0

--- verifying the restored tree against the source checksums ---
./app/orders.csv: OK
./db/data.bin: OK
>>> verification exit code: 0

The extract printed nothing and exited 0. That silence is the argument for the verification step: the command that succeeded told you a process finished, and the command that compared checksums told you the bytes are the ones you wanted. Only the second is evidence.

How far back the search can go is a retention decision

The comparison loop above hides a question. If every byte-identical result sends you one recovery point further back, what stops the search? Retention does, and it was decided weeks earlier by a policy nobody consults during an incident.

Two properties of a deduplicating repository make that policy easy to misread. The first is that a snapshot is a complete view of the source tree rather than a delta, and the listing says so: both rows in the capture reported a Size of 60.000 MiB, the size of the tree each snapshot represents. The second is that the storage does not behave that way at all. The second backup of the same 60 MiB tree added 2.062 KiB (1.370 KiB stored), because the unchanged chunks were already stored; source tree and two-snapshot repository both measured 61M.

Read those two facts together and the consequence follows. An old recovery point is not expensive to keep, and pulling one path out of it costs no more than pulling the same path out of the newest one. The instinct that older snapshots are heavy, and should be thinned first when a repository grows, comes from reading the Size column as repository growth, which it does not report.

What actually removes a recovery point is retention, and it works in two stages that are easy to conflate. forget applies the policy to snapshots. Space is reclaimed only by prune, and a chunk survives for as long as any remaining snapshot still references it. That separation is also why a dry run is the only honest way to change a policy: the capture’s dry run printed Would have removed the following snapshots: {3fe43af4} — the older of the two, and the one an engineer chasing a damage window backwards would have reached for next.

So the number worth recording for a service is not how many recovery points exist but how far back the oldest surviving one reaches, because that is the width of the damage window the estate can still answer questions about. A window narrower than the time it takes somebody to notice a bad configuration change is a gap that stays invisible until a ticket arrives too late.

Putting it back with the metadata the service needs

The final write is where the recorded stat output earns its place. The replacement is installed with the ownership and mode in force before the incident, the security context comes from policy rather than from wherever the file was staged, and both are read back before anyone declares the file fixed.

OWNER=$(stat -c '%U' "$STAGE/evidence/app.conf.damaged")
GROUP=$(stat -c '%G' "$STAGE/evidence/app.conf.damaged")
MODE=$(stat -c '%a' "$STAGE/evidence/app.conf.damaged")

install -o "$OWNER" -g "$GROUP" -m "$MODE" "$CANDIDATE" "$TARGET"
restorecon -v "$TARGET"
ls -lZ "$TARGET"

Then verify the consumer, not the file. Run the service’s own configuration check if it has one, reload or restart it according to how it picks up changes, and confirm from the service side that the value the ticket was about is the value now in effect. A reload that returns success proves the service was signalled, not that it read the file — and where the label is wrong it will routinely do the first without the second.

Production discipline

1. Resolve the request to an absolute path and a damage window before opening the repository. The recovery point follows from the window; a snapshot chosen by recency is a guess with a timestamp attached.
2. Preserve the damaged file, with its metadata, as the first command you run. It is the fallback if the restore is worse, the evidence for the root cause, and the specification for the ownership and mode you will restore to.
3. Restore into a staging target every time, including the times it feels unnecessary. restic restore --target writes wherever you point it and borg extract writes into the current directory, so staging costs one mkdir and buys the comparison.
4. Compare before replacing, and treat a byte-identical result as an answer. Identical means the recovery point already carries the damage and the search has to move further back, not that the tool malfunctioned.
5. Verify the service, not the file. Ownership, mode and security context are separate from content, a process holding an old descriptor keeps reading the old inode, and only the consumer confirms the recovery landed.

Cross-course references

• Linux for Production Sysadmins — Part XXVIII (SELinux and AppArmor) develops the labelling and profile mechanics that this lesson only uses at the surface; it is the material to read when a restored file with correct bytes, owner and mode is still refused to the service that needs it.
• Observability for Production Sysadmins — Part CIX (Incident Investigation Workflows) covers assembling a timeline from logs and deployment history, which is exactly the evidence that fixes the damage window here, and therefore the evidence that selects the recovery point.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part XVI (Restore and Switch) applies the same discipline to files that live in a repository: choosing an explicit revision, restoring one path rather than the whole tree, and inspecting the result before it becomes the working state.

Quiz