Selecting a clean recovery point

The compromise timeline ends in a single number: the earliest instant at which the estate can be shown to have been under someone else’s control. That number was expensive to produce and it exists for exactly one purpose, which is the decision this lesson makes. Everything before it is a candidate. Everything after it is suspect, including the backups that ran perfectly every night while the intruder was already inside. The gap between having that timestamp and having a recovery point is where most ransomware recoveries quietly go wrong, because the selection gets made by reflex instead of by procedure.

The newest recovery point is the reflex, and the reflex is wrong

Every instinct in an outage points at the most recent backup. It carries the least data loss, it is the one the runbook names, it is the top row of every listing, and choosing it requires no argument with anyone. In a hardware failure that instinct is correct and should be followed without ceremony.

In an intrusion it is precisely inverted. Attacker dwell time — the interval between the first foothold and the moment anybody noticed — is the interval during which every backup faithfully captured a compromised system. The newest recovery point is therefore the one that has had the most attacker activity folded into it, and the oldest surviving one is the least likely to contain anything at all. Recency and cleanliness point in opposite directions, and the tool offers no signal to distinguish them, because from the backup system’s point of view nothing unusual happened: files changed, chunks were stored, the job exited 0.

That is worth stating plainly because it inverts a habit rather than adding to it. The question stops being what is the newest thing we have and becomes what is the newest thing we can show was taken before the compromise. The two questions have different answers, different evidence, and different owners — the first is a matter of reading a listing, the second is a matter of doing work.

Enumerating candidates, including the ones the listing hides

The procedure starts by writing down the earliest-known-good instant from the timeline and then enumerating every recovery point strictly older than it. Both repository tools list recovery points with a stable identifier and a timestamp, and both listings are the input to the selection rather than the selection itself.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Two things in that output deserve attention before anything is selected. The timestamps are one second apart and the sizes are identical at 60.000 MiB, which is a compressed illustration of the real problem: a listing tells you when a capture ran, not what state the source was in when it ran, and nothing in the row distinguishes the snapshot taken before a change from the one taken after it. The footer also says Timestamps shown in local time, while an incident timeline is almost always assembled in UTC. Comparing the two without converting is how a recovery point one hour on the wrong side of the boundary gets chosen.

Borg presents the same information under archive names, which is more readable and slightly more dangerous, because a name like day1 invites people to reason about the label instead of the timestamp beside it.

$ borg list /work/repo

day1                                 Fri, 2026-08-28 13:58:06 [fae87fac26949fa0ab2994a62819407fce5de3418d8d3bc23d4608fd466988c0]
day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

The enumeration is incomplete in one situation that matters enormously here. On a versioned object store, a backup that was deleted during the intrusion does not appear in an ordinary listing at all — and an attacker holding production credentials deletes backups as a matter of course. In the object storage capture, a delete issued by an identity carrying the built-in readwrite policy exited 0 with the words Created delete marker, after which the plain listing of that object returned nothing, also at exit code 0. The bytes had not moved.

$ mc ls --versions prod/rbdr-immutable/backup-0900.tar

[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar

>>> exit code: 0

The zero-byte marker at v2 sits on top of a 38 B object at v1, and v1 has its own version id, 133fd99f-1f98-41c0-9d08-95e6e2944157, by which it can be read and restored. A candidate list built from the ordinary listing would have omitted that recovery point entirely, and the omission would have looked exactly like the backup never existing. Enumerate versions, not objects, and do it before concluding that history is too shallow.

Verifying the candidate instead of trusting its timestamp

Having a candidate is not having an answer. The timestamp says the capture ran before the earliest-known-good instant; it does not say the timeline was right. Scoping produces a floor based on the evidence that survived, and attackers delete evidence, so the honest reading of an earliest-known-good timestamp is no earlier confirmed activity, not no earlier activity. The candidate is therefore a hypothesis, and the way to test it is to restore it somewhere it can do no harm and go looking for the same indicators scoping found.

Isolation is a hard requirement, not a preference. The restore target must have no route to production, no shared credentials, and no outbound path the restored system could use to call home. Everything that follows assumes a throwaway host or virtual machine on an isolated segment, brought up with the restored filesystem mounted rather than booted, wherever mounting is enough.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00

>>> exit code: 0

$ md5sum -c source.md5
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK

>>> md5sum -c exit code: 0

That restore proves fidelity: seven of seven files, 60.000 MiB of 60.000 MiB, every checksum matching what the source held at capture time. It proves nothing whatsoever about whether the source was clean at capture time, and conflating those two proofs is the central error this lesson exists to prevent. The sweep for indicators is a separate pass, run against the restored tree with the list scoping produced.

ROOT=/srv/forensic-restore/3fe43af4
EKG_DATE='2026-08-28 13:27'

# Accounts present in the candidate but absent from the pre-incident baseline.
awk -F: '$3 >= 1000 { print $1 }' "$ROOT/etc/passwd" | sort > /tmp/candidate-users
comm -13 /tmp/baseline-users /tmp/candidate-users

# Units and cron entries written after the earliest-known-good instant.
find "$ROOT/etc/systemd/system" "$ROOT/etc/cron.d" \
     -type f -newermt "$EKG_DATE" -printf '%TY-%Tm-%Td %TH:%TM %p\n'

# Package-owned binaries whose contents no longer match the distribution.
chroot "$ROOT" /usr/bin/dpkg --verify

Four indicator classes carry most of the weight: persistence mechanisms such as units, timers and cron entries; modified or added binaries in the paths the service actually executes; accounts and authorised keys that no ticket explains; and scheduled tasks whose schedule nobody recognises. Each one is checked against a baseline rather than against intuition, which is why the baseline — a package manifest, a user list, a signed inventory — is worth capturing on a good day. A candidate passes when the sweep finds none of them and somebody writes down which checks were run.

Presenting the trade as priced options

Once a candidate verifies, one question remains and it is not a technical one. Every step further back buys confidence and costs data. Nobody in the recovery team is entitled to decide how much of the business’s data to discard, and nobody outside it can price the confidence. The engineering job is to make the trade legible, then hand it to the person who owns the consequence.

That means options, not a recommendation. Take a stated architecture: a nightly restic snapshot at 01:00 UTC, retained daily for 30 days, with the earliest confirmed attacker activity placed at 14:31 on the 20th and the first production write of each day at 08:00. The candidates and their prices follow arithmetically from those assumptions and from nothing else.

Three properties make that table usable. The loss is in hours and in units the business already thinks in, because “13.5 hours” and “one working day of orders” land differently in the same room. The confidence column states what was actually done, not an adjective. And the recommendation is absent by design: the recovery team says what each option costs and what evidence backs it, and the accountable owner picks. Record the choice, the timestamp it was made, who made it, and which candidates were declined — that record is what makes the decision defensible in a post-incident review, and it cannot be reconstructed afterwards.

Retention is what limits the length of that table, and a retention policy is not a neutral cost setting once you read it this way. The Borg capture makes the point in one line, with a policy nobody would call aggressive.

$ borg prune --list --dry-run --keep-daily=1 /work/repo

Keeping archive (rule: daily #1):            day2                                 Fri, 2026-08-28 13:58:07 [85d3e533e094eb96663fd26a148b14737667bfc4251eea6b7144827ea2d279b8]

>>> exit code: 0

One archive kept, and it is day2 — the newer one, which under the reasoning of this lesson is the candidate most likely to be contaminated. A retention policy expressed purely as keep the most recent N is optimised for the failure mode where recency is a virtue, and every ransomware recovery is the failure mode where it is not.

When nothing predates the intrusion

Sometimes the table has no rows. The earliest-known-good instant is 40 days back, retention on the affected systems is 14 days, and every surviving recovery point sits inside the compromise window. This is not a rare edge case; it is the normal outcome when dwell time exceeds retention depth, and dwell time is measured in months more often than anybody would like.

The first thing to say about that situation is what it retrospectively proves. Retention depth is a security control. It is bought and budgeted as storage cost, reviewed by people looking at a spend line, and reduced by whoever needs to make a number smaller this quarter — and the property it actually buys is the maximum dwell time the organisation can survive with its data intact. Thirty days of retention is a thirty-day ceiling on that. The argument for depth is not “we might need an old file”; it is that an intrusion discovered later than the shallowest tier is an intrusion the backups cannot answer.

When the situation is already here, the recovery changes shape rather than failing. It splits in two. Systems are rebuilt from source — infrastructure as code, configuration management, packages from signed repositories, images built from a pipeline — so that the running estate derives from artefacts that were never on the compromised hosts. Then data is extracted selectively from the contaminated recovery points: the database rows, the object store contents, the user documents, moved as data into the rebuilt systems rather than restored as filesystems on top of them. Nothing executable crosses that boundary. No binary, no unit file, no cron entry, no shell profile, no container image.

That split is slow, it is manual, and it is the reason the previous lessons insisted that the ability to rebuild from code be tested while nothing is burning. It is also the reason credentials are treated separately: a recovery point chosen before the intrusion still contains every secret the attacker had access to afterwards, so restoring it restores compromised material. Rotation belongs inside the recovery sequence, not in a follow-up ticket.

Production discipline

1. Start from the timeline, never from the listing. The input to selection is the earliest-known-good instant; the candidate is the newest recovery point strictly before it. In the restic capture two snapshots one second apart carried identical sizes and identical tags — a listing distinguishes recovery points by time, and by nothing else that matters here.
2. Convert timezones before comparing anything. restic snapshots printed Timestamps shown in local time while the timeline is assembled in UTC, and a one-hour error in that comparison selects a recovery point on the wrong side of the boundary while looking entirely correct.
3. Verify the candidate in isolation before committing to it. A restore that returns Restored 7 files/dirs (60.000 MiB) at exit code 0 has proven fidelity, not cleanliness. Mount it away from production and sweep it for the persistence mechanisms, modified binaries, added accounts and scheduled tasks that scoping identified.
4. Enumerate versions, not objects, before declaring history too shallow. Measured on MinIO RELEASE.2025-09-07T16-13-09Z, a deleted backup was absent from the plain listing at exit code 0 and still readable at version id 133fd99f-1f98-41c0-9d08-95e6e2944157 under the delete marker.
5. Hand the trade over as priced options with a named owner. State each candidate’s data loss in hours and in business units, state what verification found, decline to recommend, and write down who chose and what they declined — then treat retention depth as the security control that decides how many rows that table can ever have.

Cross-course references

• Observability for Production Sysadmins — Part XL (Log Retention) governs how far back the earliest-known-good timestamp can be established at all, which is the input this entire selection procedure depends on: if log retention is shorter than dwell time, the timeline has no verified floor and every candidate below it is a guess rather than a measured choice.
• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) turns the discrete candidate list used here into a continuous one, so for a database the selection becomes a recovery target time rather than a snapshot id, and the data-loss column in the options table drops from a backup interval to a transaction boundary.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) covers the step this lesson deliberately refuses to fold into the restore: a recovery point taken before the intrusion still carries keys, tokens and certificates the attacker later held, so rotation has to be sequenced into the recovery rather than scheduled after it.

Quiz