Select a clean recovery point

1 · Prerequisites

Confirm every item is in place before any state change.

• A scoping output that is a table rather than a sentence: one row per affected system, an earliest-known-good timestamp in UTC, the evidence that produced it, and a note saying which other systems share that boundary because they shared an identity.
• Read access to every recovery point catalogue through an identity that is not the one under suspicion, and preferably not through the compromised estate at all — a recovery workstation, an out-of-band path, a separate account on the repository host.
• An isolated verification environment: a throwaway host or virtual machine on a segment with no route to production, no shared credentials, and no outbound path a restored system could use to call home.
• A pre-incident baseline to compare the candidate against — a package manifest, a user list, an authorised-key inventory, a list of expected units and timers. Without it, the sweep for indicators has nothing to be a comparison against.
• The indicator list produced during scoping: the persistence mechanisms, modified binaries, added accounts and scheduled tasks the responders actually found, not a generic checklist.
• The retention policy and the true depth of every tier, including the object store's noncurrent-version retention, because the depth is what decides how many rows the options table can ever have.
• A named decision-maker for each affected service who is authorised to accept a quantified amount of data loss, with a way to reach them out of hours.
• Written evidence-preservation status: storage-layer snapshots or disk images taken, lifecycle rules suspended, log retention frozen. Selection cannot be finalised while the evidence that supports it is still expiring.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the timeline exists and is bounded by intrusion evidence, not by encryption evidence. If the only timestamp anybody can produce is when files became unreadable or when the alert fired, this runbook has no input. The encryption is the last act of the operation; a recovery point chosen against it sits on the wrong side of the boundary by construction.
• · Confirm each boundary came from a channel the compromise did not control. A bound taken from the affected host's own journal is a lead. The same bound corroborated by the forward proxy, the resolver, the identity provider, the firewall or the backup server's catalogue is evidence. Mark each row with which of the two it is.
• · Confirm every timestamp is normalised to UTC and every host's clock offset is recorded. Repository listings commonly print times in the local zone of the machine running them while the incident timeline is assembled in UTC. A one-hour error here selects a recovery point on the wrong side of the boundary while looking entirely correct.
• · Confirm the candidate enumeration includes recovery points that no longer appear in an ordinary listing. On a versioned object store, a backup deleted during the intrusion is hidden behind a delete marker and absent from a plain listing that exits 0. List versions, not objects, before concluding that history is too shallow.
• · Confirm evidence preservation has already happened. Snapshots or images of the affected systems, off-host logs retrieved, lifecycle rules suspended, delete markers left in place. A restore that overwrites the affected volume destroys the intruder's traces and the fallback in the same operation.
• · Confirm the verification environment is genuinely isolated, by testing it rather than by trusting the diagram. No route to production, no reused credentials, no egress. Mount the restored filesystem rather than booting it wherever mounting answers the question.
• · Confirm you have a pre-incident baseline to compare against. A sweep that reports "three accounts and eleven units" tells you nothing without the list of what was supposed to be there. If no baseline exists, say so now, because it changes what verification can conclude.
• · Confirm who signs for the data loss, by name, before any options are priced. The engineering team is not entitled to decide how much of the business's data to discard, and discovering that at the end of the procedure costs an approval cycle at the worst moment.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Start from the timeline, never from the listing. Write down the earliest-known-good instant for the system in front of you, in UTC, with its evidence, before opening any catalogue. The question is not what is the newest thing we have; it is what is the newest thing we can show was taken before the compromise.
2. 2Treat recency as a risk factor rather than a virtue for this class of incident. Encryption is late and loud, intrusion is early and quiet, and every backup taken in between faithfully captured a compromised system. The newest recovery point has had the most attacker activity folded into it and is the most likely to be contaminated; the oldest surviving one is the least likely to contain anything at all.
3. 3Enumerate candidates as every recovery point strictly older than the boundary, from each tier independently. Repository, snapshot, object store and any offline copy have different depths and different failure modes, and a candidate absent from one may exist in another.
4. 4List object-store versions rather than objects, and read the delete markers as evidence. A dated delete marker sitting above an older version records when a credential holding delete rights was used, on a system the victim host does not administer. Removing it to tidy the bucket destroys that timestamp.
5. 5Set a boundary per system, not one date for the estate. Encryption is staged across days, so the observed encryption times order nothing; the host encrypted last may have been the foothold. A single estate-wide date set at the earliest evidence found anywhere is safe and discards weeks of legitimate data on systems nobody touched.
6. 6Propagate a boundary to every system that trusted the compromised identity. The directory service, the configuration-management controller and the CI system push their boundaries downstream, because a credential compromised on one is compromised for everything that accepts it. Systems with independent credentials and no evidence of contact keep their own, later, cheaper boundary — provided you can write down why.
7. 7Name a first candidate: the newest recovery point strictly before that system's boundary. It is a candidate, not an answer. Scoping produces a floor from the evidence that survived, and attackers delete evidence, so an earliest-known-good timestamp honestly means no earlier confirmed activity rather than no earlier activity.
8. 8Restore the candidate into the isolated environment and prove fidelity first. A restore that returns the expected file count and matches checksums taken before the incident has proven that what came out equals what went in. That is a necessary result and it says nothing at all about cleanliness.
9. 9Sweep the restored tree for the specific indicators scoping found, against the baseline. Persistence mechanisms — units, timers, cron entries; binaries in the paths the service actually executes; accounts and authorised keys no ticket explains; scheduled tasks nobody recognises. Record which checks were run, not only what they returned.
10. 10Read the two filesystem timestamps together rather than trusting either alone. Content-modification time is freely settable and is evidence rather than proof; inode-change time is set by the kernel and has no ordinary interface for setting it to a chosen value. A file whose content date reads old and whose inode date reads recent was touched recently.
11. 11Do not expect a repository check to answer the cleanliness question. restic check --read-data and borg check --verify-data recompute hashes and compare them with what was recorded at capture time. An implant that was on disk when the backup ran was hashed like everything else and comes back byte-identical, which the check correctly reports as success.
12. 12Decision point — accept the candidate, or step back one interval. Accept when the sweep found none of the scoped indicators and the baseline comparison is clean. Step back when any indicator appears, when the baseline is missing so the sweep cannot conclude, or when the boundary rests on a single on-host source that the intruder could have edited. Iterating backwards from a verified candidate is a procedure; iterating forwards from an unverified one is a series of incidents.
13. 13Decision point — price the remaining trade as options and hand it over. Every step further back buys confidence and costs data. Present each candidate with its data loss in hours and in units the business already uses, and with what verification actually found. Decline to recommend: the engineering job is to make the trade legible, and the accountable owner picks.
14. 14Decision point — when no recovery point predates the intrusion, change the shape of the recovery rather than lowering the bar. Rebuild systems from source that was never on the compromised hosts — infrastructure as code, configuration management, packages from signed repositories, images from a pipeline — then extract data selectively from the contaminated points as data. Nothing executable crosses that boundary: no binary, no unit file, no cron entry, no shell profile, no container image.
15. 15Sequence credential rotation into the recovery, not after it. A recovery point taken before the intrusion still contains every secret the attacker later held. Restoring it restores compromised material, so rotation is part of this recovery's critical path.
16. 16Record the decision at the moment it is made, before the restore starts. The chosen point by identifier and timestamp, the boundary it was measured against, the candidates declined and their prices, the checks the isolated verification ran, and who decided. Failback and any later dispute both read from this record, and none of it can be reconstructed once the estate has been overwritten.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓Every affected system has its own row: a boundary in UTC, the evidence behind it, and an explicit note of which systems share that boundary through a shared identity.
• ✓Each boundary names the channel it came from, and at least one channel per system is one the compromised host could not write to.
• ✓The chosen recovery point is identified by repository, identifier and timestamp — never by the word "latest" and never by a snapshot label, which is a name somebody chose rather than a fact about time.
• ✓The chosen point's timestamp is strictly earlier than that system's boundary after both have been converted to UTC, and the conversion is shown rather than asserted.
• ✓The candidate enumeration was taken from object versions as well as object listings, so recovery points hidden behind delete markers were considered.
• ✓An isolated restore of the chosen point completed and matched a checksum or file inventory taken before the incident, establishing fidelity.
• ✓The indicator sweep ran against the restored tree with the scoped list and a pre-incident baseline, and its results are written down check by check, including the checks that found nothing.
• ✓The options table exists, states data loss for each candidate in hours and in business units, and records which candidates were declined.
• ✓A named decision-maker accepted the data loss, with the time of that acceptance recorded.
• ✓A service owner has confirmed in business terms what the chosen point contains — the last order, invoice, ticket or transaction present in it — rather than confirming that a restore process started.
• ✓Credential rotation for every secret readable from the affected systems inside their compromise windows is scheduled inside this recovery, not filed as a follow-up.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Nothing in this runbook writes to production, so the rollback is mostly about undoing what a responder does by reflex while working. Take that seriously: those changes outlive the incident and are invisible afterwards.
• ↶Destroy the isolated restore targets, and confirm the verification environment is torn down rather than left running on a segment somebody will later reconnect. A forensic copy of a compromised system is a compromised system.
• ↶Reverse any monitoring silence, disabled alert route, paused backup schedule or suspended replication put in place while investigating. A paused schedule that nobody resumes is the next incident.
• ↶Restore the lifecycle rules and retention settings that were suspended to preserve evidence, but only once the evidence has been copied somewhere with its own retention, and record the date on which they were re-enabled.
• ↶Undo speculative changes made to reach the catalogues — a temporarily broadened credential, an added firewall rule, a mounted repository, an installed tool version — and say so in the record even where each was harmless.
• ↶If a candidate was rejected after verification, keep the record of why. The rejected candidate is part of the evidence that the chosen one was chosen rather than defaulted to, and deleting it makes the decision look arbitrary in review.
• ↶A selection can be revised right up until the restore begins and not afterwards. If new evidence moves a boundary earlier after a restore has started, treat it as a new incident on a partially recovered estate rather than as a correction.

6 · Escalation

When the runbook isn't enough, contact:

• · No system has a boundary supported by anything but the affected host's own logs. Escalate to whoever owns central logging: there is no independent channel, so every candidate below the boundary is a guess wearing a timestamp.
• · Every surviving recovery point sits inside the compromise window. Escalate immediately to the incident owner and the service owners; this is the rebuild-and-extract path, it is slow and manual, and it needs to be resourced from the first hour rather than discovered in the fourth.
• · The candidate verification finds indicators in a point that predates the boundary. Escalate to the investigation lead — the timeline floor is wrong, and every boundary derived from the same evidence class is now suspect.
• · The decision-maker declines to choose, or asks the engineering team to pick for them. Escalate rather than absorbing it. Accepting data loss on behalf of a business you do not own is not a technical decision and will not survive review.
• · The boundary is earlier than the retention depth on a system whose data carries a regulatory obligation. Escalate to whoever owns that obligation before any recovery point is discarded or overwritten.
• · Anything indicates that the backup catalogue itself was accessed or altered by the intruder — unexplained retention changes, missing recovery points with no operator behind them, restore operations nobody performed. Escalate to security and stop treating the catalogue as evidence.
• · The time to complete verification exceeds the recovery time the business is working to. Escalate with the measured remaining time, so that restoring an unverified point becomes somebody's explicit, recorded decision rather than a silent shortcut.

References

1. restic documentation, Working with repositories
2. restic documentation, Restoring from backup
3. BorgBackup documentation, borg list
4. MinIO documentation, Object versioning
5. NIST SP 800-61 Rev. 2, Computer Security Incident Handling Guide
6. NIST SP 800-184, Guide for Cybersecurity Event Recovery
7. RFC 3227, Guidelines for Evidence Collection and Archiving
8. find(1) manual page — newermt, newerct and printf time formats