Execute ransomware recovery

1 · Prerequisites

Confirm every item is in place before any state change.

• A declared incident with a named incident commander, and an explicit statement of who is authorised to disconnect production systems. This runbook takes actions that stop business before anyone knows the full scope, and that authority has to exist before step one rather than be negotiated during it.
• An out-of-band working path: a recovery workstation that has never held production credentials, a channel for the response team that does not run on the compromised estate, and a way to reach the storage and identity control planes without authenticating from an affected host.
• The credential inventory for every affected system — repository passphrases, object-store keys, SSH keys, agent tokens, database roles, cloud roles — or the accepted fact that no inventory exists, which changes the rotation scope from a list to everything.
• Administrative access to the identity and object-store control planes through an account that was not used from, and whose secret was never present on, any compromised host.
• Evidence storage with its own retention and its own credentials: capacity for disk images and log copies, on a target the affected estate cannot write to or delete from.
• A rebuild path that never lived on the compromised hosts — infrastructure as code, configuration management, signed package repositories, an image pipeline — because the recovery reinstalls systems from it rather than restoring them.
• An isolated verification environment with no route to production, no shared credentials and no outbound path, together with a pre-incident baseline (package manifest, account list, authorised keys, expected units and timers) to compare a restored tree against.
• A named data owner per service, reachable out of hours, who is authorised to accept a quantified amount of data loss and to confirm in business terms what the recovered data contains.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Confirm the attacker is still to be treated as present. Encryption is the last act of an operation, not the first, so the working assumption at the start of this procedure is live access with valid credentials. Every later step is ordered by that assumption; if the team is already reasoning as though the intrusion is over, stop and correct that first.
• · Confirm which identities the affected hosts could use, and specifically which of them can delete. A production identity with an ordinary read-write policy holds delete rights on the backup target. That is the capability that turns an encryption incident into an unrecoverable one, and it is the first thing to remove.
• · Confirm whether the recovery points are protected by a retention lock and in which mode. Compliance-mode retention resists the bucket owner and the administrator for the length of the window. Governance-mode retention does not resist a principal holding the bypass permission. The two answers lead to different containment urgency.
• · Confirm that replication, mirroring and sync jobs are stopped before anything else is touched. A scheduled mirror defined as "whatever the source currently contains" will faithfully copy the encryption to the destination on its next run, exit 0, and remove the last readable copy.
• · Confirm evidence capacity and evidence targets exist before any rebuild begins. Disk images, memory where it can be taken, off-host logs, object-store version listings including delete markers. A rebuild that reuses the affected volume destroys the intrusion evidence and the fallback in one operation.
• · Confirm lifecycle and retention expiry are suspended on the evidence and on the surviving recovery points. Both the timeline and the candidate list are being consumed by an expiry clock that does not know an incident is in progress.
• · Confirm that every timestamp the team is working with is normalised to UTC, and record each host's clock offset. A one-hour error selects a recovery point on the wrong side of the boundary while looking entirely correct.
• · Confirm the data owner for each affected service by name, before options are priced. Discovering at hour six that nobody is authorised to accept data loss costs an approval cycle at the worst possible moment.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Remove the delete capability from every credential a compromised host holds, before investigating anything. The attacker is inside with valid credentials while the team is reading logs. Deletion is the only action against the backup estate that cannot be undone, so it is the only one that must be revoked on a timescale of minutes rather than after scope is understood.
2. 2Prefer reducing the capability to revoking the identity, where the choice exists. An identity that suddenly cannot authenticate tells the attacker they have been detected and invites destruction of what they can still reach; an identity that can still write but can no longer delete or overwrite leaves them working while the evidence accumulates.
3. 3Stop every scheduled job that copies current state — mirrors, sync jobs, replication, lifecycle expiry. These do not distinguish an encrypted file from a legitimately changed one. Left running, they finish the attack on schedule.
4. 4Preserve evidence before anything is rebuilt or overwritten. Image the affected volumes or take storage-layer snapshots, copy logs off-host, and list object versions including delete markers. Once a host is reinstalled, the timeline the entire recovery depends on is gone and cannot be reconstructed.
5. 5Leave delete markers in place and read them as evidence. A dated marker above a surviving version records when a credential holding delete rights was used, timestamped on a system the victim host does not administer. Removing it to tidy the bucket destroys that record.
6. 6Scope the intrusion to an earliest-known-good timestamp per system, not one date for the estate. Build each bound from authentication records, file and package changes, scheduled task and unit creation, and outbound connections — and corroborate it wherever possible from a channel the compromised host could not write to.
7. 7Propagate a boundary to every system that trusted a compromised identity. The directory service, the configuration-management controller and the CI system push their boundaries downstream. Systems with independent credentials and no evidence of contact keep their own, later, cheaper boundary, provided the reason can be written in one sentence.
8. 8Select the newest recovery point strictly predating that system's boundary — never the newest available. Recency is a risk factor in this class of incident: every recovery point taken during dwell time is a faithful copy of a compromised system, captured by a backup job that had no reason to report anything unusual.
9. 9Decision point — rebuild the system, or restore it. For any host with confirmed intrusion, rebuild. Restoring a whole machine restores whatever persistence was on disk when the backup ran: units, timers, cron entries, modified binaries, authorised keys, shell profiles. Restoring only data leaves the persistence behind.
10. 10Rebuild infrastructure from source that never lived on the compromised hosts, then bring data across as data. Infrastructure as code, configuration management, packages from signed repositories, images from a pipeline. Nothing executable crosses the boundary from the contaminated recovery point.
11. 11Rotate every credential the compromised environment held, including the backup passphrase path. A recovery point taken before the intrusion still contains every secret the attacker later obtained. Restoring it restores compromised material, so rotation belongs on this recovery's critical path rather than in a follow-up ticket.
12. 12Sequence the repository passphrase rotation deliberately, and prove key access before retiring the old material. Rotating a repository secret while it is the only path to the recovery points is how a recoverable incident becomes an unrecoverable one; escrow the new material to a custodian outside the affected estate first.
13. 13Verify the restored data in isolation, against the specific indicators scoping found. Mount rather than boot wherever mounting answers the question. Sweep for the persistence mechanisms, binaries, accounts and scheduled tasks the responders actually found, compared against the pre-incident baseline — not against a generic checklist.
14. 14Decision point — accept the verified data, or step back one recovery point. Accept when the sweep found none of the scoped indicators and the baseline comparison is clean. Step back when any indicator appears, when no baseline exists so the sweep cannot conclude, or when the boundary rests on a single source the intruder could have edited.
15. 15Connect nothing until the data has been verified and the credentials it references have been rotated. The order is rebuild, restore data, verify in isolation, rotate, then connect. Reordering the last two reconnects a service holding secrets the attacker still knows.
16. 16Have the data owner validate the recovered service in business terms before it is declared recovered. The last order, invoice, ticket or transaction present, checked against where they expected the line to fall — never a statement that a process completed.
17. 17Record the decisions as they are made, not afterwards. The containment actions and their times, the boundary per system with its evidence, the chosen recovery point by identifier and timestamp, the candidates declined, the rotation register, the verification results, and who accepted the data loss.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓Every credential reachable from a compromised host has been enumerated, and each one either had its delete capability removed or is recorded with the reason it was left, the time of the change, and who made it.
• ✓Object-store listings taken by version rather than by object show the surviving recovery points and the delete markers above them, and the markers are still present.
• ✓Every mirror, sync, replication and lifecycle-expiry job touching affected data is confirmed stopped, by inspection of the scheduler rather than by assertion.
• ✓Evidence exists off the affected estate with its own retention: images or snapshots, off-host logs, and version listings, each with a recorded checksum.
• ✓Each affected system has its own row: an earliest-known-good timestamp in UTC, the evidence behind it, the channel that produced it, and the systems that inherit it through a shared identity.
• ✓The chosen recovery point is identified by repository, identifier and timestamp, and its timestamp is strictly earlier than that system's boundary after both are converted to UTC.
• ✓Rebuilt hosts were installed from the declared rebuild path, and the restored payload contains no executable content — no units, timers, cron entries, binaries, shell profiles or container images crossed from the recovery point.
• ✓The isolated indicator sweep ran against the restored tree with the scoped indicator list and the pre-incident baseline, and each check is recorded with its result, including the checks that found nothing.
• ✓The rotation register is complete: every secret the compromised environment held is listed with the time it was rotated and who verified the new material works, and the repository passphrase appears on it.
• ✓A named data owner has confirmed, in business terms, what the recovered data contains, and has accepted the quantified data loss with the time of acceptance recorded.
• ✓No recovered service was connected to production before its data was verified and the secrets it references were rotated.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶This procedure is largely irreversible by design — a rebuilt host cannot be un-rebuilt — so the rollback covers the reversible changes a responder makes by reflex, which otherwise outlive the incident invisibly.
• ↶Restore the credential policies that were narrowed for containment, deliberately and one at a time, only after rotation is complete and only where the capability is genuinely needed. A production identity that never regains delete rights is usually the correct end state rather than a leftover.
• ↶Resume the mirrors, sync jobs and replication that were stopped, and confirm each one actually resumed. A paused schedule nobody restarts is the next incident, and it will be discovered by a failed restore months later.
• ↶Re-enable the lifecycle rules and retention settings suspended to preserve evidence, but only once the evidence has been copied somewhere with its own retention, and record the date on which they were re-enabled.
• ↶Destroy the isolated verification environment and the forensic restore targets rather than leaving them running on a segment somebody will later reconnect. A forensic copy of a compromised system is a compromised system.
• ↶Reverse the monitoring silences, disabled alert routes and paused backup schedules created while working, and say so in the record even where each was harmless.
• ↶If a recovery point was rejected after verification, keep the record of why. The rejected candidate is the evidence that the chosen one was chosen rather than defaulted to.
• ↶A recovery point selection can be revised right up until the restore begins and not afterwards. New evidence that moves a boundary earlier after a restore has started is a new incident on a partially recovered estate, not a correction.

6 · Escalation

When the runbook isn't enough, contact:

• · Any credential that can delete recovery points cannot be constrained within the first hour — the control plane is unreachable, the administrator is unavailable, or the policy model does not separate write from delete. Escalate immediately; this is the failure that decides whether recovery is possible at all.
• · The surviving recovery points sit under governance-mode retention rather than compliance-mode, and the bypass permission is held by an account the attacker may control. Escalate to whoever owns the storage platform: the retention window is not protecting what the team assumes it is protecting.
• · Every surviving recovery point falls inside the compromise window. Escalate to the incident owner and every data owner: the recovery becomes rebuild-and-extract, which is slow and manual and must be resourced from the first hour.
• · The catalogue shows retention changes, missing recovery points or restore operations nobody on the team performed. Escalate to security and stop treating the catalogue as evidence.
• · The isolated verification finds a scoped indicator in a recovery point that predates the boundary. Escalate to the investigation lead: the timeline floor is wrong and every boundary derived from the same evidence class is now suspect.
• · No credential inventory exists, so rotation scope cannot be bounded. Escalate rather than guessing — the honest scope in that case is every secret the environment held, and that is a resourcing decision.
• · The data owner declines to accept a quantified loss, or asks the engineering team to choose for them. Escalate; accepting data loss on behalf of a business you do not own will not survive review.
• · Regulatory, contractual or law-enforcement notification obligations may apply. Escalate to legal and compliance before evidence is altered or a ransom question is answered by anyone technical.

References

1. NIST SP 800-61 Rev. 2, Computer Security Incident Handling Guide
2. NIST SP 800-184, Guide for Cybersecurity Event Recovery
3. RFC 3227, Guidelines for Evidence Collection and Archiving
4. Amazon S3 User Guide: Using S3 Object Lock
5. Amazon S3 User Guide: Working with delete markers
6. MinIO Object Storage for Linux documentation
7. restic documentation, Restoring from backup
8. restic documentation, Working with repositories
9. find(1) manual page — newerct and printf time formats
10. rsync(1) manual page