Recovering without restoring the persistence an attacker left

Selecting a recovery point from before the intrusion settles which copy to use, and it quietly raises the harder question of what to do with it. The two get collapsed into one decision, and that collapse is where ransomware recoveries go wrong. The entire value of a backup tool is fidelity — it returns what it was given, byte for byte, mode bits and ownership included — and fidelity has no opinion about which of those bytes you wanted back. Restore whole systems from a compromised estate and the restore is faithful to the compromise. Restore whole systems from a point before it and the restore is faithful to the weakness the attacker later used. Neither is an ordinary restore, and neither is fixed by choosing a better date.

The foothold is a set of files, and files are what backups are good at

An intrusion is not primarily a running process. By the time anyone is looking at it, the interesting part of the attacker’s presence is a set of files on disk: a unit file, a line added to a crontab, a key appended to an authorized_keys, a replaced binary, a library named in a preload list, a plugin dropped into a directory the application scans at start-up. None of them is exotic. They have owners, modes and timestamps like everything else, and several of them are files that are supposed to exist and merely carry one line more than they should.

A backup captures those files the way it captures the rest, because there is nothing about them to distinguish from the configuration surrounding them. The archive then becomes an exact record of a compromised system, and a restore is the mechanism that puts that record back on a machine. The tool will not help here and was never designed to: there is no flag that means everything except the parts the intruder added, because the tool has no idea which parts those were. This is what makes the phrase “restore from backup” so misleading in a ransomware plan. It describes an operation whose defining property — reproduce the source exactly — is the property you need to suppress.

The obvious answer is to reach further back, and the previous lesson did exactly that: establish a compromise window and choose a point outside it. That removes the artefacts, and it is necessary. It does not remove the way in. Initial access used something that was true of the estate before the attacker touched it — an unpatched package, a service reachable from a network it should never have been on, a password that had been valid for three years, a token committed to a repository. Every one of those was present in every copy you hold, including the ones from a year ago, because they were part of the running system’s normal state. A recovery that faithfully reproduces the estate reproduces the door as well, and the second intrusion does not have to be as clever as the first.

Rebuild the platform, restore only the data into it

The strategy that answers both problems at once is to stop treating recovery as one supply line and treat it as two. The platform — the operating system and its packages, the kernel, the service definitions, the container images, the network and identity configuration — comes from code and from known-good images, and it is built rather than copied. The data — the database contents, the object store, the uploaded files, the ledgers, the accepted-but-unprocessed messages — comes from the backup, and it is the only thing that does.

This is not a philosophical split. It is a statement about where each byte came from. Anything a definition can regenerate is regenerated, which means it arrives in the state the definition describes rather than the state a compromised machine happened to be in. Everything else accumulated inside the application over time, is described nowhere, and can only come back as a copy. Drawing the line that way removes the attacker artefacts by construction rather than by detection, because there is no path by which a file the definition never mentions can appear on the rebuilt host.

Operationally the change is smaller than it sounds, and it shows up as a change in what the restore command is pointed at. You stop restoring a server and start restoring a path. A host is installed from a current image, converged from the configuration repository at a commit you have reason to trust, patched to today rather than to the recovery point, and only then does the archive get opened — into one directory that the rebuild created.

set -euo pipefail

# The rebuilt host already owns everything outside this path: packages, unit
# files, accounts, keys. Only the application data is restored into it.
ARCHIVE=/srv/recovery/media/app-data-2026-08-21.tar.gz
DIGESTS=/srv/recovery/media/app-data-2026-08-21.sha256
TARGET=/var/lib/app/data

install -d -o app -g app -m 0750 "$TARGET"
sha256sum -c "$DIGESTS"
tar xzf "$ARCHIVE" -C "$TARGET"

Whatever else that archive contains is not restored, because it is not asked for. The extraction target is a directory the rebuild produced, the digest check happens before anything is unpacked, and the ownership on the target was set by the rebuild rather than carried in from the copy.

What must not come back from a backup

The list is worth writing down before an incident, because during one it will be argued about under time pressure by people who want the service back.

System binaries and libraries. Everything under the package manager’s control comes from the package manager, not from the archive. A replaced /usr/sbin binary and a legitimate one are the same kind of object to a backup, and a package integrity check is a detection control with a false-negative rate, not a guarantee. Reinstalling is cheap; auditing a restored root filesystem is not.

Service unit files and scheduled tasks. Units and drop-ins under /etc/systemd/system, user units under a home directory, timers, entries in /etc/cron.d and the per-user spools, and anything else that runs on a schedule with nobody logged in. This is the classic persistence site precisely because it executes unattended, and it is also fully described by the configuration repository, so restoring it buys nothing.

User accounts and authorised keys. The account database, the group and sudoers files, and every authorized_keys in the estate. Accounts come from the identity source of truth; keys come from whatever enrolment process issues them. An account added during the intrusion, or an extra public key appended to an existing file, survives a restore perfectly and is invisible in any listing that does not diff against a known-good version.

Container images and the registries that hold them. An image is a build output, so it is rebuilt from its source at a pinned base and re-pushed by the pipeline. If the attacker could push to the registry, then pulling a tag — or a digest recorded after the compromise began — reinstalls their work with a green deployment status. The reflex to docker commit a compromised container is the worst version of this, because it captures exactly the writable layer the attacker wrote into.

Anything that can execute. Shell profiles and profile.d fragments, ld.so.preload, the PAM stack, web server configuration with execution directives, application plugin directories, database extensions and event triggers, repository hooks, pipeline definitions. These live among configuration files and look like configuration, and each of them is a place to put a line that runs.

set -euo pipefail

# Mounted read-only, so enumerating the recovery point cannot execute anything.
SRC=/mnt/recovery-point

find "$SRC/etc/systemd/system" "$SRC/etc/cron.d" -type f
find "$SRC/root" "$SRC/home" -maxdepth 3 -name authorized_keys -type f
find "$SRC" -xdev -type f -perm -4000
ls -l "$SRC/etc/ld.so.preload" "$SRC/etc/profile.d" || true

The genuinely hard part is the boundary. “Data only” is a line you draw, not one the filesystem draws for you, and the awkward cases are the ones where data carries executable content: an uploads directory that will happily hold a script the web server is configured to run, a database row holding a serialised object the application deserialises, a working tree whose hooks directory travels with it. For those the rule is that the data is restored to a location where the rebuilt platform will not execute it, and the platform’s willingness to execute it is fixed in the definition rather than inherited from the copy.

Rotate before the rebuilt systems come online

A rebuilt host is clean of the attacker’s files. It is not clean of the attacker’s knowledge. Everything that was readable from a compromised host must be assumed to be in someone else’s hands: service account passwords, API keys in environment files and unit drop-ins, SSH private keys and the host keys clients have pinned, database passwords, TLS private keys, keytabs, tokens the workload had exchanged for longer-lived credentials, and the credentials the backup job itself used. A pristine system that boots holding a stolen identity is a correctly built system that will authenticate the intruder exactly as it authenticates the application — with no exploit, no anomaly and no alert, because the authentication succeeds.

That makes the timing part of the design rather than a follow-up task. Rotation that finishes after the estate is serving traffic leaves a window whose length is a decision somebody made, and it is a window in which re-entry costs the attacker nothing. The order that closes it runs roughly like this.

First, the credentials that protect the backups, including the repository credentials, the object storage keys, and the encryption passphrase where the key schedule permits it. The recovery depends on that source, and an adversary who still holds those keys can destroy or alter the copies you are in the middle of restoring from. Second, the credentials that mint other credentials — the certificate authority keys, the secret platform’s root and unseal material, the cloud administrative identities, the source-control and pipeline tokens. Rotate these late and everything rotated earlier can simply be reissued by whoever still controls the issuer. Third, the service accounts and API keys the rebuilt systems will consume at start-up, which is the set that has to be in place before the first host boots rather than after it. Fourth, human accounts and their second factors, with re-enrolment rather than a password reset where the factor itself may have been captured. Fifth, as hosts appear, the per-host material: new host keys, new agent enrolment tokens, new node certificates.

The isolated recovery environment, and the shape the capture shows

All of this happens somewhere, and the somewhere matters. The rebuild and the data restore run in an environment with no route to and no route from the compromised production network, with its own identity plane, its own DNS and time source, its own package and image mirrors, and read-only access to the backup repository through an identity minted for this recovery and valid only for it. The reason for each of those is the same: a rebuild that pulls from production infrastructure has a production dependency, and production is the thing under the attacker’s control. It is also the only place you can afford to be wrong. If a restored dataset turns out to carry an executable payload after all, it detonates in a network with nothing to reach and nothing to authenticate against.

The environment has to exist before the incident, because building one during an incident means building it with credentials from a directory you no longer trust. What it looks like in practice is the pattern the Kubernetes capture puts on record. The namespace, the claim, the ConfigMap and the pod were re-applied from the manifest, and the volume the rebuild produced was empty until a separately captured archive was unpacked into it.

$ kubectl apply -f rbdr-shop.yaml

  namespace fully removed after 0s
pod Ready after 6s
namespace recreated from YAML; volume is empty:
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:35 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:35 ..

--- restore the volume contents into the NEW volume directory ---
new PersistentVolume directory: /var/lib/rancher/k3s/storage/pvc-ef5cf541-eebf-4eb3-9c91-cb4e59d6d787_rbdr-shop_rbdr-orders
$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Read as a disaster-recovery exercise that transcript is a warning that a repository is not a backup. Read as a security exercise it is the target shape. Every object in the namespace came from a definition rather than from a copy of the compromised cluster, the volume came up empty, and exactly one thing was restored into it: the bytes the application had produced, arriving at the md5 they left with. There was no route by which a file nobody declared could appear in that namespace, which is the property the whole strategy is buying.

The same split shows at container scale, and it makes the boundary concrete. In the Docker capture a file was written into the running container’s own filesystem and another into the mounted volume. Rebuilding the container and restoring only the volume brought back one of them.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

The orders came back byte-identical and /etc/app-marker did not, because the container layer was rebuilt from the image and only the volume was restored. In an ordinary recovery that missing file is a gap to close. After a compromise it is the design working: whatever had been written into the container’s own filesystem, by the application or by anyone else, was not carried forward.

One caveat rides along with this and it is not small. The rebuild is only as good as the definition it runs, so the definition needs the same scrutiny the recovery point got. If the intruder held commit access to the infrastructure repository, or push access to the registry the manifests reference, then “rebuild from code” reinstalls their work with a healthy sync status and a clean audit trail. The definition is checked out at a commit that predates the compromise window, its signatures are verified, and the images it names are rebuilt from source rather than pulled by a tag whose contents can have moved.

Production discipline

1. Rebuild the operating system, service and image layers, and restore only paths you can name. The restore step should take an explicit target directory that the rebuild created, never a whole root filesystem. If nobody can name the paths for a service, that is the finding, and it is findable today rather than during an incident.
2. Write the do-not-restore list per service before you need it. Binaries and libraries, units and scheduled tasks, accounts and authorised keys, images and registry contents, and everything that can execute. Record where each of those comes from instead, so the answer during the incident is a lookup rather than a debate.
3. Complete the rotation before the first rebuilt host boots, backup credentials first. Then the credential issuers, then the service accounts the rebuild will deploy, then human accounts and factors, then per-host material as hosts appear. A rebuilt system holding a stolen credential authenticates the intruder without raising anything.
4. Recover in an environment with no path to or from the compromised network. Its own identity, DNS, time, package and image mirrors, and a read-only backup identity minted for the recovery. Build it in advance; one built during an incident is built from a directory you no longer trust.
5. Verify the definition as carefully as the recovery point. Pin the configuration repository to a commit outside the compromise window, verify its signatures, and rebuild images from source rather than pulling tags, or the rebuild becomes the attacker’s redeployment mechanism.

Cross-course references

• Kubernetes for Production Sysadmins — Part CXXIX (Security Incident Response) is the cluster-side procedure this lesson’s strategy plugs into: it covers isolating a compromised workload and rotating ServiceAccount tokens, which is the step that has to complete before the manifests here are re-applied, or the rebuilt objects come up holding credentials the intruder already read.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) supplies the rotation mechanics this lesson only orders, and Part XVI (Rotation Without Outage) explains why the rotation normally gets deferred; a rebuild removes that objection, which is why the two have to be planned as one piece of work rather than sequentially.
• Ansible for Production Sysadmins — Part XLVII (Controller Security) is the precondition for the rebuild leg being trustworthy at all. The machine that reconstructs the estate holds credentials to every host in it, so if the controller was inside the compromise window, converging from it reinstalls the intrusion at fleet scale in a single run.

Quiz