Preserving evidence while restoring service

Choosing a recovery point that predates the intruder’s persistence keeps them out of the system you bring back. It does not tell you how they arrived, which credential they used, or whether the same door is still standing open on the forty hosts you did not restore. Those answers live inside the machines you are about to rebuild, and the ordinary sequence of a recovery — reboot, reimage, restore, return to service — removes most of them in the first twenty minutes, irreversibly and without anybody deciding to. This lesson is about the sequencing that lets both jobs happen, and about being honest that the two jobs genuinely compete.

Two jobs, one set of machines

Recovery is a programme of deliberate change: stop the affected services, wipe what is suspect, put known-good bits in its place, start again. Investigation is the opposite programme. It wants the machines exactly as the incident left them, because every artefact it works from is a side effect of what happened rather than something anybody chose to record.

Run the recovery programme first and the list of casualties is long and predictable. A reboot ends the process table, the open file descriptors and every established socket, and takes the contents of memory with them. A reimage ends the filesystem, including deleted-but-not-yet-overwritten blocks, the shell history, the package timestamps and the file that was dropped in /tmp and never removed. Log rotation ends the window silently and on a schedule that has nothing to do with the incident. Even benign remediation destroys state: a credential rotation ends the sessions that would have shown which credential was in use, and a repository repair — the subject of the next section — removes the very records that describe the damage it is repairing.

The competition is real, and it is not symmetric. Service restoration has a clock attached to it, a business owner asking about it, and a visible cost per minute. Investigation has none of those, so in any argument about priority it loses, and the loss is permanent in a way the delayed restore never is. Which means arguing about priority is the wrong move. The move that works is to make the preservation step cheap enough — seconds, not hours — that it stops competing for the clock at all, and then to choose a recovery shape that preserves the expensive evidence for free by simply not touching it.

The order the evidence disappears in

Evidence has a half-life, and the capture order follows it: take the thing that will be gone soonest, first. RFC 3227 sets this out as an order of volatility, and for the systems an infrastructure engineer actually operates it collapses to four steps.

Memory, if the capability already exists. Memory holds the things nothing else does — the decrypted payload, the key material, the process that never touched disk. It is also the step most teams cannot perform, because acquiring it needs a hypervisor that can snapshot guest RAM, or an acquisition tool that was installed before the incident. If neither is in place at 03:00, it does not get invented at 03:00. Say so plainly in the incident channel and move to step two rather than spending the window on it.

Running process, network and mount state. This costs seconds and is available on every host: what is running, what is listening, what is connected, what is mounted, and which binaries the running processes were started from. It disappears completely at the next stop or reboot, and it is the step most often skipped because it feels redundant next to a disk image — which it is not, because a disk image cannot tell you what was running.

Logs that rotate. Journald has a size and time budget, logrotate runs on its own schedule, and container logs vanish with the container. Anything already shipped to a central log store is comparatively safe and can wait; anything that exists only on the host is on a countdown that started before you arrived.

Disk. The least volatile and the most expensive to copy, which is the pair of properties that makes the sequencing decision for you: you do not have to image it if you do not destroy it.

INCIDENT=INC-2026-0828-01
EVIDENCE=/evidence/$INCIDENT
mkdir -p "$EVIDENCE"

# running state - gone at the next stop or reboot
ps auxww          > "$EVIDENCE/ps.txt"
ss -tunapo        > "$EVIDENCE/ss.txt"
lsof -nP          > "$EVIDENCE/lsof.txt"
cat /proc/mounts  > "$EVIDENCE/mounts.txt"

# logs that rotate - the window closes on a schedule, not on demand
journalctl --since "-14 days" -o export > "$EVIDENCE/journal.export"
tar -cf "$EVIDENCE/varlog.tar" /var/log

Two honesty notes about that block. Collecting evidence is itself a change: it allocates memory, writes files and moves access times. Write to a mount that is not the one under investigation, and record that you did it, so the artefacts you created are distinguishable from the ones the incident created. And the order above is a default, not a law — if the only copy of a log is about to be rotated in four minutes, that log is more volatile than the process table today.

cp -al before remediation, measured

The Borg capture taken for this course is an incident in miniature, and the useful part is not the attack but the repair. A client holding ordinary repository credentials deleted three archives from a repository configured with append_only = 1.

$ borg delete /work/aorepo::day1

  
>>> exit code: 0
$ borg delete /work/aorepo::day2
>>> exit code: 0
$ borg delete /work/aorepo::day3
>>> exit code: 0

--- what the attacker now sees ---
(empty)

Append-only forbids COMPACTION. The segments are still on disk:
repository data still occupying: 41M	/work/aorepo/data

--- the transaction log after the attack ---
transaction 5, UTC time 2026-08-28T13:58:09.445859
transaction 9, UTC time 2026-08-28T13:58:09.662365
transaction 13, UTC time 2026-08-28T13:58:09.863420
transaction 17, UTC time 2026-08-28T13:58:10.262693
transaction 21, UTC time 2026-08-28T13:58:10.480675
transaction 25, UTC time 2026-08-28T13:58:10.686367

Append-only did not refuse the deletes; all three returned exit code 0 and the archive listing came back empty. What it forbids is compaction, which is why 41M of repository data was still on disk afterwards and why the deletion is reversible at all. Look at what else survived: six transactions, the first three written when the archives were created and the last three at 13:58:10 when they were destroyed. Nobody configured that. The repository wrote the incident timeline itself, as a side effect of doing its job, and those last three transactions are the only on-system record that the deletes ever happened.

Now the repair. Rolling back to the last good transaction means removing the repository hints, index and integrity files and every segment numbered above the good transaction, which was 13.

$ cp -al /work/aorepo /work/aorepo-evidence

  hard-link copy taken for forensics

$ rm -f /work/aorepo/hints.* /work/aorepo/index.* /work/aorepo/integrity.*
$ rm segment files numbered above $GOOD
removed segment 14
removed segment 15
removed segment 16
removed segment 17
removed segment 18
removed segment 19
removed segment 20
removed segment 21
removed segment 22
removed segment 23
removed segment 24
removed segment 25

Read the two blocks against each other. Segments 14 to 25 are the range the rollback removed, and transactions 17, 21 and 25 — the attacker’s three deletes — live inside that range. The remediation that brought the recovery points back is the same operation that erased the record of how they were lost. That is not a flaw in the procedure; the procedure is correct and it worked, and after clearing a client cache that was newer than the repository, the three archives listed again and extracted at exit code 0.

The record survived anyway, because of one line executed before the first rm: a hard-link copy of the whole repository into /work/aorepo-evidence. That copy took no measurable time, consumed no meaningful space, and was the difference between an investigation that can see the attacker’s transactions and one that cannot.

Restore forward, onto infrastructure that was never involved

The second decision is architectural rather than procedural, and it resolves more of the conflict than any capture step does: build the recovered service on new infrastructure, and leave the affected systems powered off and untouched.

Restoring onto new infrastructure buys three things at once. The most expensive evidence — the disks — is preserved at zero cost and zero time, because nothing touches it. The recovered service inherits nothing you failed to find, which is the argument the previous lesson made about persistence. And the rebuild is performed from sources you can review, so the recovered estate has a provenance rather than a history.

Powering the old systems off is a real trade and should be named as one. It ends memory and running state, which is why the volatile capture comes first and why the decision to power off is taken after that capture rather than before it. What powering off buys is the disk, frozen at the moment of the shutdown, with no further writes from a service you no longer trust. A machine left running “in case we need to look at it” keeps writing logs, rotating them, expiring caches and possibly still executing whatever the intruder left behind.

The cost is capacity: somewhere to build the replacement while the original sits idle. That is a decision made when the recovery estate is designed, not at 03:00, and a recovery plan that assumes the compromised hardware will be reused has quietly assumed the investigation away.

A chain of custody an infrastructure engineer can actually keep

Chain of custody sounds like a courtroom word and mostly is one, but the part that falls to the person holding the root shell is small and entirely mechanical: record who took each artefact, when, with which exact command, and what its hash was at the moment of collection.

INCIDENT=INC-2026-0828-01
EVIDENCE=/evidence/$INCIDENT

( cd "$EVIDENCE" && find . -type f -exec sha256sum {} + ) > "$EVIDENCE.sha256"

{
  printf 'incident:     %s\n' "$INCIDENT"
  printf 'collected_by: %s\n' "$(id -un)"
  printf 'collected_at: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'source_host:  %s\n' "$(hostname -f)"
} > "$EVIDENCE.manifest"

The hash has to be taken at collection time. A hash computed a week later, after the artefacts have sat on a shared filesystem that half the team can write to, proves that nothing changed during that week — which is not the week anybody is asking about. Recording the command matters for the same reason the transcripts in this course record exit codes: tar -cf and tar -czf --exclude produce different artefacts, and six months on nobody remembers which was run.

The last obligation is to prove the preserved copy is readable, which is the same work as proving a backup is restorable and uses the same tools. In the restic capture, one copy of a repository had a damaged pack and lost a file on restore; a second copy of the same repository was put through a full read.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-2505547008
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

All 7 packs read, no errors were found, exit code 0. An evidence copy that has never been read back is in exactly the position of a backup that has never been restored: it is an assumption wearing the costume of a fact.

Production discipline

1. Capture before you change, and make the capture cheap enough to always happen. The running state block above runs in seconds. A step that takes seconds does not compete with the restore clock and therefore never gets argued away.
2. Take the hard-link copy as the first move of any repository remediation. In the measured rollback, cp -al /work/aorepo /work/aorepo-evidence ran one command before the rm that removed segments 14 through 25, and that is the only reason transactions 17, 21 and 25 still exist.
3. Restore onto new infrastructure and leave the affected systems powered off. Powering off costs memory and running state, which is why it follows the volatile capture; it buys the disk intact, at no time cost, for as long as the investigation needs it.
4. Record who, what, when and which command, and hash at collection time. The manifest is four printf lines and a sha256sum run. A hash taken later attests to the wrong interval.
5. Escalate the questions that are not yours, and preserve the options while you wait. Reporting duties and retention obligations belong to counsel. The engineering obligation is to ensure that whatever they decide is still possible when they decide it.

Cross-course references

• Observability for Production Sysadmins — Part XL (Log Retention) decides how much of the incident timeline still exists by the time the incident is noticed, and it is what makes this lesson’s third capture step urgent or unnecessary: logs already shipped to a central store with a known retention window can wait, while logs that exist only on the affected host are on a countdown that started before anybody was paged.
• Linux for Production Sysadmins — Part III (Filesystems and Files) covers inodes and link counts, which is the whole mechanism behind cp -al here. The same material explains both halves of the trade this lesson depends on: why the copy is instant and nearly free, and why it survives an unlink but shares its bytes — and therefore any in-place overwrite — with the original.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) is the credential half of the same incident. Rotating a compromised credential is itself a state-destroying change, so the rotation runbook and the capture order given here have to be sequenced against one another rather than run in parallel by two teams who each assume the other has preserved what they need.

Quiz