Evidence preservation - what recovery destroys

linux-evidence-based-diagnosis establishes the principle: capture before you disturb. This lesson is about the version of that problem you meet under time pressure, when the thing that restores service and the thing that destroys the evidence are the same action, and you have to do it anyway.

The resolution is not to delay the mitigation. It is to know exactly what each action erases, and to have a capture short enough that nobody has to choose.

What each recovery action destroys

The last row is worth dwelling on. A disk-full incident is resolved by deleting logs, and the logs deleted are the ones describing what filled the disk. The correct order is copy, then truncate:

# Substitute your own values before running:
BIG=/var/log/app/debug.log

# Copy the tail off the host BEFORE reclaiming the space
tail -c 50000000 "$BIG" > /var/tmp/incident-tail.log

# Then reclaim without breaking the writer's file descriptor
truncate -s 0 "$BIG"

The sixty-second capture

This is the whole point. If the capture is short enough, the argument between “restore service” and “preserve evidence” does not need to happen.

# Substitute your own values before running:
UNIT=checkout
OUT=/var/tmp/inc-$(date -u +%FT%H%M%SZ)-$(hostname -s)

mkdir -p "$OUT"

# Volatile first - this is what the restart takes with it
ps auxf                       > "$OUT/ps.txt"
ss -tanp                      > "$OUT/sockets.txt"
free -m                       > "$OUT/mem.txt"
uptime                        > "$OUT/load.txt"
df -hl                        > "$OUT/df.txt"
systemctl status "$UNIT" --no-pager -l > "$OUT/unit-status.txt" 2>&1

# The service journal for the window that matters
journalctl -u "$UNIT" --since '-30 min' --utc -o short-iso-precise \
                              > "$OUT/unit.log" 2>&1
journalctl -k --since '-30 min' --utc -o short-iso-precise \
                              > "$OUT/kernel.log" 2>&1

# Clock offset, so these timestamps stay interpretable later
chronyc tracking              > "$OUT/clock.txt" 2>&1

Two additions when the service is misbehaving rather than dead, and you have another thirty seconds:

# Substitute your own values before running:
PID=9188

# What is every thread doing right now
sudo cat "/proc/$PID/status"          > /var/tmp/proc-status.txt
sudo cat "/proc/$PID/limits"          > /var/tmp/proc-limits.txt
sudo ls -l "/proc/$PID/fd" | wc -l    > /var/tmp/proc-fdcount.txt

# Kernel stack of each thread - shows where a hung process is blocked
sudo cat /proc/"$PID"/task/*/stack    > /var/tmp/proc-stacks.txt 2>/dev/null

How long the evidence survives on its own

Evidence has a shelf life, and the shelf life is usually shorter than the time it takes to schedule a post-incident review. Two questions decide it.

Is the journal persistent? A volatile journal lives in /run/log/journal and is gone at the next boot — so a fenced or rebooted host has no journal for the boot that mattered.

# Persistent if this directory exists and is populated
ls -d /var/log/journal 2>/dev/null && echo "persistent" || echo "volatile"

# The configured setting
grep -rE '^\s*Storage=' /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ 2>/dev/null

# How much is retained now, and the configured caps
journalctl --disk-usage
grep -rE '^\s*(SystemMaxUse|MaxRetentionSec|SystemMaxFiles)=' \
  /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ 2>/dev/null

How fast is the window rotating? A size-capped journal on a chatty host can hold hours rather than weeks — and an incident generates far more log volume than normal, so the window shrinks exactly when you need it.

# Oldest entry still present: this is your real retention
journalctl --list-boots | head -1
journalctl -n1 -o short-iso --utc --reverse >/dev/null 2>&1
journalctl -o short-iso --utc | head -1

Security incidents have different rules

A suspected compromise inverts several of the habits above. Escalate to whoever owns security response before acting; the following exists so you do not destroy anything in the minutes before they arrive.

1. Do not reboot or power off. Memory holds the running attacker tooling, injected code, decrypted keys and network state, and none of it survives. Isolate at the network layer instead - a firewall rule, a switch port, a security group - so the host stays running but reaches nothing.
2. Do not reimage, and do not "clean it up". Removing the artefacts removes the evidence of how they arrived, and a restore from a backup taken after the compromise restores the compromise.
3. Minimise your own footprint. Every command you run changes atimes, adds shell history, and creates processes. Log what you ran and when, so the responders can separate your activity from the attacker.
4. Preserve rather than analyse. Copy evidence to write-once or append-only storage, and record who took it, when and from where. Analysis on a copy is always available later; the original state is not.
5. Treat the logs on the host as untrustworthy but still valuable. An attacker with root may have edited them, which is itself evidence. Compare against whatever was shipped centrally before the compromise.

Knowledge check