Evidence capture — what docker restart destroys

The triage lesson said: stabilise first, investigate after. That is right, and it hides a trap. Most stabilisation actions are also evidence-destruction actions, and the evidence they destroy is the only thing that would have let you explain the incident afterwards.

The resolution is not to investigate before stabilising. It is to spend ninety seconds collecting before stabilising. Collection is not investigation — you are not reading any of it yet — and ninety seconds is affordable in almost every incident.

What each action destroys

Three rows deserve emphasis.

docker restart is the most common first action and it silently overwrites the crash state. ExitCode, OOMKilled, Error and FinishedAt describe the most recent exit; once the container is running again they describe nothing useful. The single most valuable fact in an OOM incident — "OOMKilled": true — has a lifetime that ends at your first restart.

docker rm and docker compose down remove the container, and with it the log file the json-file driver was writing. docker logs on a removed container is not a permissions problem or a retention problem; there is nothing there.

And docker compose down -v removes the volumes. That is data, not evidence, and it is the one action on the table that cannot be recovered by any amount of care afterwards.

Capture in order of volatility

Collect the things that disappear fastest, first. Process and memory state vanish the moment anything stops; filesystem state survives until the container is removed; configuration survives indefinitely.

#!/usr/bin/env bash
# Usage: ./capture.sh CONTAINER_NAME
set -uo pipefail

CTR="${1:?usage: capture.sh CONTAINER}"
OUT="/var/tmp/incident-$(date -u +%Y%m%dT%H%M%SZ)-$CTR"
mkdir -p "$OUT"

# --- Most volatile: live process and network state -------------------
docker top "$CTR"                    > "$OUT/top.txt"        2>&1
docker stats --no-stream "$CTR"      > "$OUT/stats.txt"      2>&1

PID=$(docker inspect --format '{{.State.Pid}}' "$CTR" 2>/dev/null || true)
PID=${PID:-0}
if [ "$PID" -gt 0 ]; then
nsenter -t "$PID" -n ss -tanp      > "$OUT/sockets.txt"    2>&1
cat "/proc/$PID/status"            > "$OUT/proc-status.txt" 2>&1
cat "/proc/$PID/limits"            > "$OUT/proc-limits.txt" 2>&1
fi

# --- Container state and history -------------------------------------
docker inspect "$CTR"                > "$OUT/inspect.json"   2>&1
docker logs --timestamps "$CTR"      > "$OUT/logs.txt"       2>&1
docker diff "$CTR"                   > "$OUT/diff.txt"       2>&1
docker events --since 2h --until "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--filter "container=$CTR"          > "$OUT/events.txt"     2>&1

# --- Host context ----------------------------------------------------
dmesg -T | tail -200                 > "$OUT/dmesg.txt"      2>&1
journalctl -u docker --since '2 hours ago' --no-pager \
                                   > "$OUT/dockerd.txt"    2>&1
df -h                                > "$OUT/df.txt"         2>&1
free -m                              > "$OUT/free.txt"       2>&1
docker network ls -q | xargs -r docker network inspect \
                                   > "$OUT/networks.json"  2>&1

echo "captured to $OUT"

Put that script on every host now, not during the incident. The version you write while a service is down will be worse than this one and will take twenty minutes you did not have.

Read the four fields that decide the shape of the incident

docker inspect proj-api-1 --format \
'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}} finished={{.State.FinishedAt}} restarts={{.RestartCount}}'

$ docker inspect proj-api-1 --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'

exit=137 oom=true

Exit codes carry meaning worth memorising:

exit=137 with oom=false and no dmesg OOM line usually means the grace period expired during a docker stop and the daemon escalated to SIGKILL — a shutdown bug, not a memory bug. The two look identical in docker ps and have completely different fixes.

Preserving the filesystem

docker diff lists every path that differs from the image — files written, changed and deleted since the container started. It is quick, small, and often enough on its own: a config file that should not have changed, a lock file that should not exist, a log directory that filled the layer.

For anything more, snapshot the whole filesystem into an image:

# Freeze the container while the snapshot is taken (the default),
# so the filesystem is internally consistent.
docker commit proj-api-1 forensics/proj-api:incident-4471

# Then investigate at leisure, without touching the original.
docker run --rm -it --network none --entrypoint sh \
forensics/proj-api:incident-4471

Two limits to state plainly. docker commit captures the container’s writable layer, not its memory and not its volumes — a heap dump has to be taken separately, and anything under a volume mount is excluded. And it pauses the container while it works unless you pass --no-pause, which is usually what you want for consistency but is a brief additional service impact you should expect.

For a single artefact, docker cp is faster and does not pause anything:

docker cp proj-api-1:/app/logs/error.log /var/tmp/incident-4471/
docker cp proj-api-1:/tmp/heapdump.hprof /var/tmp/incident-4471/

When it is a security incident

Ordinary evidence capture and forensic evidence capture differ in what happens to the artefacts afterwards.

1. Do not restart, and do not remove. A running compromised container is a source of evidence; a restarted one is not.
2. Isolate rather than stop, so the process state survives — the containment lesson covers docker network disconnect and docker pause.
3. Hash every artefact as you collect it and record the hashes separately: sha256sum over the capture directory, written somewhere the host cannot alter.
4. Copy the capture off the host immediately. A compromised host is not a place to store evidence about its own compromise.
5. Record who captured what, when, in UTC, and with which command. Reconstructed timelines are worth much less than contemporaneous ones.
6. Escalate before cleaning. The decision to rebuild the host belongs to whoever owns the security response, not to the engineer who found it.

Sanity check