Triage — the first 15 minutes of an incident · Docker & Containers

The first 15 minutes of an incident determine whether you mitigate or escalate. The wrong move is to dive into root cause analysis before you stabilise.

There is a precondition that “stabilise first” usually leaves out, and in Docker it matters more than anywhere else. The fastest stabilisation available to you is a restart. A restart destroys the exit code, the error string, the OOM flag, the log buffer and the cgroup counters of the thing that failed. So the correct instruction is not “stabilise first” — it is capture, then stabilise, and the capture takes ninety seconds.

The clock

Minute 0-1: acknowledge. Tell the channel you are on it. Start a timestamped log of what you do, in the channel, as you do it. If you cannot take it, page someone who can and say so immediately.
Minute 1-3: capture. Run the snapshot below. Do not skip it because it feels like delay — everything after this point destroys evidence, and you get one chance.
Minute 3-6: assess impact. What is broken, who is affected, is it getting worse, and is there a work-around? This is what determines severity and who else needs waking.
Minute 6-11: stabilise. Mitigate user impact with the smallest action that could work. Roll back the deploy, restart the container, fail over, shed the bad traffic. Do not investigate the root cause yet.
Minute 11-15: communicate and decide. Update the channel with state and next step. If you have not stabilised, escalate now rather than at minute 25.
After 15 minutes: investigate. The system is stable and you have the evidence you captured at minute 1.

The minute-1 capture is the addition that turns this from advice into a procedure. Everything else in the list is standard incident practice; the capture is what makes it work on a Docker host specifically, because Docker’s state is unusually volatile and unusually easy to destroy by accident.

The ninety-second capture

Read-only / Safecapture before you touch anything— Entirely read-only. Run it as the first command of the incident.

#!/usr/bin/env bash
# Read-only incident snapshot. Safe to run at any time.
set -uo pipefail
OUT=/var/tmp/incident-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$OUT"
exec 2>>"$OUT"/capture-errors.txt

# --- Host, 5 seconds ---
date -u                                   > "$OUT"/00-when.txt
uptime; free -m; nproc                    > "$OUT"/01-host.txt
df -h; df -i                              > "$OUT"/02-disk.txt
cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory \
                                        > "$OUT"/03-pressure.txt

# --- Docker, 20 seconds ---
docker ps -a --no-trunc                   > "$OUT"/10-ps.txt
docker stats --no-stream                  > "$OUT"/11-stats.txt
docker system df -v                       > "$OUT"/12-df.txt
docker info                               > "$OUT"/13-info.txt
docker events --since 2h --until now      > "$OUT"/14-events.txt

# --- Per container, the bulk of the time ---
for c in $(docker ps -aq); do
docker inspect "$c"                     > "$OUT"/inspect-"$c".json
docker logs --timestamps --tail 1000 "$c" > "$OUT"/logs-"$c".txt 2>&1
done

# --- Kernel and daemon, 10 seconds ---
dmesg -T | tail -500                      > "$OUT"/20-dmesg.txt
journalctl -u docker.service --since '2 hours ago' --no-pager \
                                        > "$OUT"/21-dockerd.txt
journalctl -u containerd.service --since '2 hours ago' --no-pager \
                                        > "$OUT"/22-containerd.txt

# --- Network, 5 seconds ---
ss -tanp                                  > "$OUT"/30-sockets.txt
iptables-save                             > "$OUT"/31-iptables.txt
docker network ls                         > "$OUT"/32-networks.txt

echo "captured to $OUT"
du -sh "$OUT"

Three details in that script are the difference between it working and not.

--until now on the events capture. Without it, docker events follows the stream forever and the script hangs at the worst moment of the worst day.

--no-stream on docker stats. Same reason.

set -uo pipefail without -e. This script must not abort partway through because one container’s logs are unreadable. A partial capture is worth having; no capture is not.

Put this script on every host, in /usr/local/bin, before you need it. Writing it during an incident is how you end up not running it.

The restart that erased the diagnosis

The sequence is so common it is almost the default. A container is misbehaving. docker restart web. It comes back. The alert clears.

What was lost, permanently, at that moment:

State.ExitCode — the single most informative field, replaced with the new container’s state.
State.Error — the daemon’s own explanation, which is the only record of a start-time failure and is not in any log.
State.OOMKilled — the flag that separates “ran out of memory” from “something sent a kill”.
State.FinishedAt — the timestamp that would let you correlate with dmesg and the deploy log.
RestartCount — reset to zero, erasing the evidence of a crash loop.
The cgroup counters: nr_throttled, memory.events oom_kill, memory.peak. These are per-cgroup and the cgroup is destroyed with the container.
The log buffer, if the container was ephemeral or the log driver does not persist.

It happens again in three weeks. You restart it again, because that is what worked. Now you have two occurrences and zero evidence, and the third one is during a launch and the restart does not help.

The capture costs ninety seconds against an outage clock. Restarting without it saves ninety seconds once and costs the next three incidents.

Exactly which state is volatile, and why

Docker’s state has three tiers with very different lifetimes, and knowing which is which tells you what a given action costs.

Destroyed by docker restart — anything belonging to the old container instance. The exit code and error live in the daemon’s container record and are overwritten by the new run. The cgroup is torn down and recreated, taking cpu.stat, memory.events and memory.peak with it. The network namespace is destroyed and a new veth pair created, so conntrack entries and interface counters go too.

Destroyed by docker rm or docker compose down — everything above, plus the container record itself, plus the json-file log on disk under <data-root>/containers/<id>/. This is why “I restarted it and now I cannot find the logs” is a permanent condition rather than a search problem.

Destroyed by a reboot — the exec root under /var/run/docker, all network namespaces, the conntrack table, the kernel ring buffer, and the running process state of everything. dmesg in particular does not survive, which matters because the OOM killer’s record of what it chose and why lives there.

Survives everything short of deletion — the journal, if journalctl is persistent (Storage=persistent in journald.conf; check with ls /var/log/journal), and the data root.

The practical shortcut: the further down that list your action sits, the more of the capture you should have completed first. A docker inspect is free. A docker restart needs the container-level capture. A reboot needs all of it, including dmesg.

What “stabilise” looks like, with blast radius

For a Docker incident, ordered by how much they disturb:

Action	Blast radius	Reversible
`docker restart <one container>`	One service, seconds of downtime	Yes
Scale out: start another instance	None if the LB is healthy	Yes
Shed load at the proxy	The shed traffic only	Yes
Roll back to the previous image tag	One service, needs the image to be pullable	Yes, forward
`docker compose down && up` with old tag	The whole stack, ordered stop	Yes
`systemctl restart docker`	Every container on the host unless live-restore is on	No
Reboot the host	Everything, plus all volatile evidence	No

Service impact possiblecheck the blast radius before the big hammer— Two commands that tell you what a daemon restart would actually cost.

# Would a daemon restart keep containers running?
docker info --format 'live-restore={{.LiveRestoreEnabled}}'

# How many containers would it take down if not?
docker ps --format '{{.Names}}' | wc -l
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

# Is this host the only one serving? Check before you drain it.
# (substitute your own load balancer query)
curl -sS -m 3 http://192.0.2.5:8404/stats\;csv 2>/dev/null | cut -d, -f1,2,18

The first line is the one that gets skipped. systemctl restart docker on a host with live-restore disabled and eleven containers running is an eleven-service outage, entered into voluntarily, usually to fix one of them.

Roll back, do not roll forward, during the first fifteen minutes

A rollback is a return to a state that demonstrably worked. A fix is a change to a state nobody has run. Under time pressure with incomplete information, the first is much more likely to work and much easier to reason about if it does not.

The rollback that actually works has a precondition, and it is worth verifying on a calm day rather than at 03:00:

# Is the previous image still pullable, right now?
docker pull myorg/myapp:1.4.2

# Is it still on this host, so you do not need the registry at all?
docker image ls myorg/myapp

A rollback that requires a registry pull will fail if the incident is the registry, or if you are rate limited, or if the network path to it is the thing that broke. Keeping the previous release’s image on the host — by simply not pruning it — converts the rollback from a network operation into a local one.

Roll forward once you are out of the fifteen minutes and have evidence.

Docker-specific triage shortcuts

Three questions worth asking early, because each has a one-command answer that eliminates a large branch.

Read-only / Safethe three fast eliminations— Roughly thirty seconds, and each rules out a whole category.

# 1. Did anything actually change on the Docker layer in the last hour?
docker events --since 1h --until now \
--format '{{.Time}} {{.Action}} {{.Actor.Attributes.name}}' | tail -40

# 2. Is anything unhealthy or restarting right now?
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.RunningFor}}' \
| grep -Ei 'restarting|unhealthy|exited' || echo 'all containers stable'

# 3. Is the host itself the problem, rather than any container?
df -h --output=pcent,target | sort -rn | head -5
cat /proc/pressure/io /proc/pressure/memory
uptime

Question 1 is powerful in the negative. An empty event stream for the incident window means no container started, stopped, died, or was OOM-killed, which eliminates the entire “something crashed” family and points at the application or a dependency.

Question 3 catches the case where five services degraded at once, which is almost never five simultaneous application bugs and almost always a shared resource: disk, memory, or a saturated device.

Escalate on a rule, not on a feeling

When to escalate:

You cannot stabilise within 15 minutes.
The impact is growing — more users, more services, or a second host.
The cause is unknown and the symptoms are unfamiliar. Unknown cause with familiar symptoms is normal; unfamiliar symptoms mean your model is wrong.
The mitigation you are considering has a blast radius larger than the incident. Rebooting a host to fix one container is a decision that wants a second person on it.
More than one team’s system is involved.

How to escalate, in a form the next person can use:

Read-only / Safehandoff template— Fill this in and paste it. It is faster than narrating.

cat <<'EOF'
INCIDENT HANDOFF
Started (UTC):     2026-08-12T02:14Z
Symptom:           checkout API returning 502 for ~40% of requests
Impact:            all customers, checkout only; browse unaffected
Trend:             stable since 02:31, not worsening

Evidence captured: /var/tmp/incident-20260812T021600Z on app-01

Tried:
  02:18  captured snapshot
  02:22  restarted checkout container - 502s resumed after ~90s
  02:29  shed 50% traffic at HAProxy - error rate fell, latency normal
Ruled out:
  - not OOM: State.OOMKilled false, memory.events oom_kill 0
  - not disk: df 41% bytes, 22% inodes
  - not a deploy: no image change in docker events for 6h
Current hypothesis:
  connection pool exhaustion against the payments dependency

Next planned step:  raise pool size and observe; NOT yet done
I am:               staying on the call until you take over
EOF

The “ruled out” section is the part that saves the most time and the part people omit. Handing over a list of what is not wrong, with the evidence, stops the next responder repeating your first twenty minutes.

Symptoms vs causes

A symptom is “the application returns 500”. A cause is “the database connection pool is exhausted”.

You stabilise at the symptom level. You fix at the cause level. The two are different skills, and conflating them is what produces both of the classic failures: investigating while users are down, and declaring a restart to be a fix.

What happens if you skip stabilisation?

You dive into root cause analysis. The system gets worse. You spend an hour finding the cause, but the system has been down that whole hour. The business impact is the entire hour.

Stabilise first. Investigate after. Always.

The symmetric failure is worth naming too, because the correction for one tends to produce the other. An operator burned by investigating too long starts restarting immediately, every time, and after six months the team has a service that is restarted weekly, no idea why, and no evidence from any of the occurrences.

The resolution is not a compromise between the two. It is the ordering: capture, stabilise, investigate. The capture is read-only and cheap enough that it never competes with stabilisation, and it is the only step that preserves the option of ever fixing the thing properly.

Verification: did the mitigation work?

A mitigation is verified by the user-facing signal, not by the container being up.

Read-only / Safeconfirm stabilisation— Fails if the symptom is still present, regardless of container state.

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=checkout
URL=http://127.0.0.1:8080/healthz
SAMPLES=20

# Container is up and past the restart threshold
state=$(docker inspect "$CONTAINER" --format '{{.State.Status}}')
restarts=$(docker inspect "$CONTAINER" --format '{{.RestartCount}}')
[ "$state" = 'running' ] || { echo "FAIL: state=$state" >&2; exit 1; }

# The symptom itself is gone: sample the endpoint, count non-200s
fail=0
for _ in $(seq "$SAMPLES"); do
code=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' "$URL" || echo 000)
[ "$code" = '200' ] || fail=$(( fail + 1 ))
sleep 1
done

echo "state=$state restarts=$restarts failures=$fail of $SAMPLES"
[ "$fail" -eq 0 ] || { echo 'FAIL: symptom still present' >&2; exit 1; }
echo 'STABLE'

Sampling over time rather than once is what distinguishes “it worked” from “the first request after a restart happened to succeed”. A container that fails one request in five is not stabilised, and a single curl has an 80% chance of telling you it is.