Troubleshooting framework — the systematic approach

Every production incident is a puzzle. The puzzle has a solution; finding it requires evidence, not instinct. A systematic troubleshooting process beats “I think it’s the database” every time.

The part that is hard to teach is not the tree. It is the discipline of choosing a command that can disprove your current theory, rather than one that will agree with it. Almost every command in Docker will produce output that looks vaguely supportive of whatever you already believe.

The tree

flowchart TD
  S[Service unavailable] --> Q1{Is the container running?}
  Q1 -- no --> C1[Logs, restart policy, exit code]
  Q1 -- yes --> Q2{Is the app responding?}
  Q2 -- no --> C2[App logs, resource limits, deps]
  Q2 -- yes --> Q3{Is traffic reaching it?}
  Q3 -- no --> C3[Network, published ports, DNS]
  Q3 -- yes --> Q4{Is the response correct?}
  Q4 -- no --> C4[App logs, dependency health]
  Q4 -- yes --> Q5{Is it fast enough?}
  Q5 -- no --> C5[Profiles, metrics, traces]
  Q5 -- yes --> R[Working]

Work top-down. Don’t skip steps.

Each branch needs a discriminating command

A tree with no commands attached is a diagram. What makes it usable is that every node names a command whose output partitions the space — one answer sends you left, the other sends you right, and neither answer is “hmm, that looks a bit odd”.

The fourth row is the one that repays memorising. An application bound to 127.0.0.1 inside its own network namespace is unreachable from anywhere, including the host, no matter how correct every layer of Docker networking is. It is the single most common “the port mapping is broken” that is not a port mapping problem, and one ss invocation settles it.

The questions to ask

1. What changed? The most common cause of an outage is a change. What was deployed, configured, scaled, or restarted in the last hour?
2. What is the symptom? Container crashed? Service slow? Network unreachable? Logs full? Each symptom suggests different layers.
3. Where is the evidence? Logs, metrics, traces, config files, recent commits. The evidence lives somewhere; find it.
4. What does "working" look like? Know what a healthy system looks like before you diagnose an unhealthy one. Compare.
5. What is the simplest explanation? Occam's razor applies. The boring answer (full disk, expired cert, recent deploy) is usually right.
6. What is the impact? What is broken? Who is affected? What is the work-around?

“What changed?” has an answer on the host

The question is usually asked of humans and answered badly. Ask the host instead — Docker keeps an event log, and so does the package manager.

# Every container, image, volume and network event in the last 2 hours
docker events --since 2h --until now \
--format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}}'

# Just the transitions that matter
docker events --since 2h --filter event=die --filter event=oom \
--filter event=health_status --format '{{.Time}} {{.Action}} {{.Actor.Attributes.name}}'

# Did the daemon itself restart or reload?
journalctl -u docker.service --since '2 hours ago' --no-pager | head -50

# Was a package upgraded underneath us?
grep -h ' upgrade ' /var/log/dpkg.log 2>/dev/null | tail -20
sudo dnf history list --reverse 2>/dev/null | tail -10

docker events is the most under-used command in Docker operations. It is a full audit trail of daemon-visible state transitions, it is on by default, and --since reaches back into the daemon’s buffered history rather than only streaming from now. A die event carries the exit code in its attributes, so you can reconstruct a crash loop that happened while you were asleep without any external monitoring at all.

Capture before you act

Almost every stabilising action destroys evidence:

The rule that follows is short: capture, then act. Ninety seconds of capture protects you from having to reproduce a failure that may not reproduce.

#!/usr/bin/env bash
set -euo pipefail
OUT=/var/tmp/incident-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$OUT"

docker ps -a                              > "$OUT/ps.txt"       2>&1
docker stats --no-stream                  > "$OUT/stats.txt"    2>&1
docker system df -v                       > "$OUT/df.txt"       2>&1
docker events --since 2h --until now      > "$OUT/events.txt"   2>&1
docker info                               > "$OUT/info.txt"     2>&1
df -h; df -i                              > "$OUT/disk.txt"     2>&1
sudo dmesg -T | tail -300                 > "$OUT/dmesg.txt"    2>&1
journalctl -u docker.service --since '2 hours ago' --no-pager \
                                        > "$OUT/dockerd.txt"  2>&1

for c in $(docker ps -aq); do
docker inspect "$c"                     > "$OUT/inspect-$c.json" 2>&1
docker logs --timestamps --tail 500 "$c" > "$OUT/logs-$c.txt"    2>&1
done

echo "captured to $OUT"

Note --until now on the events capture. Without it the command follows the stream forever and your capture script hangs at the worst possible moment.

Evidence over intuition

# Don't say "the database is slow"
# Do say "SELECT * FROM orders took 4.2s, p99 over 1m is 4.5s"

# Don't say "the network is broken"
# Do say "tcp connect to db:5432 takes 5+ seconds; iptables DNAT rule is present; conntrack entry exists"

# Don't say "memory leak"
# Do say "container's RSS grows from 100 MB to 2 GB over 6 hours; cgroup memory.max reached at 2 GB; OOM-killed"

A claim without evidence is a hypothesis. A claim with evidence is a diagnosis. Operators who present hypotheses as diagnoses waste hours chasing the wrong lead.

The upgrade on that rule: prefer the command that can falsify your hypothesis. If you believe the network is broken, do not run ping and feel reassured when it fails — ping failing is consistent with a dozen causes. Run something that only one cause explains.

The df -i row catches a genuinely confusing failure: a filesystem with plenty of free bytes and no free inodes returns ENOSPC — “no space left on device” — to every write. df -h looks fine and the error message appears to be lying.

Bisect the path, do not walk it

When the tree gives you a long chain — client, proxy, host firewall, DNAT, bridge, veth, container namespace, application — do not test each hop in order. Test the middle one first. Each test halves the remaining space instead of shaving one hop off it.

For a published port, the middle is the host reaching the container’s own IP directly:

CONTAINER=web
NET=bridge

IP=$(docker inspect "$CONTAINER" \
--format "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}")
echo "container ip: $IP"

curl -sS -m 3 -o /dev/null -w 'direct-to-container: %{http_code}\n' "http://$IP:80/"
curl -sS -m 3 -o /dev/null -w 'via-published-port: %{http_code}\n' "http://127.0.0.1:8080/"

Direct works and published fails: the fault is in port publishing — DNAT, the proxy, or the host firewall. Both fail: the fault is at or below the container — the application, its bind address, or the namespace. Two commands, and half the stack is eliminated either way.

Document the investigation

A good incident investigation record includes:

• The alert or symptom.
• The first hypothesis and the evidence for / against.
• The commands run.
• The actual root cause.
• The fix applied.
• The follow-up actions (monitoring, runbook updates).

The second entry is the one people omit, and it is the most valuable. A record of a hypothesis that was wrong, with the command that disproved it, saves the next operator from spending twenty minutes on the same wrong idea. A record listing only the correct answer teaches nothing about how it was found.

Knowledge check