Container states, exit codes, and the event stream

“The container is down” is not a diagnosis. A container that never started, a container that started and exited 0, a container the kernel killed, and a container stuck in a restart loop all look identical in a dashboard that only counts running containers — and each one needs a different next command.

The state and the exit code together narrow the problem before you read a single line of application log.

The seven states

The daemon will tell you the list itself if you ask for a state it does not recognise:

$ docker ps --filter status=bogus -q

Error response from daemon: invalid filter 'status=bogus': invalid value for state (bogus): must be one of created, running, paused, restarting, removing, exited, dead

dead is the one people rarely see and always misread. It means the daemon tried to tear the container down and could not — usually because a mount inside it was busy, or the storage driver returned an error. The container holds a partially released writable layer and cannot be started. The fix is to find the stuck mount (mount | grep <container-id>, lsof on the merged directory) and clear it, then docker rm -f.

STATE, STATUS, and the parsing mistake

docker ps shows two different things and only one of them is safe to parse.

$ docker ps -a --format 'table {{.Names}}\t{{.State}}\t{{.Status}}'

NAMES       STATE       STATUS
web         running     Up 4 days (healthy)
worker      running     Up 2 minutes (health: starting)
importer    exited      Exited (0) 3 hours ago
cache       restarting  Restarting (1) 12 seconds ago
staged      created     Created

.State is one of the seven keywords. .Status is free text whose shape depends on health checks, uptime units, and exit codes. Scripts that test docker ps | grep Up return true for Up 2 minutes (unhealthy), which is exactly the case they were written to catch.

Use filters, which are evaluated by the daemon:

docker ps --filter status=running --filter health=unhealthy --format '{{.Names}}'
docker ps -a --filter status=exited --filter exited=137 --format '{{.Names}}'
docker ps -a --filter status=created --format '{{.Names}}'

Reading an exit code

The exit code is stored in .State.ExitCode and is printed by docker ps in the Exited (N) status string. There are three bands.

$ docker inspect --format 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}}' web

status=exited exit=137 oom=true err=

oom=true with exit 137 is the kernel OOM killer, and the fix is a memory limit or a memory leak. oom=false with exit 137 after a docker stop is your stop timeout being too short. Same exit code, opposite remedies.

.State.Error carries the runtime’s message when the container could not be started at all — the 126 and 127 cases usually leave something here.

Watching transitions instead of polling

Two commands replace a sleep 1; docker ps loop.

docker wait blocks until the container stops and prints its exit code. It is the correct way to run a one-shot container from a script:

docker start importer
rc=$(docker wait importer)
if [ "$rc" -ne 0 ]; then
  echo "import failed with status $rc" >&2
  docker logs --tail 50 importer >&2
  exit "$rc"
fi

docker events is a live stream of every lifecycle transition on the host. Filtered, it is the fastest way to see what is actually happening during a deploy or an incident:

$ docker events --filter type=container --format '{{.Time}} {{.Action}} {{.Actor.Attributes.name}}'

1786462052 start web
1786462088 health_status: healthy web
1786462311 die worker
1786462311 start worker
1786462352 die worker
1786462354 start worker

The event stream is also a post-mortem tool. --since and --until accept timestamps or relative durations, so you can replay the window around an incident after the fact:

docker events --since 1h --until 0m --filter type=container --format json

The die action carries the exit code in Actor.Attributes.exitCode, which means you can answer “what exit code did it have three restarts ago” — a question docker inspect cannot answer, because it only ever holds the most recent exit.

Reading a restart loop

A container with a restart policy that keeps failing produces a distinctive pattern. The daemon backs off between attempts — roughly doubling from a tenth of a second up to a cap of one minute — so the events start dense and thin out.

$ docker inspect --format '{{.RestartCount}} {{.HostConfig.RestartPolicy.Name}} {{.HostConfig.RestartPolicy.MaximumRetryCount}}' worker

57 on-failure 0

A MaximumRetryCount of 0 with on-failure means unlimited retries, which is how a container reaches 57 restarts without anybody being paged. Set a bound — --restart on-failure:5 — so that a genuinely broken container ends up exited and visible rather than restarting forever and invisible to a check that only counts non-running containers.

Knowledge check