Container won't start — diagnosis · Docker & Containers

The container starts and immediately exits. The most common failure mode in Docker. Usually a configuration problem, sometimes a missing dependency, rarely a Docker bug.

The advice everyone gives is “check the logs”, and it is right about a third of the time. The other two thirds of the time there are no logs, because the failure happened before the entrypoint ran and there was never a process to write anything to stdout. An empty docker logs is not the absence of evidence — it is evidence, and it points at a different half of the tree.

Capture before you retry

The single most expensive mistake in this lesson is docker rm followed by docker run again. The exited container carries the exit code, the error string, the OOM flag, the finish timestamp and the log buffer. Removing it deletes all five. Take the snapshot first; it costs three seconds.

Read-only / Safesnapshot the corpse— Run this before you restart, remove or re-create anything.

CONTAINER=web

# Everything the daemon knows about why it stopped, in one line
docker inspect "$CONTAINER" --format \
'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} err={{.State.Error}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}'

# Whatever the process managed to say
docker logs --timestamps --tail 200 "$CONTAINER" 2>&1 | tee /tmp/"$CONTAINER".log

# The daemon's own view of the last hour
docker events --since 1h --until now \
--filter container="$CONTAINER" --format '{{.Time}} {{.Action}} {{.Actor.Attributes}}'

Illustrative output from a healthy diagnosis:

status=exited exit=127 oom=false err= started=2026-08-12T02:14:07Z finished=2026-08-12T02:14:07Z

started and finished one second apart with exit=127 is a complete diagnosis before you have read a single log line. oom=false has already ruled out the memory theory. Compare that with:

status=created exit=0 oom=false err=error while creating mount source path '/srv/appdata': mkdir /srv/appdata: read-only file system

status=created, not exited. The container never ran. ExitCode is 0 and means nothing here, because there was no process to exit. The whole answer is in State.Error, which docker logs will never show you.

The exit code narrows it to two

Docker documents three exit codes as its own, and everything else as the container command’s:

Code	Meaning	Source
`125`	“the error is with Docker daemon itself”	daemon
`126`	“the specified contained command can’t be invoked”	daemon/runtime
`127`	“the contained command can’t be found”	daemon/runtime
anything else	“the exit code of the provided container command”	your process

That last row is the one people misread. 1, 2, 137 and 143 are not Docker codes at all — they are whatever your process exited with, passed straight through.

For signals, the conventional shell encoding applies: a process killed by signal N reports 128 + N. Docker’s own documentation confirms the important one — a container listed with status 137 means “a SIGKILL(9) killed them” — which anchors the rest of the table:

Code	Signal	Common origin
`137`	SIGKILL (9)	kernel OOM killer, `docker kill`, stop timeout expiry, daemon restart
`143`	SIGTERM (15)	`docker stop`, an orchestrator, an operator
`139`	SIGSEGV (11)	the process crashed
`134`	SIGABRT (6)	assertion failure, glibc heap corruption, JVM abort

137 does not mean OOM

This is the most repeated wrong diagnosis in Docker operations. Exit 137 means SIGKILL. The OOM killer sends SIGKILL, so an OOM kill produces 137 — but so does every other SIGKILL, and Docker’s own documentation lists three non-OOM causes: the container’s init process being killed manually, docker kill, and a daemon restart that takes running containers down with it.

There is a fourth that catches people constantly: docker stop sends SIGTERM, waits the timeout, and then sends SIGKILL. A container that ignores SIGTERM therefore exits 137 on a perfectly ordinary, deliberate stop. Nothing ran out of memory. Nothing was wrong at all.

The discriminator is one field:

docker inspect web --format '{{.State.OOMKilled}}'

true is the memory diagnosis. false with exit 137 means something sent a kill, and your job is to find out what — docker events and journalctl -u docker between StartedAt and FinishedAt will name it.

Confirm from the kernel side too, because OOMKilled describes the container’s cgroup and misses the case where the host ran out of memory and the global OOM killer chose your process:

sudo dmesg -T | grep -i -E 'killed process|out of memory'
docker exec web cat /sys/fs/cgroup/memory.events

memory.events has an oom_kill counter. A non-zero value is the cgroup’s own record, and it survives independently of the daemon’s flag.

The tree

flowchart TD
  S[Container is not running] --> Q1{"docker inspect .State.Status"}
  Q1 -- created --> C1["Never started. Read State.Error:<br/>mount source, port conflict, device, runtime"]
  Q1 -- exited --> Q2{"Exit code"}
  Q1 -- restarting --> C2["Crash loop. Restart policy is re-running<br/>a container that dies in under 10s"]
  Q2 -- 125 --> C3["Daemon rejected it.<br/>journalctl -u docker"]
  Q2 -- 126 --> C4["Found but not executable:<br/>chmod, noexec mount, LSM denial"]
  Q2 -- 127 --> C5["Not found: wrong path, wrong shell,<br/>missing interpreter, static/dynamic mismatch"]
  Q2 -- 137 --> Q3{"State.OOMKilled"}
  Q3 -- true --> C6["Memory limit or host OOM"]
  Q3 -- false --> C7["Something sent SIGKILL.<br/>Find the sender"]
  Q2 -- "other" --> C8["Your process exited on purpose.<br/>docker logs is now the right tool"]

Work it top-down. The status field comes before the exit code, because a container in created has an exit code that means nothing.

Cause 1 — the entrypoint does not exist (127)

docker logs web
# exec /app/server: no such file or directory

Four distinct things produce this message, and the fix differs for each:

The path is wrong. COPY put the binary somewhere else, or the WORKDIR moved and the relative path no longer resolves.
The shell is missing. ENTRYPOINT ["/bin/bash", "-c", "..."] in an Alpine image. Alpine ships BusyBox ash at /bin/sh; there is no /bin/bash. The “not found” is bash, not your script.
The interpreter in the shebang is missing. The script exists and is executable; #!/usr/bin/env python3 names an interpreter that is not in the image. The kernel reports ENOENT for the interpreter, and the message names your script, which is maddening.
A dynamically linked binary in a scratch or distroless image. The ELF loader named in the binary’s PT_INTERP header does not exist, so execve fails with ENOENT even though the binary is right there.

Read-only / Safetell them apart— Four commands, one per hypothesis.

IMAGE=myorg/myapp:1.0.0

# Does the path exist at all, and is it executable?
docker run --rm --entrypoint /bin/sh "$IMAGE" -c 'ls -l /app/server'

# Which shells does the image actually have?
docker run --rm --entrypoint /bin/sh "$IMAGE" -c 'ls -l /bin/sh /bin/bash 2>&1'

# What interpreter does the script ask for?
docker run --rm --entrypoint /bin/sh "$IMAGE" -c 'head -1 /app/entrypoint.sh'

# Is the binary dynamically linked, and is its loader present?
docker run --rm --entrypoint /bin/sh "$IMAGE" -c 'file /app/server; ldd /app/server 2>&1'

ldd printing not a dynamic executable is the good answer for a scratch image. ldd printing /lib64/ld-linux-x86-64.so.2 => not found is your diagnosis.

Why the CRLF case looks identical

A script authored on Windows and committed without normalisation has #!/bin/sh\r as its first line. The kernel’s binfmt_script handler reads everything after #! up to the newline and treats it as the interpreter path — carriage return included. It then tries to execute /bin/sh\r, which does not exist.

The error is no such file or directory, exit 127, and every check you run by hand passes: /bin/sh is there, the script is there, the script is executable. cat shows nothing wrong because the terminal swallows the carriage return.

docker run --rm --entrypoint /bin/sh myorg/myapp:1.0.0 \
  -c 'head -1 /app/entrypoint.sh | od -c | head -2'

\r \n at the end of the shebang is the whole answer. Add * text=auto eol=lf to .gitattributes, or dos2unix in the build.

Cause 2 — found but not executable (126)

Exit 126 is Docker telling you the file resolved and the kernel refused to run it. The distinct causes:

No execute bit. COPY preserves the source mode; a script committed 0644 stays 0644 in the image. RUN chmod +x in the Dockerfile, or COPY --chmod=0755.
exec format error. Wrong architecture — an arm64 image on an amd64 host, or the reverse. The daemon will happily pull a manifest for the wrong platform if you ask it to.
A noexec bind mount. The entrypoint lives on a volume mounted noexec on the host. The bit is set, the architecture is right, and the kernel still refuses.
An LSM denial. AppArmor or SELinux blocked the exec.

docker logs web
# exec /app/server: exec format error

Read-only / Safearchitecture and LSM— Confirm platform, then check whether a security module refused.

IMAGE=myorg/myapp:1.0.0

# What platform is this image, and what platform is this host?
docker image inspect "$IMAGE" --format '{{.Architecture}}/{{.Os}} variant={{.Variant}}'
docker info --format '{{.Architecture}} {{.OSType}}'

# AppArmor denials, newest last
sudo journalctl -k --since '10 min ago' | grep -i apparmor

# SELinux denials
sudo ausearch -m AVC -ts recent 2>/dev/null | tail -20

# Which profile is this container running under?
docker inspect "$CONTAINER" --format '{{.AppArmorProfile}} {{.HostConfig.SecurityOpt}}'

Cause 3 — daemon-level rejection (125)

Exit 125 means the daemon refused before any container process existed. docker logs is empty and always will be. The message is on the CLI’s stderr at the moment of the failure, and afterwards it is in State.Error — which is exactly why the snapshot at the top of this lesson matters.

The recurring ones:

`State.Error` contains	Cause
`port is already allocated`	Another process (often a previous container) holds the host port
`driver failed programming external connectivity`	Port publishing failed; usually iptables or a stale rule
`error while creating mount source path`	The bind-mount source does not exist and cannot be created
`no such file or directory` naming a host path	Bind mount source is missing
`invalid reference format`	Malformed image name — usually an unexpanded shell variable
`no space left on device`	The data root filled up

Read-only / Safeport conflict— Find who holds the host port before you change the mapping.

PORT=8080

# Which process is bound?
sudo ss -tlnp "sport = :$PORT"

# Is it another container?
docker ps -a --filter publish="$PORT" --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

# Is it a leftover docker-proxy from a container that went away badly?
pgrep -a docker-proxy | grep "$PORT"

A stranded docker-proxy holding a port after its container is gone is rare but real, and it is the case where restarting the daemon genuinely is the fix rather than a superstition — the daemon owns those processes and re-derives them from container state on start.

Cause 4 — OOM at start

A container can be OOM-killed before it ever serves a request. JVMs and Node applications are the usual victims: the heap sizing happens during startup, so if the limit is below the initial allocation the process dies within the first second.

Read-only / Safeprove the OOM— Four independent signals. Two agreeing is a diagnosis.

CONTAINER=web

docker inspect "$CONTAINER" --format 'oom={{.State.OOMKilled}} limit={{.HostConfig.Memory}}'

# The cgroup's own counters (cgroup v2)
docker exec "$CONTAINER" cat /sys/fs/cgroup/memory.events 2>/dev/null

# The kernel's record
sudo dmesg -T | grep -i -E 'killed process|memory cgroup out of memory'

# The daemon's event stream
docker events --since 1h --filter container="$CONTAINER" --filter event=oom

docker events emits a dedicated oom event for exactly this. If you see an oom event followed by a die event with exitCode=137, the diagnosis is closed.

The container that starts, then dies at the ten-second mark

An unhealthy pattern that looks like a startup failure but is not: the container comes up, runs fine for eight or nine seconds, exits, restarts, and does it again forever. docker ps sometimes shows it running and sometimes shows it restarting, depending on when you look.

The ten seconds is not a coincidence. Docker documents that “a restart policy only takes effect after a container starts successfully”, and that “starting successfully means that the container is up for at least 10 seconds”. A container that dies faster than that is on the daemon’s increasing backoff; one that dies just after resets the counter and restarts immediately, forever, with no backoff to slow it down.

So the ten-second boundary tells you something real. Under ten seconds and you will see the restarts space out. Over ten seconds and you have an infinite tight loop that will happily consume a CPU and fill your disk with logs overnight.

docker inspect --format '{{.RestartCount}}' is the number to watch. If it is in the thousands by morning, this is what happened.

Cause 5 — the healthcheck, which does not cause an exit

Worth stating plainly because it is a common wrong turn: a failing HEALTHCHECK does not stop a container. It moves State.Health.Status from starting to unhealthy and emits a health_status event. Docker Engine takes no further action — no restart policy fires on unhealthy, because the container has not exited.

docker inspect web --format '{{.State.Health.Status}}'
docker inspect web --format '{{range .State.Health.Log}}{{.End}} exit={{.ExitCode}} {{.Output}}{{end}}'

State.Health.Log keeps the last few probe results with their output. If a container is unhealthy and you do not know why, that is the field — not docker logs, which shows the application’s output rather than the probe’s.

The thing that does act on unhealthy is whatever is watching from outside: a load balancer removing the backend, Compose refusing to start a dependent service whose depends_on uses condition: service_healthy, or a Swarm service replacing the task. If your container is being killed and recreated shortly after going unhealthy, look for one of those rather than at Docker Engine.

Stepping through manually

Once the tree has told you which layer failed, reproduce it by hand with the daemon’s start logic removed.

Read-only / Safeinteractive reproduction— Replace the entrypoint with a shell and run the real one yourself.

IMAGE=myorg/myapp:1.0.0

# A shell in the real image, with the real environment
docker run --rm -it --entrypoint /bin/sh "$IMAGE"

# Inside: run the entrypoint and watch it fail with a visible error
# /app/entrypoint.sh; echo "exit=$?"

# Same, but with the real mounts and environment attached, which is
# where most "works in the shell, fails on start" cases resolve
docker run --rm -it --entrypoint /bin/sh \
--env-file /srv/app/.env \
-v /srv/app/data:/data \
"$IMAGE"

The second form matters more than the first. A container that starts fine without its mounts and fails with them has a mount problem, and running the bare image proves nothing.

Image-level diagnostics

docker image inspect myorg/myapp:1.0.0 --format '{{.Config.Entrypoint}}'
docker image inspect myorg/myapp:1.0.0 --format '{{.Config.Cmd}}'
docker image inspect myorg/myapp:1.0.0 --format '{{.Config.Env}}'
docker image inspect myorg/myapp:1.0.0 --format '{{.Config.WorkingDir}}'
docker image inspect myorg/myapp:1.0.0 --format '{{.Config.User}}'

Two of these deserve a second look. Config.User set to a numeric UID that does not own the volume you mounted is the “permission denied on a file that is obviously there” case. Config.WorkingDir pointing at a directory that only exists because a mount creates it is the “works in CI, fails in production” case.

Where the exit code physically comes from

The container’s PID 1 is a child of containerd-shim-runc-v2, not of dockerd and not of runc — runc exec’d the entrypoint and exited long before. When PID 1 terminates, the kernel delivers SIGCHLD to the shim, which calls wait4(2) and receives a status word.

That word encodes either a normal exit (the low byte of the value passed to exit(2)) or a terminating signal number. The shim reports it over its ttrpc connection to containerd, containerd raises a task exit event, dockerd records it in the container’s state, and docker inspect renders it. 137 is the shim’s report of “terminated by signal 9” rendered in the conventional 128 + N encoding.

This chain is why the exit code survives a daemon restart: the shim holds it, and the shim is not the daemon. It is also why exit code 125 is categorically different from the rest — it never came from wait4 at all, because there was never a process. It is dockerd’s own return value, and that is exactly what Docker’s documentation means by “the error is with Docker daemon itself”.

What if `docker logs` is empty and the exit code is 0?

Two very different situations, and State.Status separates them.

status=created. The container never ran. Exit code 0 is the zero value of an integer that was never written. Everything is in State.Error, and the tree above under “daemon-level rejection” applies even though the code is not 125 — a container that failed at creation rather than at start never got a code assigned.

status=exited. Your process ran and exited successfully with no output. This is almost always a one-shot command in a container you expected to be long-lived: CMD ["nginx"] without -g "daemon off;", a service that daemonises itself, or a shell script that reached the end of the file. The container did exactly what you asked; the image is the bug.

The distinguishing test is whether FinishedAt minus StartedAt is plausible for real work. A web server that “ran” for 40 milliseconds forked into the background and left PID 1 with nothing to do.

Verification

A fix is verified when the container survives past the restart-policy threshold and reports the state you expected, not when docker run printed an ID.

Read-only / Safeverify the fix— Fails loudly if the container is not still up and healthy after the threshold.

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=web

sleep 15   # past the 10s successful-start threshold

state=$(docker inspect "$CONTAINER" --format '{{.State.Status}}')
restarts=$(docker inspect "$CONTAINER" --format '{{.RestartCount}}')
health=$(docker inspect "$CONTAINER" --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}')

echo "state=$state restarts=$restarts health=$health"

[ "$state" = 'running' ] || { echo "FAIL: not running" >&2; exit 1; }
[ "$restarts" -eq 0 ]    || { echo "FAIL: restarted $restarts times" >&2; exit 1; }
case "$health" in
healthy|none) ;;
*) echo "FAIL: health=$health" >&2; exit 1 ;;
esac
echo OK

RestartCount is the check people leave out, and it is the one that catches a container that is running right now because it restarted eleven seconds ago.