Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

intermediateRuntime~15 min

Break/Fix 4: Container in restart loop — application exits, health check is the alarm

Reported symptoms

  • ●docker ps shows the container in a restart loop with high RestartCount
  • ●docker inspect shows Health.Status = healthy when the container is up, then transitioning to starting when it exits
  • ●docker logs --tail shows the application process exiting on its own with a non-zero code

Evidence

  • · docker ps -a shows the container alternates between "Up" and "Restarting"
  • · docker inspect <id> --format "{{.State.ExitCode}}" shows a consistent non-zero exit code
  • · docker inspect <id> --format "{{.RestartCount}}" climbs each cycle
  • · docker logs <id> --tail 50 shows the application process exiting on its own; the daemon restart loop is the symptom, not the cause
Diagnosis and resolutionclick to reveal

Root cause

The application process itself is exiting on its own, not because of a health check. The daemon's restart policy is firing because the process exited; the health check only reports the state of a running container. The two are separate: a failing health check on a still-running container will never trigger `restart: always` (see docker-health-checks: "Docker does not restart an unhealthy container").

Remediation

Diagnose the application, not the health check. The restart loop is the symptom; the question is why the application exits. Read the exit code with `docker inspect`, run the entrypoint command manually with `docker run --rm -it <image> <entrypoint>` to see the failure mode, and fix the application or its configuration. Increase `restart: on-failure` retries if you want time to investigate before the next restart.

Verification

`docker inspect <id> --format "{{.State.ExitCode}}"` returns 0 once the underlying fault is fixed. `docker ps` shows the container staying Up. The restart count stops climbing.

Prevention

Distinguish "exit" from "unhealthy" in your mental model and in your alerting. `restart: always` is for the case the application process exits; `Health:` is for the case the application process is running but serving broken responses. Treat them as two independent signals.

Reported symptoms

  • Container keeps restarting.
  • docker ps shows high RestartCount.
  • docker inspect shows the container has a non-zero exit code.

Evidence provided

$ docker ps -a
CONTAINER ID   STATUS                          NAMES
abc123def456   Restarting (1) 5 seconds ago    api

$ docker inspect abc123def456 --format '{{.State.ExitCode}}'
137

$ docker logs abc123def456 --tail 30
... application output, then a final line, then exit ...
... the application process exits with 137 (SIGKILL) or another non-zero code ...

The exit code 137 here is illustrative - your scenario’s exit code is whatever the application produced. The procedure below works from any non-zero exit.

Resolution path

  1. Distinguish exit from unhealthy.
  2. docker inspect <id> --format "{{.State.Status}}" reports one of running, restarting, exited, paused, dead. A container in restarting is exiting fast enough that the daemon has not yet reported it as exited. Look at {{.State.ExitCode}}.
  3. Read the application logs from the last cycle.
  4. docker logs <id> --tail 50. The application exit reason is in here, not in the health check.
  5. Run the entrypoint manually.
  6. docker run --rm -it <image> <entrypoint> reproduces the failure without the restart loop in the way. Same image, same entrypoint, no --restart.
  7. Fix the application or its configuration.
  8. Common causes: missing config file (volume not mounted), database unreachable, panic on startup, OOM kill (check dmesg and the host cgroup).
  9. Cap restart attempts if you need time to investigate.
  10. docker run --restart on-failure:5 <image> <entrypoint> retries at most five times. Five failed restarts in a row is the daemon telling you the application is broken; fix the application, do not raise the cap.

Verification

  1. Exit code returns to 0.
  2. docker inspect <id> --format "{{.State.ExitCode}}" returns 0.
  3. Container stays Up.
  4. docker ps shows Up X minutes.
  5. The restart count stops climbing.
  6. docker inspect <id> --format "{{.RestartCount}}" is stable.