Health checks — what they can and cannot do · Docker & Containers

A Docker healthcheck is a command the daemon runs inside your container on a timer. If it exits 0 the container is healthy; if it exits non-zero enough times in a row, the container is unhealthy.

That is the entire mechanism. Everything people expect to follow from it — a restart, a load balancer update, a page — is something else’s job, and on a standalone Docker host most of those jobs are unfilled. This lesson is about the gap between what the green word in docker ps means and what people read it to mean.

The state machine

A container with a healthcheck is always in exactly one of four states, and the transitions have rules that are worth knowing precisely because two of them are counter-intuitive.

Status	Meaning
`none`	no healthcheck is defined
`starting`	the container has started and no check has yet succeeded
`healthy`	the last check exited `0`
`unhealthy`	`retries` consecutive checks have failed, after the start period

The probe’s exit code is interpreted strictly:

Exit code	Result
`0`	healthy
`1`	unhealthy
`2`	reserved — treated as unhealthy
anything else	error running the probe — treated as a failure

That last row is the one that bites. A probe that cannot be executed at all — the binary is not in the image — exits 127, and the container sits unhealthy forever with an application that is working perfectly.

The five options and their real defaults

Configuration changeDockerfile— Every value here is explicit. The defaults are listed below and two of them are surprising.

HEALTHCHECK --interval=10s \
          --timeout=3s \
          --start-period=60s \
          --start-interval=2s \
          --retries=3 \
CMD curl -fsS --max-time 2 http://localhost:8080/healthz || exit 1

Option	Default	What it bounds
`--interval`	`30s`	time between checks once the container is past the start period
`--timeout`	`30s`	how long one check may run before it is killed and counted as a failure
`--retries`	`3`	consecutive failures needed to go `unhealthy`
`--start-period`	`0s`	grace window after start during which failures do not count
`--start-interval`	`5s`	time between checks during the start period

The --timeout default of 30s matching the --interval default of 30s is worth pausing on: with both at their defaults, a hanging check occupies the entire interval, so checks run back to back with no gap. Set --timeout to a fraction of --interval, not a value near it.

`--start-period` does not delay the first check

The near-universal reading of --start-period=60s is “wait 60 seconds before checking”. That is not what it does, and the difference matters when you are trying to work out why a container went unhealthy.

Checks run from the beginning. What the start period changes is the accounting: while the container is still starting and less than start-period has elapsed, a failed check does not increment the failure streak, so it cannot produce unhealthy. A successful check during the start period immediately transitions the container to healthy and ends the start period early.

That design is deliberate and better than a delay would be: a fast application is marked healthy in a second or two rather than waiting out a grace period sized for the worst case, while a slow one is protected. It also means checks during startup use --start-interval (default 5s) rather than --interval, so the polling is tighter exactly when you want to know as soon as possible.

The practical consequence: you can set --start-period generously — 60s, 120s for a JVM — with no cost to a fast start. The only cost of a large start period is that a container which never becomes healthy takes that much longer to be reported as unhealthy.

No start period is why a slow application flaps

A container with no --start-period and --retries=3 --interval=10s must answer its health endpoint within roughly 30 seconds of process start or it is marked unhealthy. An application that migrates a database schema, warms a cache or loads a large model on boot will not, and it goes unhealthy while it is doing exactly what it should.

By itself that is only a wrong label. Combined with anything that acts on health, it is an outage that repeats: an orchestrator or an external supervisor restarts the container, the restart begins the slow boot again, and the container never gets far enough to become healthy. This is the classic startup crash loop, and the entire fix is one flag.

Size --start-period from a measured cold start on the slowest host you run on, then double it. It costs nothing when the application is fast.

Detection-time arithmetic

Two numbers matter and people usually only think about one.

Time to detect a failure ≈ retries × (interval + worst-case check duration), where the worst case is timeout.

With the defaults — interval 30s, timeout 30s, retries 3 — that is up to 3 minutes of a broken container reporting healthy.

Configuration	Worst-case detection
defaults (`30s` / `30s` / `3`)	~180s
`--interval=10s --timeout=3s --retries=3`	~39s
`--interval=5s --timeout=2s --retries=2`	~14s
`--interval=2s --timeout=1s --retries=2`	~6s

Tightening this is not free. The probe runs inside the container, against the same CPU and memory limits as the application, and a check that costs 40 ms every 2 seconds is 2% of a core doing nothing but answering itself. More importantly, a short retries makes the check sensitive to a single slow response — one GC pause longer than timeout and a healthy container is marked unhealthy.

The shape that works: retries of at least 2 so a single blip cannot flip the state, timeout well under interval, and an interval sized against how quickly anything downstream can actually react. Detecting a failure in 6 seconds is pointless if the only consumer is a human reading a dashboard every morning.

The false green

A healthcheck’s value is entirely determined by how much of the request path it exercises. Most of them exercise none of it.

Check	Proves	Misses
`CMD exit 0`	nothing	everything
`nc -z localhost 8080`	a socket is listening	the process that accepted it may be deadlocked
`curl -f http://localhost:8080/`	the HTTP server answers	routing, the database, the queue, the disk
`curl -f http://localhost:8080/healthz` returning a constant `200`	the framework is routing	every dependency
`/healthz` that opens a DB connection and runs `SELECT 1`	the app can reach its database	write path, disk full, downstream APIs

The rule: a healthcheck must exercise the dependency path it claims to cover, or it is asserting rather than checking.

An endpoint that returns 200 OK from a handler with no body is not measuring your application; it is measuring your web framework. When the database connection pool is exhausted and every real request is failing, that endpoint keeps returning 200, docker ps keeps saying healthy, and the dashboard is green over a total outage.

`curl` without `-f` reports every HTTP error as success

HEALTHCHECK CMD curl -s http://localhost:8080/healthz

This check passes when the endpoint returns 500. It passes when it returns 503. It passes when it returns 404 because you renamed the route. curl exits 0 whenever it successfully completed an HTTP transaction, and a 500 response is a completed transaction.

-f (--fail) is what makes curl exit non-zero on a status of 400 or above. -s silences the progress meter, -S restores error messages, and --max-time bounds the whole operation so the probe cannot outlive Docker’s own --timeout and get killed mid-flight.

The form to use, and the reason for every flag:

curl -fsS --max-time 2 http://localhost:8080/healthz || exit 1

The trailing || exit 1 normalises curl’s various failure exit codes (7 for connection refused, 22 for an HTTP error, 28 for timeout) to the 1 that Docker interprets as unhealthy. Without it the check still fails — any non-zero is a failure — but the health log records a number instead of a meaning.

The container is permanently unhealthy and the application is fine

docker ps shows (unhealthy). The application serves requests correctly from outside. Restarting does not help. The health log shows an empty output and an exit code of 127.

127 is “command not found”. The image does not contain curl. Slim, Alpine, distroless and scratch-based images frequently do not, and the HEALTHCHECK line was copied from a Dockerfile for an image that did. The check has never once run.

Fix it with something the image definitely has:

A Go/Rust binary in a scratch image — give the application a healthcheck subcommand and use CMD ["/app", "healthcheck"]. This is the most robust option and costs a few lines of application code.
Alpine — wget is in busybox: CMD wget -q --spider --timeout=2 http://localhost:8080/healthz || exit 1.
A language runtime already present — CMD python3 -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/healthz', timeout=2).status==200 else 1)".

And verify it, rather than trusting it, with the exec form below.

Read-only / Saferun the probe by hand before you trust it— The exit code is the whole answer. Do this once per image, not once per incident.

CONTAINER=web

# What is the daemon actually configured to run?
docker inspect -f '{{json .Config.Healthcheck}}' "$CONTAINER"

# Run the same command in the same container and read the exit code.
docker exec "$CONTAINER" curl -fsS --max-time 2 http://localhost:8080/healthz
echo "exit=$?"

# Prove it fails when it should: point it at a route that does not exist.
docker exec "$CONTAINER" curl -fsS --max-time 2 http://localhost:8080/definitely-not-a-route
echo "exit=$?"

The second docker exec is the part people skip and the part that matters. A check that has never been observed failing is a check you have not tested — you have confirmed it can return zero, which was never in doubt.

The misconception: Docker does not restart an unhealthy container

This is the single most important sentence in the lesson.

A container marked unhealthy keeps running. The daemon writes the status, emits an event, and does nothing else. It does not stop it, does not restart it, and does not stop it receiving traffic on a published port.

Restart policies and health status are separate mechanisms that do not interact. restart: always restarts a container that exits. An unhealthy container has not exited — that is the definition of being a running container with a failing probe — so the restart policy never fires. Setting restart: unless-stopped in the hope of getting “restart when unhealthy” produces exactly nothing.

What does consume health status:

Consumer	Effect	Available in standalone Docker
Compose `depends_on: condition: service_healthy`	delays dependent startup only	yes
Docker Swarm	replaces the task	only in Swarm mode
Traefik `loadbalancer.healthcheck`	stops routing — its own check, not Docker’s	yes, but it is Traefik’s probe
`docker events` / `docker ps --filter health=unhealthy`	tells you	yes, if something is watching
Kubernetes liveness probe	restarts the container	not Docker

Note the Traefik row carefully: a proxy that “respects health” is running its own probe against the application, not reading Docker’s State.Health.Status. Two independent health checks with different intervals and different endpoints is the normal state of affairs, and they can disagree.

Configuration changethe supervisor Docker does not ship— Watch the event stream and act. This is the smallest honest version of 'restart when unhealthy'.

#!/usr/bin/env bash
set -euo pipefail

# Only ever restart containers that opt in with a label, so this can
# never act on a database that happens to fail one probe.
docker events --filter 'event=health_status' --format '{{.Actor.Attributes.name}} {{.Status}}' | while read -r name status; do
  [ "$status" = 'health_status: unhealthy' ] || continue
  optin=$(docker inspect -f '{{index .Config.Labels "autoheal"}}' "$name" 2>/dev/null || true)
  [ "$optin" = 'true' ] || continue
  logger -t autoheal "restarting unhealthy container $name"
  docker restart "$name"
done

Automatic restart on unhealthy is a decision, not an obvious improvement

Before you run the loop above, be clear about what it does to each kind of workload:

A stateless web container — restarting is usually right and usually harmless.
A database — restarting is usually wrong. An unhealthy PostgreSQL is often mid-recovery or under load, and a restart interrupts recovery and starts it again from the beginning.
Anything with a slow start — restarting begins the crash loop described earlier. Fix --start-period first.
A dependency failure — if the check covers the database and the database is down, every replica is unhealthy and every replica restarts, simultaneously, against a database that is already struggling. That is a self-inflicted thundering herd.

The label opt-in is not decoration. It is the difference between a supervisor and a hazard.

Where the health state lives and who runs the probe

The probe is executed by dockerd, not by anything inside the container and not by the container’s init process. The daemon creates an exec — the same mechanism as docker exec — attached to the container’s namespaces and cgroup, runs the command, and reads the exit code. That has four consequences you can observe:

The probe consumes the container’s resources. It runs inside the container’s cgroup, so its CPU counts against --cpus and its memory against --memory. A heavyweight probe on a tightly limited container competes with the application, and in a container near its memory limit the probe can be the allocation that triggers the OOM kill.
The daemon owns the timer. Restart dockerd and the timers are re-established; containers that were healthy are re-evaluated from starting. This is normal and briefly shows health: starting across the host after a daemon restart.
On --timeout the probe is killed outright. The API description is explicit: the check is considered failed and the process is “forcibly terminated without a graceful shutdown”. A probe that starts a transaction and is killed at timeout leaves that transaction to be cleaned up by the database, every interval, forever. This is why a health endpoint must have no side effects.
The health log is bounded. The daemon keeps the last five results, and truncates each probe’s output to about 4 KB. A check that prints a stack trace gives you the first 4 KB of it and nothing from six failures ago.

Reading the health log

The status word is the least useful part of the output. The log is where the answer is.

Read-only / Safewhy it is unhealthy— FailingStreak, plus the last five probe results with exit code and output.

$ docker inspect --format '{{json .State.Health}}' web | python3 -m json.tool

{
  "Status": "unhealthy",
  "FailingStreak": 7,
  "Log": [
      {
          "Start": "2026-08-12T02:14:31.118Z",
          "End": "2026-08-12T02:14:33.121Z",
          "ExitCode": -1,
          "Output": "Health check exceeded timeout (2s)"
      },
      {
          "Start": "2026-08-12T02:14:41.130Z",
          "End": "2026-08-12T02:14:41.288Z",
          "ExitCode": 22,
          "Output": "curl: (22) The requested URL returned error: 503"
      }
  ]
}

Illustrative output

Read the exit codes as a diagnosis:

Exit code in the log	Means
`-1` with “exceeded timeout”	the probe was killed at `--timeout`; the application is hanging, not erroring
`7` (curl)	connection refused — nothing is listening on that port inside the container
`22` (curl)	the server answered with 4xx or 5xx — the application is up and rejecting
`28` (curl)	curl’s own `--max-time` fired before Docker’s timeout
`127`	the probe binary is not in the image

FailingStreak is the counter that --retries is compared against. Watching it climb and reset is how you distinguish a genuinely broken service from one that fails one probe in ten — the second is a --retries or --timeout problem, not an application problem.

Read-only / Safefleet-wide health— Three views: what is unhealthy now, what has no check at all, and what is flapping.

# Unhealthy right now
docker ps --filter health=unhealthy --format 'table {{.Names}}\t{{.Status}}'

# Containers with NO healthcheck - usually a bigger finding than an
# unhealthy one, because nothing is watching them at all.
docker ps --filter health=none --format 'table {{.Names}}\t{{.Image}}'

# Watch transitions live. Flapping shows as repeated pairs.
docker events --filter 'event=health_status' --format '{{.Time}} {{.Actor.Attributes.name}} {{.Status}}'

The health=none query is the one to run on a host you have just inherited. An unhealthy container is at least being measured; a container with no healthcheck is invisible to every mechanism in this part of the course, including depends_on: service_healthy, which silently degrades to “started” when the dependency has no check.

Two levels, and which one Docker gets

Most well-built applications expose two endpoints:

/health/live — is the process responsive? Cheap, no dependencies. Answers “should this be restarted?”
/health/ready — can it serve a real request right now? Checks the dependencies it needs. Answers “should this get traffic?”

Kubernetes consumes both, as liveness and readiness probes, and does different things with them. Docker has exactly one healthcheck and no built-in consumer, which forces a choice.

Point Docker’s healthcheck at readiness. The reasoning: the only things that consume it on a standalone host are depends_on: service_healthy and whatever you write, and both of those are asking “is it usable yet”, which is readiness. Liveness has no consumer because Docker will not restart on unhealthy anyway.

Readiness that checks a shared dependency makes failures correlated

Point /health/ready at the database and every replica of the service goes unhealthy at the same instant the database does — which is truthful and occasionally catastrophic. If anything acts on that status, one database blip becomes a full-fleet event.

The compromise that holds up: check the dependency, but check it cheaply and with a short cache. Reuse a pooled connection rather than opening one, run SELECT 1 rather than a real query, cache the result for a few seconds so a hundred probes do not become a hundred queries, and set --retries to at least 2 so a one-second blip does not propagate.

And keep the write path out of it entirely. A health endpoint that inserts a row runs on every interval on every replica forever, and it will eventually be the largest single writer in your database.

Verification that can fail

Read-only / Safeprove the check detects the failure it claims to— Break the dependency deliberately and confirm the status changes. Nothing else is verification.

set -euo pipefail
APP=web
DEP=postgres

# Baseline: it should be healthy.
docker inspect -f '{{.State.Health.Status}}' "$APP"

# Break the dependency the check claims to cover.
docker stop "$DEP"

# Wait out (retries x interval) plus a margin, then look again.
sleep 45
docker inspect -f '{{.State.Health.Status}}' "$APP"

# Put it back and confirm recovery. One success is enough to go healthy.
docker start "$DEP"
sleep 20
docker inspect -f '{{.State.Health.Status}}' "$APP"

If the middle reading is still healthy, your healthcheck does not check what you think it checks — and you have found that out on a Tuesday afternoon rather than during an incident. That sequence is the only meaningful test of a healthcheck, and it takes ninety seconds.

What if the healthcheck itself is what is failing?

Two directions, with opposite symptoms and opposite risks.

Always-green. The probe cannot fail: CMD exit 0, a curl without -f, an endpoint that returns a static 200. Nothing alerts, depends_on: service_healthy waits for nothing, and the first signal that anything is wrong is a user. This is the more dangerous failure because it produces no evidence — you cannot find it by looking at health status, only by testing the probe against a deliberately broken dependency.

Always-red. The probe cannot succeed: missing binary (exit 127), wrong port, wrong path, --timeout shorter than the endpoint’s normal response time. This is loud, and it is loud in a way that causes damage: depends_on: service_healthy blocks the entire stack from starting, and any supervisor acting on health restarts a working container in a loop.

The diagnostic that separates them from a real application failure is the same in both cases: run the probe by hand with docker exec and read the exit code, then run it against something you know is broken and check that it fails. A probe you have only ever seen return 0 is an untested probe.

To undo a bad healthcheck without rebuilding the image, override it at run time — --no-healthcheck disables it entirely, or --health-cmd replaces it — which is enough to get a blocked stack started while you fix the Dockerfile.

Knowledge check

Where next

Restart policies are the mechanism people reach for when they discover Docker will not restart an unhealthy container. The next lesson covers what they actually do, including the backoff, and why they are not that mechanism.