Healthchecks in Compose — and what they do not do · Docker & Containers

A healthcheck is a command the daemon runs inside the container to decide whether the container is healthy. The result feeds four things:

The docker ps STATUS column, as (healthy) or (unhealthy).
The /containers/<id>/json API response, under State.Health.
depends_on: condition: service_healthy during startup.
docker compose up --wait, which blocks until every service is running or healthy.

Nothing else. In particular a healthcheck does not restart the container, does not remove it from anything, and does not page anyone. It is a signal, and a signal is only as good as what produces it.

Anatomy

services:
  api:
    image: myorg/api:1.4.0
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
      start_interval: 2s

Field	Default	What it controls
`test`	—	The command. `NONE`, `CMD` (exec form) or `CMD-SHELL` (through `/bin/sh -c`)
`interval`	`30s`	Gap between checks, measured from the end of the previous one
`timeout`	`30s`	How long one run may take before it is SIGKILLed and counted as a failure
`retries`	`3`	Consecutive failures needed to flip `healthy` to `unhealthy`
`start_period`	`0s`	Grace window at startup during which failures are not counted
`start_interval`	`5s`	Gap between checks during the start period. Requires Engine 25.0+
`disable`	`false`	Equivalent to `test: ["NONE"]`; switches off a check inherited from the image

The exit status is the whole protocol:

0 — healthy.
1 — unhealthy.
2 — reserved. Do not use it; a check that exits 2 is not “extra unhealthy”, it is undefined.

Anything the check writes to stdout or stderr is captured — the first 4096 bytes — and stored where you can read it back. That is the single most useful debugging property of the whole mechanism and almost nobody uses it.

How the daemon actually runs the check

For every container with a healthcheck, dockerd starts a dedicated goroutine. It does not poll from outside. It issues an exec into the container — the same code path as docker exec — so the check runs in the container’s own namespaces, as the container’s user, against the container’s 127.0.0.1. That is why curl http://127.0.0.1:8080/healthz works from a healthcheck even when the port is not published to the host.

You can watch this happen on the daemon event stream:

docker events --filter event=exec_start --filter event=health_status

Two consequences that bite in production. First, the check costs resources inside the container’s cgroup. A curl process every second in a container with pids_limit: 20 and a tight memory limit is not free; during an incident, when everything is already slow, healthchecks are part of the load. Second, the check binary must exist in the image. A distroless or scratch image has no curl, no wget and no /bin/sh, so CMD-SHELL cannot even start.

The state machine is small. It begins at starting. Any success sets healthy immediately, from any state. Failures increment a counter and only at retries consecutive failures does it become unhealthy — and during start_period, failures do not increment that counter at all, though a success during the start period ends the grace window early and all subsequent failures count.

Reading why a check is failing

docker ps tells you a container is unhealthy. It does not tell you why. State.Health.Log does, and it holds the last five runs with their exit codes and captured output.

Read-only / Safehealth log— The captured output of recent healthcheck runs.

$ docker inspect --format '{{json .State.Health}}' shop-api-1 | jq '.Status, .FailingStreak, (.Log[-1] | {Start, ExitCode, Output})'

"unhealthy"
4
{
"Start": "2026-08-12T04:11:07.318Z",
"ExitCode": 7,
"Output": "curl: (7) Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server\n"
}

Illustrative output

Exit code 7 is curl’s “could not connect”, not the healthcheck protocol’s

The daemon treats any non-zero as unhealthy, and the specific value is the clue: 7 means nothing is listening, 22 means the server answered with an HTTP error, 28 means the request timed out. Three completely different faults, distinguished for free by a field most people never read.

Read-only / Safetriage— Separate a broken healthcheck from a broken service.

SERVICE=api

# 1. What does the daemon think?
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Health}}'

# 2. Run the exact check by hand and read the exit status
docker compose exec "$SERVICE" curl -fsS http://127.0.0.1:8080/healthz
echo "exit: $?"

# 3. If step 2 fails with 'executable file not found', the check is the bug
docker compose exec "$SERVICE" sh -c 'command -v curl || echo NO CURL IN IMAGE'

Step 3 is the one that saves the most time. It distinguishes “the service is down” from “the healthcheck was never able to run in the first place”.

The healthcheck that was never going to work

A team hardens an image: multi-stage build, final stage FROM gcr.io/distroless/static, no shell, no package manager, 12 MB. Good work. The healthcheck block from the old Compose file comes along unchanged:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/healthz"]

There is no curl in that image. The exec fails immediately with exec: "curl": executable file not found in $PATH, which the daemon records as a non-zero exit, so after retries the container is permanently unhealthy. The application is fine. It is serving traffic perfectly.

What makes this expensive is what depends on health. Every service with depends_on: api: condition: service_healthy refuses to start. Every docker compose up --wait in the deploy pipeline times out and reports a failed deploy. If a Traefik or nginx setup routes on health, the healthy container is pulled out of the pool. The outage is entirely manufactured by the monitoring.

The tell is in State.Health.Log: an Output field containing executable file not found, and a Start-to-end duration of a couple of milliseconds. A real failing service takes at least as long as a TCP connection attempt.

The fix is to ship a check the image can run. Modern Go and Rust services often expose a self-check subcommand — test: ["CMD", "/app/server", "healthcheck"] — which is the cleanest answer because it needs no extra binary. Otherwise add a static wget or a tiny purpose-built probe to the final stage on purpose, and know that you did.

The three healthchecks that lie

The one that always passes

test: ["CMD-SHELL", "curl -f http://localhost/ || exit 0"]

The || exit 0 was added during an incident to stop the container being marked unhealthy, and never removed. It reports healthy unconditionally. A healthcheck that cannot fail is strictly worse than no healthcheck, because it converts “we do not know” into “we are fine”.

The one that checks the wrong layer

curl -f http://localhost/ against a reverse proxy or an application server returns 200 from a default page, an error template, or a maintenance placeholder. wget --spider / has the same problem. So does a TCP-connect check: the socket being open says the process bound a port, which happened before it loaded its config.

A /healthz endpoint is only useful if it fails when the service cannot do its job. That usually means it touches the database, or at least reports the result of the last attempt to.

The one that is too shallow, or too deep

pg_isready -U app returns 0 as soon as the postmaster accepts connections. On a Postgres replaying WAL after an unclean shutdown, that happens before it will run a query — it answers FATAL: the database system is starting up. Adding -d app forces a real database name into the check and is strictly better.

Deep checks fail the other way. A /healthz that queries three downstream services on every run, at interval: 5s, across 20 containers, is 12 requests per second of pure overhead — and when the downstream is struggling, the healthchecks pile on and turn a slow dependency into a cascading outage. Check what this service needs to serve a request, not the whole system.

Timings are a statement about the service, not a template

The defaults — 30 second interval, 30 second timeout, 3 retries, no start period — describe nothing in particular. Two numbers matter:

start_period must exceed worst-case cold start. A JVM service with a warm-up phase, a Postgres running initdb, an Elasticsearch forming a cluster: measure the slow path, then double it. Too short and the container is marked unhealthy before it ever had a chance, which with restart: unless-stopped produces a crash loop that looks like an application bug.

interval * retries is your detection time. interval: 30s with retries: 3 means up to 90 seconds of serving errors before anything notices. If that is too slow, shorten the interval — and remember it is also how long a service_healthy dependency will wait before Compose gives up on the whole up.

Set start_interval to something short (2 to 5 seconds) so startup is detected quickly without paying that rate for the container’s whole life.

Verification that can fail

Read-only / Safeprove it— Confirm the check both passes when it should and fails when it should.

SERVICE=api

# It reports healthy now
docker compose ps --format '{{.Service}} {{.Health}}' | grep "^$SERVICE healthy$"

# It notices a real fault: stop the dependency the check exercises,
# then confirm the status flips within interval * retries
docker compose stop db
sleep 40
docker inspect --format '{{.State.Health.Status}}' "$(docker compose ps -q "$SERVICE")"
docker compose start db

If the status stays healthy after its database has been stopped for longer than interval * retries, the check does not exercise the database and you have learned something the happy path would never have told you.

What if the healthcheck itself is broken?

There are two shapes, and they fail in opposite directions.

A check that always succeeds reports healthy forever. A container with a hung application appears healthy, depends_on releases its dependents, --wait returns success, and your load balancer keeps sending traffic to a process that answers nothing. This is the dangerous direction, because every automated system downstream is now confidently wrong. The only defence is testing the failure path deliberately, as in the verification above.

A check that always fails — the missing-curl case, a typo in a path, a timeout shorter than the check takes — marks a working service unhealthy. This is loud, so it gets found, but it is expensive while it lasts: dependents will not start, deploys time out, and if anything in the stack acts on health, a fine service is removed from rotation.

There is also a third case worth naming, because it looks like neither: a check that is flapping. FailingStreak in State.Health resets to zero on every success, so a check that passes one run in four never reaches retries and never reports unhealthy, while the service is failing 75% of real requests. If docker events --filter event=health_status shows repeated transitions, or the health log shows alternating exit codes, the timeout is probably too tight for the check you wrote.