Healthcheck timing — the detection-latency budget

The previous lesson covered what a healthcheck can and cannot detect. This one is about when it detects. Four numbers govern that, they interact, and most healthchecks in production carry the values from whichever blog post the author copied.

The four numbers

$ docker run --help | grep -- '--health-'

      --health-cmd string                Command to run to check health
    --health-interval duration         Time between running the check
    --health-retries int               Consecutive failures needed to
                                       report unhealthy
    --health-start-interval duration   Time between running the check
    --health-start-period duration     Start period for the container
                                       health-retries countdown
    --health-timeout duration          Maximum time to allow one check

In a Dockerfile and in Compose the same four appear as --interval/interval, --timeout/timeout, --retries/retries and --start-period/start_period, with --start-interval/start_interval added in Docker 25.0.

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
            --start-period=60s --start-interval=2s \
  CMD curl -fsS http://localhost:8080/health/ready || exit 1

The arithmetic

Docker runs a probe, waits for it to finish (or kills it at timeout), then waits interval before running the next one. A container is marked unhealthy after retries consecutive failures.

The worst case is a failure that begins immediately after a probe succeeded, where each subsequent probe hangs until the timeout:

worst-case detection ≈ retries × (interval + timeout)

With the values above: 3 × (30 + 5) = 105 seconds.

The best case is a failure that begins just before a probe, where each failing probe returns immediately (connection refused rather than a hang):

best-case detection ≈ (retries − 1) × interval

Which is 60 seconds. So this configuration notices between one and two minutes after the fact.

Pick the row from the requirement, not the other way round. “We must take an unhealthy container out of rotation within 30 seconds” points at row two. “This is a batch worker nobody notices for an hour” points at row four and saves you the probe cost.

The start period, and why it exists

A service that takes 45 seconds to load a model, run migrations or warm a cache will fail its first several probes. Without a start period, retries: 3 at a 10-second interval marks it unhealthy about 20 seconds in — and if anything acts on health, it gets killed and restarted, whereupon it fails to start again in exactly the same way. That is the classic crash loop where the container is not broken; the timing is.

--start-period suppresses that. During the start period, probe failures do not count towards retries and the container reports starting rather than unhealthy.

Two details are worth knowing precisely:

• A successful probe during the start period ends the start period immediately. From that moment consecutive failures count normally. So a generous start period costs you nothing when the service is quick — it is a ceiling, not a fixed delay.
• --start-interval sets a separate, usually shorter, probe interval that applies only during the start period. This is what lets you have interval: 30s in steady state and still mark a fast-booting container ready two seconds after it is actually up.

What the state machine reports

docker ps shows the health in the STATUS column, and the raw state is available through the API:

$ docker inspect api --format '{{.State.Health.Status}} failing={{.State.Health.FailingStreak}}'

healthy failing=0

The cost side

A probe is a process. --health-interval=5s on 60 containers means 720 process spawns a minute inside your containers, every minute, forever. That is usually negligible — and occasionally it is not:

• A probe that shells out to a language runtime (python -c ..., node -e ...) costs tens of megabytes of RSS and 100 ms+ of CPU each time. Against a container with --memory 128m, sixty probes an hour that each briefly allocate 40 MB is a real risk of an OOM kill attributed to the application.
• A probe that queries the database on every run multiplies your connection churn by the number of containers. Under load, when the database is already the bottleneck, the healthchecks pile on.
• A hanging probe is held for the full timeout before being killed. With a long timeout and a short interval, probes can overlap in practice and accumulate.

Timing configurations that cannot work

Two more that are worth checking in any config you inherit:

• timeout larger than interval. Docker does not stop you. A 30-second timeout with a 10-second interval means probes are still running when the next is due, and the effective interval becomes the probe duration. Detection latency is then nothing like the number you computed.
• start_period shorter than the real startup time, on a service whose slow start is caused by a dependency. During a dependency outage every container restarts, all of them start slowly because the dependency is still recovering, and the healthcheck kills them before it recovers. See the dependency-failures lesson in this part for the full shape of that cascade.

Knowledge check

Where next

Knowing the container is unhealthy is only useful if something reads that state. The next lesson covers the health event stream and what you can build on it.