What restart policies and healthchecks cannot recover

You have set restart: unless-stopped and written a HEALTHCHECK. The reasonable expectation is that the container now recovers from things.

It recovers from some things. This lesson is the map: which failure modes each layer catches, which it detects without acting on, and which nothing available on a single host will address. The last column is the one that decides whether you need a second host.

The taxonomy

Two rows deserve the rest of the lesson: the wedged process, and the daemon.

The wedged process

A process that is running but not working is invisible to the restart policy. The policy is a reaction to an exit; nothing exited.

This is the failure mode that matters most in production, because it is what most application bugs look like. A connection pool exhausted by a slow dependency, a deadlock between two locks, a garbage collector thrashing, a thread pool full of requests waiting on a socket that will never answer — in every case the process is alive, docker ps says Up 4 days, and the service is down.

The healthcheck detects it. And then, on standalone Docker, nothing happens.

$ docker ps --format '{{.Names}} — {{.Status}}'

proj-api-1 — Up 4 days (unhealthy)
proj-web-1 — Up 4 days (healthy)

Up 4 days (unhealthy) is a stable state. The daemon will report it for as long as the container runs. It does not restart the container, it does not stop routing to it, and it does not tell anyone.

The usual answer is a watcher: a small container subscribed to docker events that restarts anything entering unhealthy.

docker events \
--filter 'event=health_status' \
--format '{{.Time}} {{.Actor.Attributes.name}} {{.Status}}'

It works, and it has a real cost: the watcher needs the Docker socket, and the Docker socket is root on the host. You are trading a class of application outage for a large increase in blast radius if that container is ever compromised. If you take that trade, put the socket behind a proxy that permits only the calls the watcher needs, and read the daemon-socket lesson in the certificates part first.

What “the restart policy takes effect” actually means

Two documented behaviours surprise people, and both explain incidents that otherwise look like Docker ignoring configuration.

The ten-second rule. The documentation states that a restart policy only takes effect after a container starts successfully, and that starting successfully “means that the container is up for at least 10 seconds and Docker has started monitoring it.” A container that dies in two seconds every time is not in the regime the policy was designed for.

Manual stops are sticky. When you manually stop a container, the restart policy “is ignored until the Docker daemon restarts or the container is manually restarted”. This is the intended behaviour and it is also the trap: a container you stopped during an incident at 02:00 comes back on its own when the host reboots for patching three weeks later, in a state nobody expects.

The retry behaviour differs by policy. on-failure accepts a maximum retry count — on-failure:5 gives up after five. always and unless-stopped take no such limit, and the daemon keeps retrying with a growing delay between attempts. A container in that state is not “stuck”; it is doing exactly what you asked, forever.

docker inspect proj-api-1 --format \
'restarts={{.RestartCount}} status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} started={{.State.StartedAt}}'

A RestartCount in the hundreds against a StartedAt from four minutes ago is a crash loop wearing the costume of a healthy container. docker ps will show it as Up 3 seconds if you happen to look during an attempt.

The daemon is a single point of failure

Restarting the Docker daemon — for an upgrade, or because it wedged — stops every container on the host, unless live restore is enabled:

{
"live-restore": true
}

With it, containers keep running across a daemon restart. This turns a Docker upgrade from a full-stack outage into a control-plane blip, and it is the single highest-value line in daemon.json for a single-host deployment.

Its limits are worth stating plainly: it covers daemon restarts, not host reboots; while the daemon is down you have no API, so nothing can be started, stopped or inspected; and container logs are not being collected by the daemon during that window.

Sanity check