Reading health state — inspect, events, and acting on transitions

Docker computes health and then does nothing with it. The daemon does not restart an unhealthy container, does not remove it from a load balancer, and does not tell anyone. It records a state and emits an event, and everything useful is built on top of one of those two.

This lesson is about getting that signal out.

Polling: the current state

The cheap, obvious approach is to ask.

$ docker ps --filter health=unhealthy --format '{{.Names}}\t{{.Status}}'

payments-worker   Up 2 hours (unhealthy)
redis-cache       Up 41 minutes (unhealthy)

The health filter accepts starting, healthy, unhealthy and none — the last being containers with no healthcheck defined at all, which on most hosts is the largest and most interesting group.

$ docker ps --filter health=none --format '{{.Names}}'

legacy-cron
adminer

For one container, the full picture including recent probe output:

$ docker inspect payments-worker --format '{{json .State.Health}}' | jq '{Status, FailingStreak, Log: [.Log[] | {End, ExitCode, Output}]}'

{
"Status": "unhealthy",
"FailingStreak": 47,
"Log": [
  {
    "End": "2026-08-11T09:12:03.441Z",
    "ExitCode": 1,
    "Output": "curl: (7) Failed to connect to localhost port 8080 after 0 ms: Could not connect to server\n"
  }
]
}

Three things in that output do real diagnostic work. ExitCode separates “the probe ran and reported failure” (1) from “the probe could not run” (127, command not found — the distroless trap from the previous lesson). Output is the probe’s combined stdout and stderr, which is usually the actual error message. And FailingStreak of 47 against retries: 3 tells you this has been failing for a long time and nothing reacted.

The problem with polling

State.Health.Log holds five entries. A container that flapped unhealthy at 02:14 and recovered at 02:16 has, by the time you look at 09:00, no trace of it whatsoever — the five slots have long since been overwritten by successful checks and Status reads healthy.

Polling every 30 seconds is not a fix either. A flap shorter than your poll interval is invisible, and flapping is precisely the symptom you most want to catch, because it is what a saturated resource or an intermittent dependency looks like.

The event stream

The daemon emits an event on every health transition — not on every probe, only when the computed status changes.

$ docker events --filter event=health_status --format '{{.time}} {{.Actor.Attributes.name}} {{.Action}}'

1786518843 payments-worker health_status: unhealthy
1786518991 payments-worker health_status: healthy
1786519104 payments-worker health_status: unhealthy

That output is the flap that polling missed, with timestamps you can line up against a deploy or a load spike.

The event stream is also retrospective. The daemon keeps recent events in memory, so --since replays them:

$ docker events --since 12h --until 1h --filter event=health_status --format '{{.time}} {{.Actor.Attributes.name}} {{.Action}}'

1786490112 redis-cache health_status: unhealthy
1786490233 redis-cache health_status: healthy
1786496741 redis-cache health_status: unhealthy

A watcher that survives its own restart

The naive watcher is docker events | while read ..., which loses everything that happened while it was down. Recording the last timestamp seen and passing it back as --since closes most of that gap:

#!/usr/bin/env bash
# health-watch.sh - record health transitions to a durable log.
set -euo pipefail

STATE=/var/lib/health-watch/last-seen
LOGFILE=/var/log/health-transitions.log
mkdir -p "$(dirname "$STATE")"

# Resume from the last event we recorded; fall back to one hour ago on
# a cold start so a first run has some context.
since=$(cat "$STATE" 2>/dev/null || echo "1h")

docker events --since "$since" \
  --filter event=health_status \
  --format '{{.time}} {{.Actor.Attributes.name}} {{.Action}}' |
while read -r ts name action; do
  printf '%s %s %s\n' "$ts" "$name" "$action" >> "$LOGFILE"
  printf '%s' "$ts" > "$STATE"
done

Run it under systemd with Restart=always and the restart gap becomes “events between the last one written and the moment the process came back”, which is usually nothing.

Reacting automatically, and why to be careful

The tempting next step is a watcher that restarts anything unhealthy. Off-the-shelf tools do exactly this, driven by a container label.

Think about it before you deploy it.

The lesson is not “never automate”. It is that automatic remediation needs the same guards a human would apply:

1. Scope it. Opt containers in by label, not everything by default. Stateless services with fast, dependency-free startup are the safe population.
2. Rate-limit it. A cap of N restarts per container per hour, and a global cap across the host, turns a storm into an alert.
3. Check the dependency first. If every container on the host went unhealthy within the same minute, the common cause is not in the containers. Restarting them is the wrong action.
4. Always alert, even when it worked. A silent self-heal is a failure you never investigated. Three silent self-heals a night is a broken service pretending to be a healthy one.

For most Docker hosts, the right answer is narrower than a general auto-healer: use the health signal to notify, use the restart policy to handle process exit, and reserve automated restart-on-unhealthy for services you have specifically reasoned about.

Exporting health off the host

Health state needs to reach whatever pages you, which means turning it into a metric or a log line. The mechanics of scraping, alert rules and dashboards belong to the monitoring part of this course; what belongs here is the shape of the signal you are exporting.

Two facts govern the design. The state is a three-valued enum, not a number, so a metric must encode it as a label or as separate series. And the transition timestamps are the diagnostic gold — an alert that says “unhealthy now” is far less useful than one that says “flapped four times in the last hour”, and only the event stream can tell you the second.

Knowledge check

Where next

The next lesson puts all of this together on a container that stayed green through a real outage, and works out why.