medium riskservice affecting~25 min

Runbook: Health check flapping between healthy and unhealthy

1 · Prerequisites

Confirm every item is in place before any state change.

Shell access on the Docker host and permission to run docker inspect and docker events
The container name is known, and the container has a health check - docker ps shows a health state in parentheses
You can read the health check command itself, from the Dockerfile, the Compose file, or docker inspect
The application timing is known or measurable - how long a cold start takes, and how long the check endpoint normally takes to answer
Recreating the container is acceptable, because health check parameters cannot be changed on a running container

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· CONTAINER=web
· docker ps --filter name="$CONTAINER" --format '{{.Names}} {{.Status}}'
· docker inspect -f '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' "$CONTAINER"
· docker inspect -f '{{json .Config.Healthcheck}}' "$CONTAINER"
· docker inspect -f '{{range .State.Health.Log}}{{.Start}} exit={{.ExitCode}} out={{.Output}}{{end}}' "$CONTAINER"
· docker events --since 6h --until 0m --filter container="$CONTAINER" --filter event=health_status
· docker inspect -f '{{json .HostConfig.RestartPolicy}} restarts={{.RestartCount}}' "$CONTAINER"

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Run docker inspect -f '{{.State.Health.Status}}' "$CONTAINER"; expect healthy, unhealthy or starting. Anything else means the container has no health check and this runbook does not apply.
2Read the recorded probe results with docker inspect -f '{{json .State.Health.Log}}' "$CONTAINER"; expect up to five entries, each with Start, End, ExitCode and Output. Docker keeps only the most recent five, so capture them now.
3Compare Start and End on the failing entries; expect a duration well under the configured timeout. A duration equal to the timeout means the check was cut off, not that it failed.
4Read the configuration with docker inspect -f '{{json .Config.Healthcheck}}' "$CONTAINER"; expect Interval, Timeout, StartPeriod and Retries in nanoseconds. Divide by 1000000000 to get seconds.
5Check whether the failures cluster at container start with docker events --since 6h --until 0m --filter event=health_status; expect transitions to unhealthy shortly after each start if the start period is too short.
6Read the check command from the same Config.Healthcheck output and decide what it actually tests; expect it to touch only this container. A check that queries a database or another service is a dependency check, not a health check.
7Run the check command by hand ten times with docker exec and time each run; expect consistent durations. A p99 above the configured timeout is the whole explanation.
8Confirm what the restart policy will and will not do with docker inspect -f '{{json .HostConfig.RestartPolicy}}' "$CONTAINER"; expect no relationship to health, because Docker does not restart a container for being unhealthy.
9Decide between the two outcomes and record which - either the check parameters are wrong and need widening, or the check is correct and has found a real intermittent fault that must be escalated.
10If the parameters are wrong, recreate the container with corrected values and watch at least ten consecutive probes before declaring it fixed.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker inspect -f '{{.State.Health.Status}}' "$CONTAINER" prints healthy
✓docker inspect -f '{{.State.Health.FailingStreak}}' "$CONTAINER" prints 0
✓Every entry in .State.Health.Log has ExitCode 0, across all five retained probes
✓docker events --since 1h --until 0m --filter container="$CONTAINER" --filter event=health_status returns no transitions after the change
✓The measured p99 of the check command, timed by hand over ten runs, is less than half the configured timeout
✓The container reaches healthy within the start period after a deliberate restart, rather than passing through unhealthy on the way

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Health check parameters cannot be changed on a running container - docker update has no health options - so every change here means recreating the container. Capture docker inspect output before you do
↶To revert a Compose change, restore the previous healthcheck block in the Compose file and run docker compose up -d for that service; Compose recreates the container with the old settings
↶To revert a docker run change, recreate the container with the previously recorded --health-interval, --health-timeout, --health-retries and --health-start-period values
↶If a health check was disabled with --no-healthcheck as a stopgap, that is a loss of signal rather than a fix. Re-enable it as soon as the real parameters are known, and note in the incident record how long it was off
↶If a dependency was removed from the check command, restoring it will restore the flapping. Fix the dependency monitoring separately before putting it back

6 · Escalation

When the runbook isn't enough, contact:

· The check command is fast and consistent by hand but still fails intermittently under load: escalate to the application team, because the check is right and the application is intermittently unable to serve
· Health transitions correlate with another service becoming slow: escalate to that service owner, and remove the cross-service dependency from this check
· The container is marked unhealthy and nothing acts on it because no orchestrator is present: escalate to the platform team to decide the intended remediation, since Docker alone will not restart it
· Flapping began after a base image change with no application change: escalate to whoever owns the image, attaching the health check command and the log entries

A flapping health check is a measurement problem until proven otherwise. The container oscillates between healthy and unhealthy, alerts fire and clear, and everybody assumes the application is unstable. Roughly as often, the application is fine and the check is asking the wrong question, on too short a clock, about something it does not own.

This runbook separates the two, and it ends with the thing most teams discover too late: Docker does not restart an unhealthy container.

Symptoms

docker ps alternates between (healthy) and (unhealthy).
Alerts fire and clear on their own within minutes.
.State.Health.FailingStreak is non-zero but small.
The application appears to serve traffic normally throughout.

Step 1: Read the health log

Docker records the result of each probe, and this is the whole first half of the diagnosis. It keeps only the five most recent entries, so capture them before you change anything.

Read-only / Safehealth state and log— Reads the recorded probe results. No side effects.

CONTAINER=web

docker inspect -f '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' "$CONTAINER"

# The full log, one entry per probe
docker inspect -f '{{json .State.Health.Log}}' "$CONTAINER" | jq .

# Readable form without jq
docker inspect -f '{{range .State.Health.Log}}{{.Start}} exit={{.ExitCode}} {{.Output}}
{{end}}' "$CONTAINER"

Each entry carries Start, End, ExitCode and Output. Read them in this order:

Field	What it tells you
`ExitCode`	`0` healthy, `1` unhealthy, `2` is reserved and must not be used
`End` minus `Start`	How long the probe took. Compare against `Timeout`
`Output`	The first 4096 bytes of the check’s stdout and stderr
`FailingStreak`	Consecutive failures so far, against `Retries`

Step 2: Do the arithmetic

Read-only / Safehealth check configuration— Reads the configured probe parameters. No state change.

docker inspect -f '{{json .Config.Healthcheck}}' "$CONTAINER" | jq .

# Durations are reported in nanoseconds. 30000000000 is 30 seconds.
docker inspect --format '{{json .Config.Healthcheck}}' "$CONTAINER"

The defaults, from the Dockerfile reference:

Option	Default	Compose key	Meaning
`--interval`	`30s`	`interval`	Time between probes once running
`--timeout`	`30s`	`timeout`	Maximum time to allow one probe
`--start-period`	`0s`	`start_period`	Grace window for a slow boot
`--start-interval`	`5s`	`start_interval`	Probe spacing during the start period
`--retries`	`3`	`retries`	Consecutive failures before `unhealthy`

Two numbers follow from these, and both matter:

Time to notice a real failure is roughly retries times interval. With the defaults that is about 90 seconds — three failures, 30 seconds apart.
Time available for a slow boot is start_period. The documentation is explicit: “Probe failures during the start period don’t count towards the maximum retry count.” During that window probes run every start_interval, and the first success ends the window early.

Set timeout comfortably below interval. A timeout longer than the interval means a slow probe is still running when the next one is due, and the effective probe rate is no longer the one you configured.

Step 3: Time the check by hand

Read-only / Saferun the probe manually— Executes the health check command directly. Runs whatever the check runs, so it is only read-only if the check is.

# Extract the command
docker inspect -f '{{json .Config.Healthcheck.Test}}' "$CONTAINER"

# Run it ten times and time each one
for i in $(seq 1 10); do
/usr/bin/time -f '%e s' docker exec "$CONTAINER" \
  sh -c 'wget -q -O /dev/null http://127.0.0.1:8080/healthz' 2>&1 | tail -1
done

You are looking for the spread, not the average. A check that usually takes 80 milliseconds and occasionally takes 4 seconds will flap against a 2 second timeout while looking perfectly healthy in every manual test you run.

Step 4: Decide what the check is actually testing

A health check that tests a dependency turns one outage into several

A check like curl localhost/healthz where /healthz opens a database connection is not reporting the health of this container. It is reporting the health of the database, through this container.

The consequences compound. The database has a two-second hiccup; every container that checks it goes unhealthy simultaneously. If a load balancer removes unhealthy backends, the entire service disappears for a database blip that would otherwise have caused a few slow requests. If Compose depends_on conditions are in play, restarting anything cascades.

A health check answers one question: can this container serve its own requests right now? Liveness of dependencies belongs in monitoring, where it can page a human, not in a probe that mechanically removes capacity.

The honest test is whether a check would fail while the container is perfectly capable of serving. If it would, it is checking the wrong thing.

Sort the check you have into one of these:

Check	Tests	Verdict
`CMD-SHELL wget -q --spider http://127.0.0.1:8080/healthz`	The process is listening and serving	Correct
A `/healthz` that returns static content	The HTTP stack works	Correct, if slightly shallow
A `/healthz` that queries the database	The database	Wrong — move it to monitoring
A `/healthz` that calls three downstream APIs	Everything else	Wrong, and it will flap constantly
`CMD pg_isready` inside the database container	Itself	Correct
A check that writes to disk or the database	Itself, expensively	Risky — probes run forever, at `interval`

Step 5: The mechanism nobody expects

Docker does not restart an unhealthy container

Health status and restart policy are separate mechanisms that do not talk to each other. A restart policy — no, on-failure, always, unless-stopped — reacts to the container exiting. An unhealthy container has not exited. It is running, it is labelled unhealthy, and it will stay that way indefinitely.

The documentation for restart policies describes exit codes and manual stops. Health status is not among the triggers, and this is not an oversight — Docker Engine on its own has no remediation step to take.

Teams discover this during an incident, having assumed for months that restart: always plus a health check equalled self-healing. It does not. It equals a container that will restart if it crashes, and a label that says it is broken which nothing reads.

If you need remediation, it has to come from somewhere else: an orchestrator that acts on health, a supervisor that watches docker events --filter event=health_status and takes action, or a human responding to an alert on that same signal. Choose one deliberately and write it down.

Read-only / Safehealth transitions over time— Reads the daemon event stream for a bounded window. No state change.

# Every health transition in the last six hours
docker events --since 6h --until 0m \
--filter container="$CONTAINER" --filter event=health_status

# What the restart policy will actually do
docker inspect -f '{{json .HostConfig.RestartPolicy}} restarts={{.RestartCount}}' "$CONTAINER"

Events are emitted on transition, not on every probe, so a container that has been stably healthy for days produces no health_status events at all. An empty result over a long window is therefore good news, and a dense one is your flap history with exact timestamps to correlate against deploys and traffic.

Step 6: Change the parameters

Health check settings are part of the container’s configuration. There is no docker update option for any of them, so applying a change means recreating the container.

Service impact possiblerecreate with corrected timing— Recreates the container. The service is briefly unavailable.

# Compose: edit the healthcheck block, then recreate just this service
#   healthcheck:
#     test: ["CMD-SHELL", "wget -q --spider http://127.0.0.1:8080/healthz || exit 1"]
#     interval: 15s
#     timeout: 5s
#     retries: 3
#     start_period: 60s
#     start_interval: 5s
docker compose up -d --no-deps web

# docker run: the equivalent flags
docker run -d --name web \
--health-cmd 'wget -q --spider http://127.0.0.1:8080/healthz || exit 1' \
--health-interval 15s \
--health-timeout 5s \
--health-retries 3 \
--health-start-period 60s \
myorg/myapp:1.4.2

docker inspect -f '{{json .Config.Healthcheck}}' web

Note the || exit 1 in the check command. The health check interprets exit status directly — 0 healthy, 1 unhealthy, and 2 reserved — so a command whose failure exit code is something else should be normalised explicitly.

Common patterns

Evidence	Diagnosis	Action
Unhealthy only in the first minute after start	`start_period` shorter than cold boot	Raise `start_period` above the measured cold start
Probe duration equals `timeout` exactly	Probe cut off, not failed	Raise `timeout`, or make the check cheaper
Fails when another service is slow	Check tests a dependency	Remove the dependency from the check
`Output` empty, exit code non-zero	Check binary missing from the image	Use a tool the image actually contains
Flaps under load only	Real intermittent fault, or CPU throttling	Check `nr_throttled` before touching the check
Marked unhealthy and nothing happens	Restart policy does not act on health	Expected. Add an orchestrator or an alert
`FailingStreak` resets before reaching `retries`	Genuine flapping, below the alarm threshold	Lower `retries` to catch it, or accept it and monitor
Whole stack will not start after a restart	`depends_on` waiting on a check that never passes	Fix the upstream check’s `start_period` first