Failure lab — the container that stayed healthy through an outage

Every failure lab in this course has a symptom you can see. This one has a symptom you cannot: the incident is total, and the monitoring built on Docker health says nothing is wrong. It is the most common healthcheck failure in production and the hardest to argue with, because the evidence appears to be on the healthcheck’s side.

The scenario

A three-container stack: an nginx reverse proxy, an API, and PostgreSQL. The API’s Dockerfile carries what looks like a reasonable healthcheck.

# compose.yaml
services:
  proxy:
    image: nginx:1.29-alpine
    ports: ['443:443']
    depends_on:
      api:
        condition: service_healthy

  api:
    image: example.com/api:2.3.0
    healthcheck:
      test: ['CMD', 'curl', '-fsS', 'http://localhost:8080/health']
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    environment:
      DATABASE_URL: postgres://api@db:5432/app

  db:
    image: postgres:18-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]

And the API’s /health handler, which is where the whole incident comes from:

@app.get("/health")
def health():
    return {"status": "ok"}

At 09:12 the database’s connection pool is exhausted by a runaway report query. Every API request that touches the database now blocks for 30 seconds and returns a 500.

Symptoms

• Users report the site is down. The proxy returns 502 and 504.
• docker ps shows all three containers Up and (healthy).
• No container has restarted. RestartCount is 0 across the board.
• The Docker event stream contains no health_status events at all.
• Any alerting built on Docker health is silent.

The on-call engineer’s first ten minutes go into disbelief, because the platform is insisting nothing is wrong.

Diagnosis

The discriminating step is to stop trusting the status and run the probe yourself, then run a real request, and compare.

1. Read what the probe actually is, rather than what you remember configuring.
2. Execute the probe by hand inside the container.
3. Execute a request that exercises the real code path, from the same place.
4. If the first succeeds and the second fails, the probe is the bug.

Step one, from the host:

$ docker inspect api --format '{{json .Config.Healthcheck}}' | jq

{
"Test": ["CMD", "curl", "-fsS", "http://localhost:8080/health"],
"Interval": 30000000000,
"Timeout": 5000000000,
"StartPeriod": 20000000000,
"Retries": 3
}

Step two and three, inside it:

$ docker exec api curl -sS -o /dev/null -w 'health=%{http_code} in %{time_total}s\n' http://localhost:8080/health

health=200 in 0.002s

$ docker exec api curl -sS -o /dev/null -w 'orders=%{http_code} in %{time_total}s\n' http://localhost:8080/api/orders

orders=500 in 30.114s

There it is. The probe returns in two milliseconds because it touches nothing. /health is a function that returns a literal — it proves the HTTP server is accepting connections and the process is scheduled, and nothing else. Every dependency the service actually needs is outside its scope.

Confirming the real fault takes one more look, at the dependency the probe never checked:

$ docker exec db psql -U postgres -At -c 'SELECT state, count(*) FROM pg_stat_activity GROUP BY state;'

active|97
idle in transaction|3

Against a default max_connections of 100, that is a pool with nothing left to give.

Two variants of the same bug

Before fixing it, note that this incident has siblings that produce identical symptoms and are worth recognising:

The probe targets localhost and the outage is in publishing. curl http://localhost:8080/health from inside the container is green whether or not the port publish, the proxy upstream, or the Docker network path works. The container is genuinely healthy; users still cannot reach it. Health inside a namespace says nothing about reachability from outside it.

The probe checks a dependency the request path does not use. The mirror image: a healthcheck that pings Redis while the failing code path is PostgreSQL. It looks thorough and covers the wrong thing.

Both come from the same mistake — the probe was written from the architecture diagram rather than from the request path.

Recovery

Recovery has two phases, and they are in this order because the first one buys time and the second one takes a deploy.

1. Clear the immediate cause. Terminate the runaway backends so the pool drains — the command is below. Service returns within seconds.
2. Confirm from the outside, not from the health status. Make a real request through the proxy and check both the status code and the latency.
3. Fix the endpoint. Make /health/ready acquire a connection from the pool and run SELECT 1, with its own short timeout so the probe cannot itself block for 30 seconds.
4. Fix the probe. Point the healthcheck at the readiness endpoint, and set timeout above the readiness check's internal timeout so a slow dependency reports unhealthy rather than being killed mid-check.
5. Prove it fails. Break the dependency deliberately and watch the status flip. Until you have seen that, you have replaced one untested hypothesis with another.

Step one, in full:

$ docker exec db psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 minutes';"

  pg_terminate_backend
----------------------
t
t
(2 rows)

The fixed endpoint and probe:

@app.get("/health/live")
def live():
    # Liveness: is this process able to serve at all?
    return {"status": "ok"}

@app.get("/health/ready")
def ready():
    # Readiness: can it do the job? Own timeout, well under the probe's.
    with db.connect(timeout=2) as conn:
        conn.execute("SELECT 1")
    return {"status": "ready"}

    healthcheck:
      test: ['CMD', 'curl', '-fsS', 'http://localhost:8080/health/ready']
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
      start_interval: 2s

Proving the fix

This is the step almost everyone skips, and it takes ninety seconds. Stop the dependency and watch the transition arrive:

$ docker stop db && docker events --since 0s --until 90s --filter event=health_status --format '{{.Actor.Attributes.name}} {{.Action}}'

api health_status: unhealthy

Then start it again and confirm the container recovers on its own:

$ docker start db && sleep 30 && docker ps --filter name=api --format '{{.Names}} {{.Status}}'

api Up 24 minutes (healthy)

Prevention

Three rules that would have prevented this specific outage:

• A readiness check must fail when the service cannot serve. If you cannot describe a plausible production fault that flips it to unhealthy, it is not a readiness check.
• Test the probe against a broken dependency before you ship it. A healthcheck is code, and it is code with no test coverage unless you write one.
• Never let health status be the only source of truth. The proxy’s error rate, or a synthetic request from outside the host, catches the entire class of failure where the container is fine and the service is not.

Knowledge check

Where next

That closes the health and failure-detection part. Detection is only the first term of the incident timeline; the parts on troubleshooting and incident response cover what happens after the alert fires.