Break/Fix 19: Every healthcheck is green and every request times out

Reported symptoms

The pager fires on edge 504s. The first thing anyone checks is whether the backends are up, and they emphatically are:

NAMES        STATUS                    PORTS
app-1        Up 6 days (healthy)       8080/tcp
app-2        Up 6 days (healthy)       8080/tcp
app-3        Up 6 days (healthy)       8080/tcp

Six days of uptime, zero restarts, three healthy replicas, and not one request completing. The investigation goes to the proxy, then to the network, then to the database — the application is the last place anyone looks, because the application says it is fine.

Diagnosis

Start by refusing to accept the healthcheck as evidence. Ask what it actually tested.

docker inspect app-1 --format '{{json .Config.Healthcheck}}' | jq
docker inspect app-1 --format '{{json .State.Health}}' | jq '.Log[-3:]'

{
  "Test": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
  "Interval": 30000000000,
  "Timeout": 5000000000,
  "Retries": 3
}

Now read the handler. If /health returns a constant, the probe has proved exactly two things: the process exists, and the listener accepts connections. Both are true of a completely deadlocked service.

Confirm the deadlock rather than inferring it:

docker stats --no-stream app-1
docker exec app-1 ss -tn state established '( sport = :8080 )' | wc -l

Near-zero CPU with the connection count pinned exactly at the worker pool size is the signature. A busy service has varying counts; a deadlocked one has a count equal to its concurrency limit and a flat CPU line, because every worker is parked in a blocking call.

Then find what they are parked on:

# Go
docker exec app-1 kill -QUIT 1        # if the app dumps goroutines on SIGQUIT
# Java
docker exec app-1 jstack 1 | grep -A5 'BLOCKED\|WAITING'
# any language: what is the process actually waiting for
docker exec app-1 cat /proc/1/task/*/stack 2>/dev/null | head

On the dependency side, the counterpart evidence is usually unambiguous:

SELECT pid, state, wait_event_type, query_start, left(query, 60)
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY query_start;

Sessions idle in transaction, hours old, holding locks. Every application worker is queued behind them, waiting forever because no acquisition timeout was ever set.

Resolution path

1. Mitigate first: restore service before you fix the design.
2. Terminate the stuck dependency sessions (SELECT pg_terminate_backend(pid) ... for the idle-in-transaction set), or restart the replicas in a rolling fashion. Note the time — you have just destroyed the evidence, so capture the dumps first.
3. Make the probe capable of failing.
4. Replace the static /health with two endpoints: /livez, which stays trivial and answers "the process is not wedged", and /readyz, which acquires a pooled connection, runs SELECT 1 under a 2-second timeout, and returns 503 when the pool is exhausted.
5. Point the container healthcheck at readiness.
6. ```
7. HEALTHCHECK --interval=15s --timeout=3s --start-period=45s --retries=3 \
8. CMD curl -fsS http://localhost:8080/readyz || exit 1
9. ```
10. Keep timeout below interval so probes cannot overlap, and set start-period long enough to cover the slowest legitimate boot.
11. Give the application a bounded wait.
12. Set a pool acquisition timeout and a statement timeout. A worker that fails in two seconds is an error rate you can see; a worker that waits forever is an outage you cannot.
13. Fix the leak that started it.
14. Idle-in-transaction sessions mean a code path opens a transaction and does not close it on some branch. Set idle_in_transaction_session_timeout on the database as a backstop.
15. Make the proxy act on readiness.
16. Confirm the edge proxy health check targets /readyz too, so a container that reports unready is actually drained rather than merely labelled.

Verification

1. The probe can fail on purpose. In staging, stop the database and confirm the container reports unhealthy within interval * retries — 45 seconds with the settings above.
2. Docker records the failure. docker inspect app --format "{{json .State.Health}}" | jq ".Log[-1]" shows a non-zero ExitCode and the 503 body.
3. The proxy drains the unready replica. Confirm in the proxy stats, not by assumption.
4. Recovery is automatic. Restore the dependency; the container returns to healthy with no restart and RestartCount unchanged.
5. Production recovers on the real signals. Request success rate and p99 latency return to baseline, and the established-connection count moves off the worker cap.

Prevention

• Apply one test to every probe you own: name a failure it would miss. If the list includes “the database is unreachable” or “the worker pool is exhausted”, the probe is not measuring availability.
• Never let an application wait without a deadline. Every pool acquisition, every outbound call, every query gets a timeout. A system with no timeouts converts every dependency slowdown into a total deadlock, and there is no probe clever enough to fix that.
• Alert on the SLI, not the probe. Success rate and latency measured at the edge would have paged in the first minute of this incident; probe state never would have, in six days.
• Exercise the failure. A dependency-kill drill in staging, once a quarter, is the only thing that proves the probe, the proxy and the restart policy all behave the way the diagram says.