A four-second storage stall turned into a two-hour outage, and the database never recovered on its own after the storage came back

Reported symptoms

Storage reports a four-second write stall at 14:02:11 during a controller failover, and confirms normal service from 14:02:15.

Database latency rises from 4 ms to 190 ms within thirty seconds — and does not come back down. Two hours later, with storage healthy the whole time, latency is still above 150 ms and the application is timing out.

Connection count climbed from 90 to 465 in those same thirty seconds and has stayed there.

Host CPU is pinned at 100 percent, almost all of it user time. iowait is under 2 percent.

No single query is slow. The same statements run in a few milliseconds when tested from a separate session.

Restarting the application tier restores service within a minute, and nobody can explain why.

Evidence provided

The pool is configured with a minimum of 5 and a maximum of 20 per pod, and it grows when a checkout waits longer than 50 ms. There are 40 pods, so its ceiling is 800 — the observed 465 is well inside it.

Average query duration measured at the database is 190 ms. The same query from a separate psql session takes 1.4 ms.

And this is the curve for this hardware:

$ for C in 4 8 16 32 64 128 256 400; do pgbench -n -S -c $C -j 4 -T 15; done

 clients          tps   avg_latency_ms
     4  38876.335703            0.103
     8  52719.473989            0.152
    16  67119.364271            0.238
    32 124820.373482            0.256
    64 173535.067834            0.369
   128 161277.529189            0.794
   256 141665.950566            1.807
   400 125352.085856            3.191

The application restart dropped connection count to 200 and latency to 6 ms within 40 seconds.

Work the evidence before reading on

1. Storage was healthy from 14:02:15. Why was the database still slow at 16:00?
2. CPU is saturated in user time and iowait is under 2 percent. What does that rule out?
3. The curve peaks at 64 clients. Where on it was the cluster sitting?
4. Why did restarting the application work, and what does that tell you about what is missing?

Root cause

The loop

The restart worked because it is the only thing that reduces connections

Nothing in the database, the pool, or the platform could bring the count down. The application restart broke the loop by force.

That the system had exactly one recovery mechanism, that it required a human, and that the human could not explain why it worked — that is the actual defect. The stall is routine; controllers fail over.

Resolution

Break the loop by reducing connections. Nothing else works while it is running, and the database cannot break it alone.

If you can restart or scale down the application tier, that is the fastest path and it is legitimate. Do it deliberately rather than as a mystery:

kubectl rollout restart deployment/orders-api

If you cannot, shed connections from the database side, and keep shedding as pools reopen them, until the count is near the knee:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'idle'
  AND state_change < now() - interval '30 seconds';

Confirm you are treating the right thing before touching storage or queries:

SELECT count(*) FILTER (WHERE state = 'active') AS active,
       count(*) FILTER (WHERE state = 'idle')   AS idle,
       count(*)                                 AS total
FROM pg_stat_activity WHERE backend_type = 'client backend';

SELECT wait_event_type, wait_event, count(*)
FROM pg_stat_activity
WHERE backend_type = 'client backend' AND state = 'active'
GROUP BY 1,2 ORDER BY 3 DESC LIMIT 10;

A storm shows waits concentrated in LWLock and Lock. A real storage problem shows waits in IO and iowait on the host. Different pictures, different responses — and two hours went into the wrong one here.

Once the count is near the knee, latency collapses within seconds. There is nothing else to fix in the database.

Verification

Connection count sits near the measured knee and latency is normal.

Host CPU is no longer saturated.

Test that the loop is broken. Inject a stall deliberately in a maintenance window and watch the connection count rise and then return, with no operator involved. A system that recovers only when a human restarts it has not been fixed — it has been reset.

pods x pool_max is under the knee, and that arithmetic is written where the next person to change the pod count will see it.

If a pooler was introduced, backend count stays flat while client count varies. That property is directly observable:

$ pgbench -f real.sql -c 400 -j 4 -T 25   # against the server, then against the pooler

  direct 400 clients : 2042 tps, 195.9 ms, 465 server backends
pooled 400 clients : 2243 tps, 178.4 ms,  65 server backends

Prevention

Cap the pool below the knee, and make it queue rather than grow. A pool that adds connections in response to latency is a feedback amplifier pointed at your database. A pool that queues converts overload into fair waiting — survivable, and visible.

Measure the knee once, for your hardware and workload. It is a short benchmark and it turns pool sizing from an argument into arithmetic.

Put a pooler in transaction mode in front, sized at the knee.

Alert on connection count, not only on latency. Count moved thirty seconds before anybody noticed the latency, and count is what you act on.

Alert on the shape of the wait events. LWLock/Lock dominating means contention; IO dominating means storage.

Set idle_session_timeout for application roles, so connections opened during a burst do not persist for two hours:

ALTER ROLE app SET idle_session_timeout = '5min';

Rehearse the trigger. A four-second storage stall is routine. If the system cannot absorb one without a human, that is worth learning before the controller fails over on its own schedule.