Readiness probe failure · Break/Fix · Kubernetes

Reported symptoms

checkout-api runs twelve replicas behind a ClusterIP Service and an Ingress. On Tuesday at 21:04 it began returning 503 to every request. At 21:10 it stopped, without anyone having changed anything.

The post-incident notes read like four different problems:

Nothing crashed. Every Pod was Running for the whole window with a restart count of zero. Every alert the team owns is built on restarts, CrashLoopBackOff or container exit codes, and not one of them fired.
Nothing logged. The application log shows ordinary 200s until 21:04, then silence, then ordinary 200s from 21:10. No exception, no timeout, no connection error.
The health endpoint is fine. kubectl exec into any Pod and curl the readiness path: 200, about twenty milliseconds. During the incident. Afterwards. Every time anyone has tried it.
The dependencies are fine. The database team confirms nothing happened. CoreDNS is healthy. The ingress controller is healthy. A curl from another Pod straight to a checkout-api Pod IP succeeded at 21:06, while the Service itself was returning 503.

The one thing that looked like a lead was timing: the incident started roughly forty seconds after a routine node drain removed three of the twelve replicas. The drain was written up as the cause and the maintenance procedure was changed.

Then it happened again on Wednesday at 12:41. No drain. No deploy. No infrastructure change. Just lunchtime.

Evidence provided

Read-only / Safecaptured at 21:06 — nine of nine not ready, none restarted

$ kubectl -n shop get pods -l app=checkout-api

NAME                            READY   STATUS    RESTARTS   AGE
checkout-api-7f9c4d5b8-2xqjw    0/1     Running   0          6d
checkout-api-7f9c4d5b8-4hn7z    0/1     Running   0          6d
checkout-api-7f9c4d5b8-8bkcm    0/1     Running   0          6d
checkout-api-7f9c4d5b8-9wrtd    0/1     Running   0          6d
checkout-api-7f9c4d5b8-dq4vs    0/1     Running   0          6d
checkout-api-7f9c4d5b8-jm2pl    0/1     Running   0          6d
checkout-api-7f9c4d5b8-n6zxh    0/1     Running   0          6d
checkout-api-7f9c4d5b8-rk8fw    0/1     Running   0          6d
checkout-api-7f9c4d5b8-t3vqn    0/1     Running   0          6d

Illustrative output

Read-only / Saferead the failure text carefully — it is not a status code

$ kubectl -n shop describe pod checkout-api-7f9c4d5b8-2xqjw | grep -E 'Readiness|Unhealthy'

    Readiness:  http-get http://:8080/healthz delay=5s timeout=1s period=5s #success=1 #failure=2
Warning  Unhealthy  21s (x14 over 4m)  kubelet  Readiness probe failed: Get "http://10.244.7.31:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Illustrative output

Read-only / Safenine addresses in the slice, none of them ready

$ kubectl -n shop get endpointslice -l kubernetes.io/service-name=checkout-api -o jsonpath='{.items[*].endpoints[*].conditions.ready}'

false false false false false false false false false

Illustrative output

Read-only / Safea capped CPU quota shared by the request path and the probe path

$ kubectl -n shop get deploy checkout-api -o jsonpath='{.spec.template.spec.containers[0].resources}'

{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"250m","memory":"256Mi"}}

Illustrative output

The metrics for the window, plotted at ten-second resolution:

time      ready_endpoints   req_per_replica   cfs_throttled_ratio
21:03:50       12                 310               0.06
21:04:10        9                 413               0.19
21:04:50        9                 413               0.38
21:05:10        4                 930               0.41
21:05:20        0                   0               0.00
21:05:35        9                   0               0.00
21:05:50        9                 413               0.36
21:06:10        3                1240               0.44
21:06:20        0                   0               0.00

Work the evidence before reading on

Two things in the evidence are doing all the work, and both are easy to read past.

The kubelet event says context deadline exceeded, not HTTP probe failed with statuscode: 503. Those are two different failures. What does each one tell you about whether the application answered?
Work the probe arithmetic from the Readiness: line. How many seconds of consecutive failure remove a Pod from the EndpointSlice, and how many seconds of success put it back? Are those two numbers the same?
Look at the ready_endpoints column. It does not decay towards zero and settle. It collapses, returns to full, and collapses again. What kind of system produces that shape, and what does it need besides a feedback path to produce it?
Look at req_per_replica at 21:05:10 and at 21:05:20. Explain why the exec-and-curl test returns 200 in twenty milliseconds every single time anyone runs it.

Before continuing: the drain was blamed on Tuesday and there was no drain on Wednesday. Was the drain a cause or a trigger, and what distinguishes the two?

Root cause

1. A timeout is not a status code

The kubelet’s HTTP probe opens a connection to the Pod IP and waits timeoutSeconds for response headers. If the application answers with a 5xx, the event records the status code. If it does not answer in time, the event records context deadline exceeded.

The first says the application was asked and declined. The second says the application was never heard from. They lead to completely different places, and this incident produced the second one throughout while being investigated as though it were the first — which is why so much time went into the database, the dependency graph and the application logs, all of which describe a process that was perfectly willing to answer.

2. The probe shares the container’s CPU quota

The handler behind /healthz runs inside the container. The container has limits.cpu: 500m, which the kernel enforces as a CFS quota, so the process is descheduled once it exhausts its slice within each accounting period.

At 21:04 the drain took nine replicas’ worth of traffic and gave it to nine replicas — 413 requests per replica per second where there had been 310. The throttled-periods ratio went from 0.06 to 0.38. A handler that returns in twenty milliseconds on an idle Pod is being interrupted, repeatedly, on a Pod that is at its quota.

The probe is therefore not measuring health. It is measuring how much CPU the container has left, which is a measurement of load.

3. Removing a server from an overloaded pool is positive feedback

timeout=1s period=5s #failure=2 means two consecutive misses, ten seconds apart at worst, and the Pod’s address is marked not-ready in the EndpointSlice. kube-proxy stops sending it traffic.

That traffic does not disappear. It goes to the Pods still in the slice, whose per-replica load rises, whose handlers now miss the same deadline.

Step	Ready endpoints	Requests per replica
Before the drain	12	310
After the drain	9	413
First Pods drop out	4	930
Cascade completes	0	0

The system’s response to overload is to withdraw servers, which is the one action guaranteed to increase overload on everything that remains. Forty seconds after the first missed probe, the Service had no ready addresses at all.

4. Why it recovered by itself, and why that was the worst part

#success=1 at period=5s means one successful probe returns a Pod to the slice, and with zero traffic every probe succeeds immediately. So the entire fleet came back Ready within one period — all nine at once — and full production traffic landed on all of them simultaneously.

Then it collapsed again.

This is why the incident is an oscillation rather than an outage. It also explains the single most misleading piece of evidence in the whole case: any Pod you can reach with kubectl exec during the collapse is a Pod that has been removed from service and therefore has no traffic. Measuring it tells you what it does when idle. The measurement is only possible in the state that guarantees the answer is 200.

Under the hood

An oscillation needs three things: a feedback path, a gain above one, and a delay. This tier had all three, and two of them are in the probe configuration.

The feedback path is readiness to EndpointSlice to kube-proxy to load distribution to handler latency to readiness.

The gain is set by the fleet size. Removing one of nine servers raises load on the rest by about 12 percent; if 12 percent is enough to push the next handler past the deadline, one departure causes more than one departure, and the cascade runs to completion.

The delay is the propagation chain. The kubelet updates Pod status, the API server persists it, the EndpointSlice controller rebuilds the slice, and kube-proxy on every node reprograms its rules. That chain takes a few seconds, which means the system is always acting on a picture of readiness that is several seconds old.

The asymmetry in the probe timing is the other half of it. Leaving takes failureThreshold × periodSeconds — ten seconds. Returning takes successThreshold × periodSeconds — five. The fleet is quicker to re-enter service than to leave it, so it re-enters together, at full load, before any of it has warmed up.

None of these three components is a defect on its own. #failure=2 is a reasonable choice. A 500m CPU limit is a reasonable choice. A few seconds of endpoint propagation is inherent. The failure is the composition, and no component owner would have flagged it in review.

Deleting the probe is not the fix, and neither is the exec test

Two moves are very tempting during this incident and both are wrong.

The first is kubectl exec ... curl localhost:8080/healthz, which returns 200 in twenty milliseconds and reads as proof that the probe is lying. It is not proof of anything. The only Pods you can conveniently exec into during the collapse are Pods that have already been removed from service, and the act of checking them cannot reproduce the condition being investigated. If you must test from inside, test a Pod that is still in the EndpointSlice, and time the request.

The second is removing the readiness probe, or setting it to something that always passes. That restores service immediately, and it does so by routing traffic to Pods regardless of whether they can serve it. It also disarms every rollout you will ever do afterwards: the readiness gate is what stops a broken image from replacing a working one, and a probe that always passes lets the Deployment controller retire the old ReplicaSet in front of a new one that cannot serve a single request.

The probe is not wrong to exist. A one-second deadline on a code path that shares a throttled CPU quota with production traffic is wrong.

Resolution

Scale out first. Adding replicas lowers per-replica load below the point where the handler misses its deadline and takes the fleet out of the oscillating region without changing any configuration. Give it a named owner and an end time, so the temporary replica count does not silently become the architecture.
Measure the p99 of the health path under load, not at idle. Every number in the new probe configuration has to be justified against that measurement; a timeout chosen for looking generous is the same mistake with a larger constant.
Raise timeoutSeconds above that p99 and raise failureThreshold so one bad window cannot deschedule a Pod. Record why each value was chosen next to the value, because the next person to tune it will otherwise only see that it is unusually large.
Make the handler cheap and local. It should report whether this process can accept and serve a request, and it should do no work whose cost grows with traffic. Anything that queries a dependency belongs on a separate endpoint that dashboards read and the kubelet does not.
Raise or remove the CPU limit so the probe is not competing for a capped quota. The Pod stays Burstable and keeps its request; what goes away is the throttling that turns a twenty-millisecond handler into a one-second one.
Re-examine the drain procedure that was changed on Tuesday. The drain was a trigger, not a cause, and a procedure changed to avoid a trigger will not survive the next lunchtime peak.
Add the ratio of ready endpoints to desired replicas as a first-class alert for every Service in the namespace. This is the only signal that moved during the incident; restarts, exit codes and error logs were all flat.
Write the two failure messages into the triage runbook side by side — a status code means the application declined, a context deadline means it never answered — because that one distinction is worth about an hour of an incident.

Verification

The loop reproduces on the old configuration in staging. Drive load until the p99 of the health path approaches one second, remove a quarter of the replicas, and watch the ready count collapse. A fix for a failure you have not reproduced is a hypothesis.
The same test passes on the new configuration, with the fleet staying Ready throughout. Both halves matter: a test that only ever passes proves the test is weak, not that the fix is strong.
Zero Unhealthy events carrying context deadline exceeded across a full production peak. Count them rather than eyeballing them, and count them over a peak rather than over a quiet afternoon.
Ready endpoints as a fraction of desired replicas never falls below the agreed floor, through a drain and through a peak. Restarts stayed at zero for this entire incident, so any verification built on restarts verifies nothing here.
The probe can still fail. Stop the listener in a canary Pod and require it to leave the EndpointSlice inside the window the new timing implies. A probe relaxed enough to stop producing false negatives is easily relaxed enough to stop producing true ones.
The throttled-periods ratio at peak sits below the threshold you set. That is the mechanism that made a twenty-millisecond handler miss a one-second deadline, and it is the number that will tell you when the tier is drifting back towards the same condition.
A rollout still stalls on a genuinely broken image. Deploy a deliberately broken build to a staging replica set and confirm the Deployment refuses to progress. This checks that relaxing the probe did not disarm the gate it exists to provide.

Prevention

A readiness probe must not share fate with the thing it gates. If the probe path competes for the same throttled CPU as the request path, the probe is a saturation detector wired to a capacity switch — and saturation is exactly when capacity must not be withdrawn.
timeoutSeconds is an SLO statement. It must exceed the p99 of the probe path under the worst load the tier is meant to survive, not under the load it happens to be carrying when someone picks the number.
Alert on ready endpoints over desired replicas, per Service. This whole failure class produces no restarts and no error logs, and is therefore invisible to alerting built on either.
Review a probe change as a traffic-routing change, because that is what it is. It belongs in the same review category as a Service selector or an Ingress rule.
Load-test the probe, not only the application. The old configuration passed every test that had ever been run against it, because every one of them ran against an idle Pod.
Never wire a dependency check into a readiness probe on a horizontally-scaled tier. It is this same mistake wearing different clothes: every replica fails at the same instant, and a partially degraded dependency becomes a total outage.
Teach the two messages. statuscode: 503 and context deadline exceeded look equally like “the probe failed” and lead to entirely different investigations.

What if the same settings had been on the liveness probe instead?

Then the recovery would not have taken six minutes; it would have taken half an hour, and it would have looked like an application crash.

A readiness failure removes the Pod from the EndpointSlice and leaves the process alone, which is why this fleet recovered the moment traffic drained away. A liveness failure with the same timing would have had the kubelet kill each container instead. Every container restarts, every restart lands into an empty cache and a cold connection pool, and the restarts accumulate into CrashLoopBackOff — at which point the exponential back-off keeps Pods down for progressively longer periods long after the load that caused the first timeout has gone.

The evidence would also have pointed somewhere else entirely. Restart counts in the hundreds, Killing container events, Back-off restarting failed container: a picture that reads unambiguously as a broken application, and sends the investigation to the code rather than to the probe.

This is the reason a check that is sensitive to load must never be attached to liveness. Liveness answers one question — is this process wedged badly enough that only a restart will help — and load is never the answer to it. Where the two probes share a path and share timing, they also share this failure mode, and the liveness half of it is far more expensive.

Why this matters

Six minutes of total outage on a checkout tier, a second one the following day, a maintenance procedure changed for the wrong reason, and not one alert in the estate that fired.

The mechanism worth carrying is that a readiness probe is a control loop, not a test. It observes the workload and it acts on the workload, and any loop that observes a quantity affected by its own action can oscillate. The probe here was well within its documented behaviour on every axis; the composition of a one-second deadline, a capped CPU quota, an asymmetric failure and success threshold, and a few seconds of endpoint propagation turned nine healthy Pods into a system with two stable-looking states and no willingness to stay in either.

The habit that generalises is to distrust a measurement that can only be taken in the condition that guarantees the answer. Exec-and-curl returns 200 because the Pod you can reach is the Pod with no traffic. In any investigation, ask what the act of measuring has changed before you believe what it returned.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. A timeout is not a status code

2. The probe shares the container’s CPU quota

3. Removing a server from an overloaded pool is positive feedback

4. Why it recovered by itself, and why that was the worst part

Resolution

Verification

Prevention