Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

advancedkubernetes-pod~35 min

Liveness probe restart loop

Reported symptoms

  • ●The ingress 5xx rate for the checkout service crossed two percent at 09:38 and has not come back down
  • ●The database team reports that the connection-establishment rate on the payments primary is roughly twenty times its normal value
  • ●Every checkout-api replica shows STATUS Running with a RESTARTS count that climbs by one every two to four minutes
  • ●No Pod ever displays CrashLoopBackOff, so the alert the team actually watches for never fired
  • ●Nothing was deployed: the last rollout finished six days ago and the running image digest is unchanged
  • ●CPU on two of the five workers is visibly elevated, which sent the first hour of the incident into a noisy-neighbour hunt

Evidence

  • · kubectl get pods -l app=checkout-api shows RESTARTS between 6 and 11 with STATUS Running on all twelve replicas
  • · kubectl describe pod reports Last State Terminated, Reason Completed, Exit Code 0 - not OOMKilled, not Error
  • · The same describe output shows Liveness and Readiness both configured as http-get on :8080/healthz
  • · Events carry Warning Unhealthy from the kubelet naming the liveness probe, followed immediately by Normal Killing
  • · kubectl logs --previous ends with an ordinary SIGTERM shutdown sequence and no error of any kind
  • · A timed request to /healthz from inside a replica returns in about 1.9s under load and about 20ms off peak
  • · kubectl get endpointslice shows the checkout-api slice collapsing from twelve ready addresses to two and recovering, repeatedly
Diagnosis and resolutionclick to reveal

Root cause

The liveness probe and the readiness probe both point at /healthz, and /healthz performs a dependency check: it queries the database and pings the cache before returning 200. The probe carries timeoutSeconds: 1. Under morning load that dependency check takes longer than a second, so the kubelet records three consecutive liveness failures within thirty seconds and kills the container - a container that was healthy, responsive, and serving traffic. This is the failure the course names explicitly: a liveness probe must test whether the process is making progress, never whether its dependencies are reachable. Two consequences follow from the single mistake and they are what make the incident look like three separate faults. Because liveness restarts the container, every replica drops its database connection pool and the replacement immediately opens a new one, which is the connection storm the database team is seeing - and that storm slows the database, which slows /healthz, which fails more probes. Because readiness reads the same endpoint, every replica leaves the EndpointSlice at roughly the same moment, which turns a partial slowdown into a total loss of endpoints and the 5xx at the edge. The loop is self-feeding: the restarts are now the main reason the probe keeps failing.

Remediation

Break the loop before fixing the manifest, because the restarts are currently sustaining the fault. The cheapest intervention that stops the killing is a patch to the liveness probe alone - raise timeoutSeconds to comfortably exceed the endpoint's observed p99 and raise failureThreshold - and it is worth being explicit that this patch is itself a rollout, so it restarts every replica once. That single controlled restart is cheaper than a restart every three minutes, but it must be batched through maxUnavailable rather than dropped on the database all at once. If the incident is severe enough that even one more restart wave is unacceptable, the alternative hold is to remove the liveness probe entirely, which costs you automatic recovery from a genuine hang; that is a legitimate choice only with a named owner and a stated expiry, because a probe removed during an incident is a probe nobody reinstates. The durable fix is to split the endpoint: a cheap /livez that confirms the process is responsive and touches nothing external, pointed at by the liveness probe, and /readyz retaining the dependency check for readiness. Size the liveness timeout from the measured latency of /livez, not from a default.

Verification

Verify from the restart counter and from the edge, not from a single healthy probe response. Record restartCount for every replica, wait at least one hour spanning a peak period, and require that no counter has moved; a probe that passes once proves nothing about a fault that only appears under load. The EndpointSlice for the Service must hold all replicas continuously rather than oscillating, and the ingress 5xx rate must return to its pre-incident baseline, which is the only signal independent of both the kubelet and the application. Then prove the guard can still fail, because a liveness probe that can no longer kill anything is not a liveness probe: on one canary replica, point the liveness probe at a path that returns 503, confirm the kubelet restarts that container within failureThreshold multiplied by periodSeconds, and revert. Finally re-read the probe stanza in kubectl describe on a freshly created Pod to confirm the new configuration is what the kubelet is actually running.

Prevention

Treat the liveness probe as the one probe that is allowed to destroy running state, and give it the narrowest possible test: process responsiveness, no network calls, no dependency checks, no shared endpoint with readiness. Derive timeoutSeconds and failureThreshold from measured latency rather than from the defaults, since the default one-second timeout is shorter than a garbage collection pause on many runtimes. Recognise that a readiness probe which checks a shared dependency fails on every replica simultaneously and therefore converts a degraded dependency into a complete outage; a service that can serve stale or read-only responses should stay ready and degrade instead. Alert on the rate of change of restart counts rather than only on CrashLoopBackOff, because restarts spaced minutes apart never enter that state and this incident was invisible to the alert the team trusted. Review probe configuration at the same time as resource requests during any latency investigation, and keep a record of which endpoints each probe calls so the next person does not have to read application source during an incident.

Reported symptoms

At 09:38 on a Tuesday the on-call engineer for checkout-api takes three alerts inside four minutes, from three systems that do not talk to each other:

  • The edge: 5xx rate on the checkout hostname crosses two percent and keeps climbing.
  • The database team, by chat: the connection-establishment rate on the payments primary is roughly twenty times its usual value. Query latency has risen but no single query looks pathological.
  • The platform dashboard: CPU on worker-2 and worker-3 is noticeably higher than on the other three workers.

The checkout-api Deployment runs twelve replicas on a five-node kubeadm cluster at v1.34.1. Nothing has been deployed for six days; the running image digest matches the one recorded at the last rollout.

The first hour goes into the CPU signal, because it is the only one that points at a place rather than a symptom. That hour produces nothing: the two hot nodes host four of the twelve replicas, the other eight replicas are on quiet nodes, and all twelve are equally affected.

What nobody looks at for that hour is the RESTARTS column, because every Pod reads Running and 1/1, and the alert the team trusts fires on CrashLoopBackOff, which never appears.

Read-only / SafeRunning, and restarting every few minutes
$ kubectl get pods -n prod -l app=checkout-api
NAME                            READY   STATUS    RESTARTS      AGE
checkout-api-6f9c4d7b8-2xk4m    1/1     Running   9 (2m ago)    6d
checkout-api-6f9c4d7b8-4tzq9    0/1     Running   7 (41s ago)   6d
checkout-api-6f9c4d7b8-8n6vp    1/1     Running   11 (3m ago)   6d
checkout-api-6f9c4d7b8-9wqrs    1/1     Running   6 (94s ago)   6d

Illustrative output

Six days of uptime and eleven restarts, all of them this morning.

Evidence provided

The Pod description carries the whole case, if you read past the first screen.

Read-only / Safetwo probes, one endpoint, one-second timeout
$ kubectl describe pod -n prod checkout-api-6f9c4d7b8-8n6vp
    State:          Running
    Started:      Tue, 18 Aug 2026 09:52:11 +0000
  Last State:     Terminated
    Reason:       Completed
    Exit Code:    0
    Started:      Tue, 18 Aug 2026 09:49:02 +0000
    Finished:     Tue, 18 Aug 2026 09:52:09 +0000
  Ready:          True
  Restart Count:  11
  Liveness:       http-get http://:8080/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
  Readiness:      http-get http://:8080/healthz delay=5s timeout=1s period=5s #success=1 #failure=3

Illustrative output

Reason: Completed with Exit Code: 0 is the single most informative line on the page. It says the previous container was asked to stop and stopped cleanly. It is not OOMKilled, so this is not the memory limit. It is not Error with a non-zero code, so the application did not crash.

Read-only / Safethe kubelet names the probe and the reason
$ kubectl get events -n prod --field-selector involvedObject.name=checkout-api-6f9c4d7b8-8n6vp --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON      OBJECT                             MESSAGE
3m          Warning   Unhealthy   pod/checkout-api-6f9c4d7b8-8n6vp   Liveness probe failed: Get "http://10.244.3.17:8080/healthz": context deadline exceeded
3m          Normal    Killing     pod/checkout-api-6f9c4d7b8-8n6vp   Container checkout failed liveness probe, will be restarted
3m          Normal    Pulled      pod/checkout-api-6f9c4d7b8-8n6vp   Container image already present on machine
3m          Normal    Started     pod/checkout-api-6f9c4d7b8-8n6vp   Started container checkout

Illustrative output

The probe did not receive an error status. It timed out.

Read-only / Safethe container that died was healthy when it was asked to stop
$ kubectl logs -n prod checkout-api-6f9c4d7b8-8n6vp --previous --tail=6
09:52:07 INFO  received SIGTERM, beginning shutdown
09:52:07 INFO  draining 34 in-flight requests
09:52:08 INFO  closing database pool (18 connections)
09:52:09 INFO  shutdown complete

Illustrative output

Read-only / Safethe endpoint is a hundred times slower under load
$ kubectl exec -n prod checkout-api-6f9c4d7b8-2xk4m -- wget -q -O /dev/null -T 5 http://127.0.0.1:8080/healthz
real  0m1.94s
(same command at 04:00 the same morning: real  0m0.02s)

Illustrative output

Read-only / Safetwo of twelve endpoints ready, sampled during the incident
$ kubectl get endpointslice -n prod -l kubernetes.io/service-name=checkout-api -o jsonpath='{.items[*].endpoints[*].conditions.ready}'
true true false false false false true false false false false false

Illustrative output

Work the evidence before reading on

The application has not changed. The image has not changed. The probe configuration has not changed either - it has been in the manifest for months.

  1. The previous container exited with code 0 and reason Completed. Which hypotheses does that single fact eliminate, and what is the only actor that asks a healthy container to stop?
  2. The liveness and readiness probes call the same path. What does each one do when that path is slow, and how do those two behaviours combine?
  3. /healthz takes 1.9 seconds under load. The liveness probe carries timeout=1s, period=10s, failure=3. How long does a replica survive?
  4. Every restart closes a connection pool and the replacement opens a new one. Twelve replicas are doing this every few minutes. What does that do to the thing /healthz is measuring?

Before continuing: name the one change that would stop the restarts, the one change that would stop the 5xx, and explain why they are not the same change.

Root cause

1. The liveness probe is testing the database

/healthz queries the database and pings the cache before returning 200. That is a perfectly reasonable readiness check: a replica that cannot reach its datastore should not receive traffic.

It is a catastrophic liveness check. The course states the rule directly: a liveness probe should test whether the process is making progress, not whether its dependencies are healthy. Restarting a container does not repair a database. All it does is destroy a working process and everything it was holding.

With timeout=1s, period=10s and failure=3, a replica whose dependency check has slowed past one second is killed thirty seconds later. Nothing about the container was wrong. The kubelet did exactly what the manifest told it to do.

2. The restart is what keeps the probe failing

This is the part that turns a bad configuration into an incident that will not settle.

Each liveness kill sends SIGTERM, the application closes its pool, and the replacement container opens a fresh pool on startup. Twelve replicas restarting on a two-to-four minute cycle produce a sustained storm of new connections against the primary. Connection establishment is expensive; the database spends its time on setup rather than on queries; queries get slower; /healthz gets slower; more probes time out.

flowchart LR
    A["/healthz slows past 1s"] --> B["3 liveness failures in 30s"]
    B --> C["kubelet kills the container"]
    C --> D["connection pool torn down and rebuilt"]
    D --> E["connection storm on the primary"]
    E --> A

The trigger was ordinary morning load. The reason it did not subside when load plateaued is that the restarts became the dominant source of the latency they were reacting to.

3. The readiness probe made it an outage instead of a slowdown

Readiness reads the same endpoint. When /healthz slows, it slows for every replica at once, because they all depend on the same database. So every replica fails readiness within the same few seconds and leaves the EndpointSlice together.

That is why the edge saw 5xx rather than elevated latency. A Service whose EndpointSlice has two ready addresses out of twelve is not a degraded service; with four of those twelve simultaneously restarting, it is periodically a service with nowhere to send a request at all.

Three alerts, three teams, one probe stanza.

Resolution

  1. Confirm the diagnosis on a second replica before changing anything: the same Exit Code 0 with reason Completed, the same Unhealthy event naming the liveness probe, and the same probe stanza. One Pod is an anecdote.
  2. Decide between the two holds explicitly and write the choice down. Patching the liveness probe stops the kills but is itself a rollout; removing the liveness probe stops the kills without a rollout of the probe logic but leaves a genuine hang unrecoverable until someone notices.
  3. If you patch: raise timeoutSeconds well above the observed p99 of the endpoint and raise failureThreshold, then set maxUnavailable so the resulting restart wave is batched rather than simultaneous. A single controlled wave is cheaper than a restart every three minutes; an uncontrolled one is another connection storm.
  4. If you remove the probe instead, record the owner and the time it goes back in. A liveness probe deleted during an incident is a liveness probe that stays deleted.
  5. Watch the database connection-establishment rate as the restarts stop. It should fall within one or two probe periods, and its fall is the confirmation that the loop was self-feeding rather than externally driven.
  6. Add a cheap /livez endpoint to the application that confirms only that the process is responsive: no database, no cache, no outbound call of any kind.
  7. Point the liveness probe at /livez and leave the readiness probe on the dependency-checking endpoint, renamed /readyz so nobody re-merges them by accident.
  8. Size the liveness timeout from the measured latency of /livez plus headroom for a garbage collection pause, not from the one-second default.
  9. Separately, review whether the readiness check should fail at all when the datastore is slow. A checkout service that can serve reads from cache is more useful ready and degraded than uniformly unready.

Verification

  1. Restart counters are frozen. Record restartCount for all twelve replicas, wait at least an hour that spans a load peak, and require that not one has moved. A probe that succeeds once tells you nothing about a fault that only appears under load.
  2. The EndpointSlice is stable and full. Sample the ready conditions repeatedly rather than once; the broken state produced a slice that was briefly correct between collapses.
  3. The edge agrees. The 5xx rate for the checkout hostname is back to its pre-incident baseline. This is the only check that is independent of both the kubelet and the application.
  4. The database is calm. Connection-establishment rate is back to its normal band, which also confirms the connection storm was a consequence rather than a cause.
  5. The guard can still fail. On one canary replica, point the liveness probe at a path that returns 503 and confirm the kubelet restarts that container within failureThreshold multiplied by periodSeconds. Then revert. A liveness probe that has only ever passed is an untested liveness probe.
  6. The running configuration is the intended one. Read the Liveness and Readiness lines from kubectl describe on a Pod created after the change, not from the manifest in Git - the manifest is what you meant, the describe output is what the kubelet is enforcing.
  7. No Unhealthy events for a full business day: query events filtered on reason Unhealthy across the namespace, and require an empty result over a period that includes a peak.

Prevention

  • Give the liveness probe the narrowest test in the system: is this process responsive. No database, no cache, no outbound HTTP. It is the only probe that destroys running state, so it gets the check least likely to be wrong.
  • Never let liveness and readiness share an endpoint. The moment they share one, the readiness semantics win by default and the liveness probe silently becomes a dependency check.
  • Derive timeoutSeconds and failureThreshold from measured latency. The one-second default is shorter than a garbage collection pause on several common runtimes, which makes the default probe a periodic restart on any workload with a large heap.
  • Understand that a readiness check on a shared dependency fails everywhere simultaneously, and therefore promotes a degraded dependency into a total outage. Where the service can degrade instead - stale reads, cached responses, a queued write - staying ready is the better behaviour.
  • Alert on the rate of change of kube_pod_container_status_restarts_total, not only on CrashLoopBackOff. Restarts spaced minutes apart never enter that state, and this incident was invisible to the alert the team trusted.
  • Record, next to each Deployment, which endpoint each probe calls and what that endpoint touches. During an incident nobody should have to read application source to find out whether a health check talks to the database.