Probe failures — readiness, liveness, and startup

A probe failure is the kubelet removing the Pod from the Service (readiness) or restarting the Pod (liveness). The events and the probe configuration are the diagnostic. The kubelet emits an Unhealthy event naming the probe and the reason it returned — the HTTP status code, the exit code, or the timeout.

The three probes

A Pod can have three probes:

1. Readiness probe. Determines whether the Pod is ready to serve traffic. If the probe fails, the Pod is removed from the Service’s EndpointSlice.
2. Liveness probe. Determines whether the Pod is alive. If the probe fails, the kubelet restarts the container.
3. Startup probe. Determines whether the Pod has started. Until the startup probe succeeds, the readiness and liveness probes are not run.

flowchart TD
    A[Pod starts] --> B{Startup probe?}
    B -->|Configured| C{Startup passes?}
    B -->|Not configured| D[Readiness + Liveness]
    C -->|Yes| D
    C -->|No| E[Container killed]
    D --> F{Readiness passes?}
    F -->|Yes| G[Pod in EndpointSlice]
    F -->|No| H[Pod removed from EndpointSlice]
    D --> I{Liveness passes?}
    I -->|Yes| J[Container runs]
    I -->|No| E

The three probes are designed for different failure modes. A misconfigured probe causes the kubelet to act on the workload in the wrong way.

The readiness probe failure

A readiness probe failure is the most common. The Pod is running, but the Ready condition is False. The EndpointSlice does not include the Pod. The Service routes no traffic to the Pod.

A real kubectl describe pod for a readiness probe failure:

Containers:
  billing:
    Image:      registry.example.com/billing:1.2.3
    State:      Running
      Started:    Fri, 16 Aug 2026 04:23:01 +0000
    Ready:      False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  256Mi
    Readiness:      http-get http://:8080/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  ContainersReady  False
  PodScheduled   True
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Unhealthy       2m    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

The events tell the operator:

• The readiness probe is failing with HTTP 503.
• The probe is http-get http://:8080/healthz with a 10s delay, 1s timeout, 5s period, 1 success, 3 failures.

The remediation is to investigate why the application is returning 503 on /healthz.

The liveness probe failure

A liveness probe failure causes the kubelet to restart the container. The Pod is in CrashLoopBackOff (effectively) because the kubelet is restarting it faster than the liveness probe’s failure threshold.

A real kubectl describe pod for a liveness probe failure:

Containers:
  billing:
    Image:      registry.example.com/billing:1.2.3
    State:      Waiting
      Reason:   CrashLoopBackOff
    Last State: Terminated
      Reason:   Completed
      Exit Code:  0
    Ready:      False
    Restart Count:  12
    Liveness:    http-get http://:8080/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Unhealthy       1m    kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Killing         1m    kubelet            Killing container billing
  Warning  BackOff         30s   kubelet            Back-off restarting failed container billing

The events tell the operator:

• The liveness probe is failing with HTTP 500.
• The kubelet is killing the container.
• The container is restarting.

The remediation is to investigate why the application is returning 500 on /healthz or to fix the liveness probe’s configuration.

The startup probe failure

A startup probe failure causes the kubelet to kill the container before the readiness and liveness probes run. The Pod is in CrashLoopBackOff because the application cannot start.

Containers:
  billing:
    Liveness:     http-get http://:8080/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:8080/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
    Startup:      http-get http://:8080/healthz delay=10s timeout=1s period=10s #success=1 #failure=30
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Unhealthy       5m    kubelet            Startup probe failed: HTTP probe failed with statuscode: 503
  Warning  Killing         5m    kubelet            Killing container billing

The startup probe is the gate; it runs before the readiness and liveness probes. If the application is slow to start, the startup probe’s failureThreshold (30 in this example) is the allowance.

Common causes of probe failures

• Wrong path. The probe’s path is wrong (e.g., /healthz instead of /health).
• Wrong port. The probe’s port is wrong.
• Slow startup. The application takes longer than the startup probe’s threshold to start.
• Wrong status code. The application returns 200 for /healthz but the probe expects 204.
• External dependency. The application’s /healthz checks the database, and the database is unreachable.

Production discipline

A probe failure is the kubelet acting on the workload. The discipline is to read the events, identify the probe, and apply the remediation. A cluster that has frequent probe failures is a cluster that has misconfigured probes or buggy workloads.

• Distinguish readiness, liveness, and startup. The three probes have different effects.
• Fix the probe, not just the application. A probe that is too aggressive is a probe that will fail again.
• Test the probe in staging. The probe’s behavior is the same in production; the difference is the consequence.

Quiz