Liveness probes and CrashLoopBackOff — when to restart a container

Liveness probes are the most invasive probe. A liveness probe failure causes the kubelet to restart the container. Use them sparingly and for the right reasons. This lesson covers what liveness probes should test, the difference from readiness, and the diagnostic patterns for CrashLoopBackOff.

What the liveness probe controls

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3

The kubelet runs the liveness probe on the configured schedule. If the probe fails (consecutive failures > failureThreshold), the kubelet kills the container and restarts it. The container’s restartCount increments.

flowchart LR
    Probe[Liveness probe] -->|succeeds| OK[Container OK]
    Probe -->|fails failureThreshold times| Kill[kubelet kills container]
    Kill --> Restart[Container restarts]
    Restart --> Probe

The key property: liveness probes restart the container. They are the only probe that does this. Readiness probes remove the Pod from Endpoints but do not restart. Startup probes gate the readiness/liveness probes during startup.

When to use liveness probes

The right use cases:

• Hangs and deadlocks: the process is alive (responding to signals, taking CPU) but not making progress.
• Stuck event loops: the process is in a state that cannot recover without a restart.
• Memory leaks that the runtime can detect (rare; most memory leaks need external action).

The wrong use cases:

• “Not ready to serve traffic”: use readiness.
• “Still starting up”: use startup.
• “Application error”: an application returning 5xx from /healthz is not a hang; restarting it may not fix the underlying issue and may make things worse.
• “Database is unreachable”: a Pod that depends on a database cannot serve traffic while the database is down. Liveness failure would restart the Pod, but restarting doesn’t fix the database. The right behaviour is to fail readiness (remove from Endpoints) until the database is back.

The probe should test whether the process is making progress, not whether its dependencies are healthy.

The probe logic

A liveness probe that is too strict causes restart loops. The classic example:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

If the application’s /healthz endpoint takes 5+ seconds to respond under load (GC pause, slow query), the liveness probe times out, fails twice in a row, and the kubelet kills the container. The Pod restarts; the application’s health was fine, just slow.

The fix: tune the liveness probe for the application’s actual behaviour. Use a separate /livez endpoint that checks only “is the process responsive” without expensive dependency checks.

@app.get("/livez")
async def livez():
    # Cheap: just confirm the event loop is responsive
    return {"status": "ok"}, 200

@app.get("/ready")
async def ready():
    # More expensive: check dependencies
    if not db.is_connected():
        return {"status": "not ready", "reason": "db"}, 503
    return {"status": "ok"}, 200

The standard Kubernetes pattern: separate /livez (cheap process responsiveness check) from /ready (dependency check).

Liveness loops

A liveness loop is when a liveness probe causes repeated container restarts. Symptoms:

• Pod in CrashLoopBackOff.
• restartCount is high and incrementing.
• kubectl logs --previous shows the same error on every restart.

Common causes:

• Liveness probe is too strict: the probe path is wrong, the probe times out under load, or the application has a bug that the probe detects but a restart doesn’t fix.
• Application cannot recover from a state: e.g., a configuration file is missing and every restart fails the same way.
• Liveness probe runs during shutdown: the application handles SIGTERM by returning Not-Ready, but the liveness probe (running in parallel) returns failure, and the kubelet restarts the container mid-shutdown.

The fix for the third case: implement the liveness endpoint to ignore the shutdown state. Or use a preStop hook that delays SIGTERM long enough for the kubelet to stop running probes.

CrashLoopBackOff — the diagnosis

A Pod in CrashLoopBackOff has containers that start, crash, and restart with backoff. The diagnostic pattern:

1. Get the previous logs: kubectl logs <pod> --previous. This reads the log file of the last container instance that exited.
2. Describe the Pod: kubectl describe pod <pod>. Look at the events for the last few restarts. The reason field tells you what the kubelet saw.
3. Check containerStatuses: kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].state}'. Look for terminated.reason: OOMKilled (memory exceeded), terminated.reason: Error (non-zero exit), or terminated.exitCode.
4. Check the cgroup OOM counter on the node: SSH to the node and read the cgroup’s memory.events. oom_kill 1 confirms a cgroup OOM.

flowchart TD
    Crash[CrashLoopBackOff] --> Q1{--previous logs empty?}
    Q1 -- yes --> OOM{OOMKilled?}
    OOM -- yes --> Mem[Memory limit too low]
    OOM -- no --> Create[Container creation error]
    Q1 -- no --> Q2{Describe events?}
    Q2 -- Liveness --> Live[Liveness probe too strict]
    Q2 -- Failed --> App[Application error on start]
    Q2 -- BackOff --> Image[Image pull issue or runtime error]

Common exit causes:

• Exit code 0: the application exited successfully without completing work (e.g., a startup script that ran once). With restartPolicy: Always, the kubelet restarts it; with Never, the Pod terminates.
• Exit code 1: the application exited with an error. Read the logs.
• Exit code 137 (128+9 SIGKILL): OOMKilled or forced termination. Check memory limits.
• Exit code 143 (128+15 SIGTERM): graceful termination. Read the logs for the application’s shutdown sequence.

Tuning liveness probes

The probe fields, balanced against restart sensitivity:

• periodSeconds: how often the kubelet runs the probe. Default 10. Lower = faster reaction to hangs; higher = less probe load.
• timeoutSeconds: per-probe timeout. Default 1. Must be greater than the application’s typical response time.
• failureThreshold: consecutive failures before restart. Default 3. Higher = tolerate brief stalls (GC).
• initialDelaySeconds: delay before first probe. Default 0. Use a startupProbe instead of a long initialDelay.

For a typical web service:

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

The window from healthy to killed: failureThreshold * periodSeconds = 30s worst case. The probe must respond within timeoutSeconds = 2s per attempt.

Production patterns

Liveness for hangs only:

@app.get("/livez")
async def livez():
    # Confirm the process is responsive
    if event_loop.is_blocked():
        return {"status": "blocked"}, 500
    return {"status": "ok"}, 200

The endpoint checks for actual process hangs. It does NOT check dependencies — those go in /ready.

Liveness probe with exec:

livenessProbe:
  exec:
    command: ["sh", "-c", "test -f /tmp/alive || exit 1"]
  periodSeconds: 30
  failureThreshold: 3

The application writes a heartbeat file every N seconds; the probe checks the file exists. If the application is hung and not writing the file, the probe fails.

Liveness probe with grpc:

livenessProbe:
  grpc:
    port: 9090
  periodSeconds: 10

For gRPC services, this calls the standard gRPC health check. The application must implement the grpc.health.v1.Health service.

Avoiding the restart loop

If a Pod is in CrashLoopBackOff and the application cannot recover, repeated restarts waste resources and may make things worse. The pattern:

1. Diagnose with --previous logs and describe.
2. Fix the underlying issue (config, dependency, application bug).
3. Roll forward with a fixed manifest.
4. If the fix cannot be deployed immediately: scale down the Deployment to 0 replicas to stop the restart loop, then redeploy.

The Deployment controller will recreate Pods as soon as the Deployment’s replicas is non-zero. Use this as a “pause” to stop the loop while you investigate.

Cross-course references

• The Linux course part VI-Linux-Processes covers process restart patterns; liveness probes are the cluster-level equivalent.
• The Docker course part XXX-Docker-Lifecycle covers container restart policy; liveness probes are the cluster-level extension.
• The Ansible course part XXXV-Ansible-Scripting covers service restart discipline; liveness probes are the cluster-level equivalent.

Quiz

Production discipline

• Liveness probes for hangs and deadlocks only. Not for dependencies, not for “ready to serve,” not for “still starting.”
• Separate /livez from /ready. Livez is a cheap process responsiveness check; readyz is a dependency check. They have different purposes.
• Tune liveness probes for the application’s actual response time. A probe with timeoutSeconds: 1 will fail during normal GC pauses.
• Diagnose CrashLoopBackOff with logs --previous and describe. The exit code and reason in containerStatuses[*].state.terminated are the diagnostic keys.
• Scale to 0 if CrashLoopBackOff cannot recover. Stop the restart loop, fix the manifest, redeploy.