Probes — liveness, readiness, and startup

Probes are the kubelet’s health checks. They are the primary mechanism the cluster uses to know whether a Pod is alive, ready to serve traffic, or still starting. The kubelet runs the probe handlers in the container’s network namespace on a configured schedule. The probe result feeds the Pod’s Status.Conditions and the Service EndpointSlice. This lesson walks the three probe types, the four probe executors, and the operational patterns.

The three probe types

A Probe is a per-container spec. The Pod spec can declare up to three probes per container:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

The three types:

• Liveness probe: determines if the container is alive. If the probe fails, the kubelet restarts the container. The probe is the “is the process” check.
• Readiness probe: determines if the container is ready to serve traffic. If the probe fails, the kubelet removes the Pod from the Service’s EndpointSlice. The probe is the “is the process ready” check.
• Startup probe: determines if the container has started. The probe runs only at startup; the liveness and readiness probes do not run until the startup probe succeeds. The probe is the “is the process fully started” check.

The three probes have distinct semantics. A failing liveness probe is a process that is unhealthy; the kubelet restarts it. A failing readiness probe is a process that is healthy but not yet serving traffic; the kubelet removes the Pod from the EndpointSlice. A failing startup probe is a process that has not yet started; the kubelet waits.

The four probe executors

The kubelet supports four probe executors:

• HTTP: httpGet issues an HTTP GET to the configured path and port. The probe succeeds if the response code is 2xx or 3xx. The kubelet does not verify the response body.
• TCP: tcpSocket opens a TCP connection to the configured port. The probe succeeds if the connection is established. The kubelet does not send any data.
• gRPC: grpc issues a gRPC health-check request. The probe succeeds if the response is SERVING. The kubelet requires the application to implement the gRPC health-checking protocol.
• exec: exec runs a command inside the container. The probe succeeds if the command exits with code 0. The kubelet runs the command via the runtime’s exec API.

The default executor depends on the application. An HTTP service uses httpGet; a database uses tcpSocket; a gRPC service uses grpc. An exec probe is the most flexible but the most expensive (it requires a process spawn).

The probe timing

The probe’s timing fields:

• initialDelaySeconds: the time to wait before the first probe. The kubelet does not run the probe before this delay.
• periodSeconds: the time between probes. The kubelet runs the probe every periodSeconds.
• timeoutSeconds: the time the probe has to succeed. If the probe takes longer, the kubelet considers the probe failed.
• failureThreshold: the number of consecutive failures before the probe is considered failed. The default is 3.
• successThreshold: the number of consecutive successes before the probe is considered successful. The default is 1 for liveness and startup; 1 for readiness (the readiness probe only fires on failure, not on success).

The total time for a probe to fail is initialDelaySeconds + (failureThreshold * periodSeconds). A 30-second startup probe with failureThreshold: 30 and periodSeconds: 10 gives the application 5 minutes to start.

The probe semantics

A failing liveness probe restarts the container. The restart is graceful; the kubelet sends SIGTERM to the container, waits for the grace period, then sends SIGKILL. The container’s restartCount is incremented.

stateDiagram-v2
    [*] --> Running: container starts
    Running --> ProbeFailed: probe fails
    ProbeFailed --> ProbeFailed: increment failureCount
    ProbeFailed --> Restarting: failureCount >= failureThreshold
    Restarting --> Running: SIGTERM, then SIGKILL, then restart

A failing readiness probe removes the Pod from the EndpointSlice. The Pod is still running; the kubelet continues to run the probe. When the probe succeeds, the Pod is added back to the EndpointSlice.

A failing startup probe prevents the liveness and readiness probes from running. The startup probe is the gate that the other two use.

The probe’s failure modes

The failure modes of a probe:

The kubectl describe pod shows the probe’s last failure and the event. The kubelet’s logs show the probe’s detailed output.

The probe’s design

A probe is a contract between the application and the cluster. The contract should be:

• Liveness: “the container is alive, restart me if I’m not.” The probe should catch unrecoverable failures (a deadlock, a corrupt state). The probe should not catch transient failures (a slow request, a database reconnect).
• Readiness: “the container is ready to serve traffic.” The probe should return success when the application is ready to serve and failure when it is not. The probe should return success before the application’s first request.
• Startup: “the container is fully started.” The probe should return success when the application has finished its initial work. The probe should not return success before the application is ready.

The probe’s path and port should be specific to the health check. A probe that checks the same path as the application’s main route is a probe that returns success on the application’s first request, regardless of the application’s state.

The probe’s effects on the cluster

The probe’s effects:

• Liveness: the kubelet restarts the container. The restart is logged; the container’s restartCount is incremented. The cluster’s kube_pod_container_status_restarts_total counter is incremented.
• Readiness: the kubelet removes the Pod from the EndpointSlice. The Pod’s traffic is reduced to zero; the Pod is still running. The cluster’s kube_pod_container_status_ready flag is set to false.
• Startup: the kubelet delays the liveness and readiness probes. The container is running but not yet checked.

The cluster’s metrics expose the probe’s success rate. The kube_pod_container_status_last_terminated_reason field records the reason for the last termination.

The probe’s gotchas

• A liveness probe that runs kill -0 is a probe that always succeeds. The probe should check the application’s actual state, not the process’s existence.
• A readiness probe that requires the database to be reachable is a probe that fails during a database failover. The probe should check the application’s state, not the database’s state.
• A startup probe that runs forever is a probe that delays the liveness probe forever. The probe should have a bounded timeout.
• A probe that requires authentication is a probe that fails when the authentication service is down. The probe should not require authentication.

The probe’s anti-patterns

• No probe. A Pod without a probe is a Pod that the cluster treats as always ready. The Pod is added to the EndpointSlice immediately; traffic is sent to the Pod before the application is ready. The fix: add a readiness probe.
• Probe that checks the database. A readiness probe that requires the database is a probe that fails during a database failover. The fix: check the application’s state, not the database’s state.
• Probe that requires the cache. A readiness probe that requires the cache is a probe that fails when the cache is cold. The fix: check the application’s state, not the cache’s state.
• Probe with too-low timeout. A probe with timeoutSeconds: 1 fails on a slow application. The fix: increase the timeout, or fix the application.
• Probe with too-low failureThreshold. A probe with failureThreshold: 1 fails on a single transient error. The fix: increase the threshold, or fix the application.

Quiz

Production discipline

• Probes are part of the application’s deployment. A Pod without a probe is a Pod that the cluster cannot reason about. The probe’s design is part of the application’s design.
• Liveness probes should catch unrecoverable failures. A probe that catches transient failures is a probe that causes cascading restarts.
• Readiness probes should check the application’s state. A probe that checks an external dependency is a probe that fails during the dependency’s failure.
• Startup probes should have a bounded timeout. A probe that runs forever is a probe that delays the liveness probe forever.
• Audit probes at every release. A new application version that removes the probe endpoint is a Pod that fails probes. The audit catches the removal before the deployment.
• Watch the probe’s success rate. A falling success rate is a signal that the application is unhealthy. The cluster’s metrics expose the success rate.