Readiness probes — controlling Service traffic and rolling updates

The readiness probe is the most consequential probe in production. It is the gate that determines whether a Pod receives traffic, whether a rolling update can proceed, and whether the Service considers the Pod alive. This lesson covers how the readiness probe works, its relationship with Endpoints, and the production patterns that get it right.

What the readiness probe controls

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 1

The kubelet runs the readiness probe on the configured schedule (default every 10s). If the probe succeeds, the container is marked ready. If the probe fails, the container is marked not ready.

The result feeds into the Pod’s Ready condition and the Service’s Endpoints list:

flowchart LR
    Probe[Readiness probe] -->|succeeds| Ready["Ready=True"]
    Probe -->|fails| NotReady["Ready=False"]
    Ready --> Endpoints[Endpoints list]
    NotReady -.->|excluded| Endpoints
    Endpoints --> Traffic[Service routes traffic]

A Pod with Ready=False is excluded from the Endpoints list. The Service’s kube-proxy rules do not route traffic to it. This is what makes the readiness probe the gate for traffic.

The probe types

Three standard types plus gRPC:

• httpGet: HTTP GET to a path/port; success = 2xx or 3xx.
• tcpSocket: open a TCP connection; success = connection established.
• exec: run a command in the container; success = exit code 0.
• grpc: gRPC health check; success = SERVING status.

The right probe type depends on the application:

• HTTP services: httpGet to a /ready endpoint.
• TCP services (databases, custom protocols): tcpSocket to the listening port.
• Background workers: exec to check a queue or state file.
• gRPC services: grpc to a health check method.

The probe should be cheap. A heavy probe adds load on every Pod every periodSeconds.

What “ready” should mean

The readiness probe should reflect whether the Pod can serve traffic right now, not whether the application is healthy in a deeper sense. Distinguish:

• Ready to serve: the application has bound on its port, its dependencies are reachable, it has finished any in-process warmup. Traffic should flow.
• Healthy: the application can respond correctly under load, the database is reachable, the cache is hot. Liveness probe territory.

The standard readiness check for a web service:

@app.get("/ready")
async def ready():
    # Quick checks: dependencies reachable, config loaded
    if not db.is_connected():
        return {"ready": False, "reason": "db disconnected"}, 503
    if not config.is_loaded():
        return {"ready": False, "reason": "config missing"}, 503
    return {"ready": True}, 200

The check returns 200 when ready, 503 when not. The kubelet interprets 2xx as success and 5xx as failure.

The rolling update dependency

A Deployment’s rolling update proceeds like this:

sequenceDiagram
    participant D as Deployment
    participant New as New ReplicaSet
    participant Old as Old ReplicaSet
    participant E as Endpoints
    D->>New: scale up to maxSurge
    New->>E: add new Pod when Ready=True
    New->>D: ReplicaSet is at desired replicas
    D->>Old: scale down by maxUnavailable
    Old->>E: remove old Pod from Endpoints

The Deployment controller:

1. Scales up the new ReplicaSet by maxSurge (default 25%).
2. Waits for the new Pods to become Ready.
3. Once new Pods are Ready, scales down the old ReplicaSet by maxUnavailable (default 25%).
4. Repeats until the rollout is complete.

If the readiness probe is strict (e.g., requires external dependencies that are slow to come up), the new Pods stay not-Ready for too long. The rolling update stalls.

If the readiness probe is too lenient (always returns 200), the new Pods are added to Endpoints before they’re actually ready. Traffic flows to a half-started Pod; users see errors.

The right readiness probe is calibrated: it returns Ready when the Pod is genuinely ready to serve traffic, and Not-Ready when it isn’t.

Common readiness probe patterns

HTTP endpoint with dependency checks:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 2

TCP port check (databases, low-level services):

readinessProbe:
  tcpSocket:
    port: 5432
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3

Exec check (custom logic):

readinessProbe:
  exec:
    command: ["sh", "-c", "test -f /tmp/ready"]
  initialDelaySeconds: 5
  periodSeconds: 5

The exec form runs in the container’s filesystem. It is useful when the readiness state is not exposed via HTTP or TCP.

Tuning readiness probes

The probe timing fields:

• periodSeconds: how often the kubelet runs the probe. Default 10. Lower means faster reaction to readiness changes; higher means less probe load.
• timeoutSeconds: per-probe timeout. Default 1. The probe must respond within this; otherwise it’s a failure.
• failureThreshold: consecutive failures before Not-Ready. Default 3. Higher means tolerate brief failures (e.g., GC pauses).
• successThreshold: consecutive successes before Ready. Default 1 (must be 1).

For a fast-failing readiness check (HTTP to /ready):

periodSeconds: 5      # probe every 5s
timeoutSeconds: 1     # each probe times out in 1s
failureThreshold: 2   # 2 failures = Not-Ready
successThreshold: 1   # 1 success = Ready

The window from Ready to Not-Ready (in the worst case) is failureThreshold * periodSeconds = 10s. The window from Not-Ready to Ready is successThreshold * periodSeconds = 5s.

Production patterns

Strict readiness, fast rollout: probe that requires all dependencies; rolling update configured with maxSurge: 100%, maxUnavailable: 0. New Pods come up, become Ready, old Pods are removed one at a time.

Lenient readiness, gradual rollout: probe that requires only “process is alive”; rolling update with maxSurge: 25%, maxUnavailable: 25%. New Pods are added quickly; old Pods are removed gradually.

Readiness during shutdown: implement graceful shutdown in the application. When the application receives SIGTERM, it should:

1. Set the readiness probe to return Not-Ready (e.g., return 503).
2. Wait for the probe to fail and the Pod to be removed from Endpoints.
3. Then exit.

This gives the Endpoints controller time to remove the Pod from the Endpoints list before the application stops responding. The result: in-flight requests complete, new requests go to other Pods.

# Shutdown sequence
import signal
ready = True

def handle_sigterm(*args):
    global ready
    ready = False
    # Wait for kubelet to remove us from Endpoints (a few seconds)
    time.sleep(10)
    # Exit
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

@app.get("/ready")
async def ready_endpoint():
    return ({"ready": ready}, 200 if ready else 503)

This is the canonical graceful shutdown pattern. The readiness probe is the control signal; SIGTERM is the trigger.

Cross-course references

• The Linux course part VI-Linux-Processes covers process states; readiness probes are the cluster-level equivalent of “is this process accepting connections.”
• The Docker course part XXX-Docker-Lifecycle covers Docker health checks; readiness probes are the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service readiness; readiness probes are the cluster-level extension.

Quiz

Production discipline

• Calibrate readiness probes to actual readiness time. Too strict = stalled rollouts; too lenient = traffic to half-started Pods.
• Implement graceful shutdown via the readiness probe. When SIGTERM arrives, set readiness to false, wait for Endpoints to update, then exit.
• Tune periodSeconds and failureThreshold for fast reaction. A probe that runs every 10s with failureThreshold=3 takes up to 30s to react. For latency- sensitive services, use 5s and 2.
• Configure progressDeadlineSeconds on Deployments. Abort rollouts that don’t progress; prevents a stuck rolling update from holding old Pods hostage.
• Treat readiness as the control plane for traffic. It is the only signal the Endpoints controller reads to decide which Pods receive traffic. Make it accurate.