Readiness, preStop, and Service traffic — gating rollout safety

Every Kubernetes rollout strategy depends on the same chain of events: a Pod is created, it becomes Ready, the Endpoints controller includes it in the Service’s Endpoints, kube-proxy programs the dataplane to route traffic to it. When the rollout terminates an old Pod, the chain runs in reverse: preStop runs, SIGTERM is sent, the Pod is removed from Endpoints, kube-proxy drains it, and after the grace period SIGKILL arrives. This lesson is about every link in that chain and the production failures that come from breaking any one of them.

The full chain, end to end

sequenceDiagram
    participant K as kubelet
    participant EP as Endpoints Controller
    participant KP as kube-proxy
    participant UL as Upstream LB
    participant P as Pod
    K->>P: Create container
    P->>K: readinessProbe passes
    K->>EP: Pod Ready=True
    EP->>KP: Endpoints updated
    KP->>UL: dataplane programmed
    UL->>P: traffic flows
    Note over P: rollout begins
    UL->>P: preStop runs
    P->>K: shutdown
    K->>EP: Pod Ready=False
    EP->>KP: Endpoints updated
    KP->>UL: dataplane drains
    UL-->>P: no new traffic
    K->>P: SIGTERM after grace period
    K->>P: SIGKILL

If any link in the chain is broken — readiness is wrong, the Endpoints controller is not running, kube-proxy is out of sync, the upstream load balancer has a long drain interval — the rollout is unsafe.

The readiness probe is the gate

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

The probe runs against the Pod’s IP at the configured path. Until it passes, the Pod’s Ready condition is False. The Endpoints controller excludes not-Ready Pods from the Service’s Endpoints. Traffic only flows to Ready Pods.

flowchart TB
    A[New Pod starts] --> B["Container Running<br/>Ready=False"]
    B --> C{readinessProbe}
    C -->|pass| D["Ready=True<br/>Endpoints include IP"]
    C -->|fail 3x| E["Ready=False<br/>stays out of Endpoints"]
    D --> F[Traffic flows]
    E --> G[Deployment rollout stalls]

preStop — coordinating with load balancers

When a Pod is terminated, the Endpoints controller marks it not-Ready, and after a small delay removes its IP from the Endpoints object. kube-proxy and the CNI dataplane pick up the change. But external load balancers and Ingress controllers have their own drain intervals, which can be slower than the kubelet’s default termination grace period.

The standard pattern:

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: web
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 15"]

The preStop runs before SIGTERM. A 15-second sleep gives the upstream load balancer time to drain. Combined with a terminationGracePeriodSeconds: 60, the Pod has 15s of preStop

• 45s for in-flight requests to drain after SIGTERM.

flowchart LR
    A[Pod terminating] --> B["preStop<br/>e.g. sleep 15"]
    B --> C["Endpoints controller<br/>removes IP"]
    C --> D["kube-proxy<br/>dataplane drains"]
    D --> E[SIGTERM sent]
    E --> F{grace period<br/>elapsed?}
    F -->|no| E
    F -->|yes| G[SIGKILL]

Without preStop, the upstream LB may still send traffic to the Pod after SIGTERM but before the Endpoints change propagates — producing 502/504 errors during every rollout.

Readiness gates for external systems

A readiness gate is a Pod condition reported by an external controller. The Pod is not Ready until the gate is True.

spec:
  readinessGates:
  - conditionType: pods.web.example.com/ready

An external controller (e.g., a service mesh sidecar) sets the condition when the Pod is fully wired into the mesh. The Service does not route to the Pod until the mesh reports it ready.

This is how production rollouts safely handle external dependencies: cache priming, downstream service registration, external API warm-up. The Deployment waits for the gate; the gate is owned by the operator’s controller.

The chain breaks in stages

When a rollout is unsafe, the failure usually traces to one link:

Each one requires its own evidence. A blind kubectl rollout undo is the operator’s last resort, not the first response.

Inspecting the rollout state

kubectl get pods -l app=web -n prod -o wide
# NAME                  READY   STATUS    AGE
# web-7c8d9b1f8-abcd    1/1     Running   5m
# web-7c8d9b1f8-efgh    1/1     Running   5m
# web-6b3d5a7e9-ijkl    1/1     Running   30s

kubectl describe pod web-6b3d5a7e9-ijkl -n prod | grep -A 2 Conditions
# Conditions:
#   Type              Status
#   Ready             True
#   ContainersReady   True
#   PodReadyToStartContainers  True

$ kubectl get endpoints web -n prod -o yaml | head -40

apiVersion: v1
kind: Endpoints
metadata:
name: web
namespace: prod
subsets:
- addresses:
- ip: 10.244.1.12
  nodeName: worker-01
- ip: 10.244.2.34
  nodeName: worker-02
- ip: 10.244.1.55
  nodeName: worker-01
ports:
- port: 8080

If a new Pod’s IP is not in the Endpoints list, the rollout chain has broken before the Endpoints controller step.

Quiz

Production discipline

• Readiness must be application-aware. A real probe verifies dependencies and warms state, not just port reachability.
• preStop is mandatory for stateful traffic drains. A 10-15 second sleep costs nothing; an unsynced LB costs every user 1-2 errors.
• terminationGracePeriodSeconds must exceed the longest expected in-flight request. A 60-second value is a safe default for HTTP; tune upward for long-lived connections.
• Rollout metrics are first-class. The Deployment controller’s progressDeadlineSeconds should match the alerting threshold; a stuck rollout is a page.
• Test the chain in staging. Synthesise a rollout that scales replicas: 0 mid-flight. If readiness and preStop are wrong, you will see it in the staging logs, not in production.

The strategy you choose — RollingUpdate, Recreate, blue/green, canary — is the visible half. The invisible half is the chain that drains and admits traffic. Operators who understand the chain own the rollout.