Pod lifecycle, restart policy, and termination

A Pod’s lifecycle is the choreography between the kubelet, the CRI, the application process, and the controller that manages it. This lesson walks through that lifecycle from creation to termination, explains the restart policy options, and covers the discipline around graceful shutdown.

The Pod lifecycle states

A Pod goes through five high-level states (status.phase):

• Pending: accepted by the cluster, but not yet running. The scheduler is choosing a node; the kubelet is pulling images and starting the sandbox.
• Running: bound to a node; at least one container is running, starting, or restarting.
• Succeeded: all containers have terminated successfully (exit code 0) and will not be restarted.
• Failed: at least one container has terminated in failure (non-zero exit code) and will not be restarted (or has not yet been restarted).
• Unknown: the state cannot be obtained, typically because the kubelet cannot communicate with the API server (network partition, node failure).

stateDiagram-v2
    [*] --> Pending
    Pending --> Running: bound to node, containers start
    Running --> Succeeded: all containers exit 0, restartPolicy != Always
    Running --> Failed: container exits non-zero, restartPolicy = Never
    Running --> Running: container restarts (restartPolicy)
    Running --> Unknown: kubelet unreachable
    Unknown --> Running: kubelet recovers
    Pending --> Failed: admission rejection, image pull failure

The transitions are not all possible. A Pending Pod typically goes to Running once bound. A Running Pod goes to Succeeded only if restartPolicy: Never or OnFailure and all containers exit 0. A Running Pod goes to Failed only if a container exits with a non-zero code and restartPolicy: Never is set.

Part IX covers Pod phases in depth; this lesson focuses on restart policy and termination.

restartPolicy

spec:
  restartPolicy: Always  # default for Deployments, StatefulSets, DaemonSets

Three values:

• Always (default for Pods managed by Deployments, StatefulSets, DaemonSets): every container exit triggers a restart, regardless of exit code. The Pod stays in Running state.
• OnFailure (default for Jobs): a container is restarted only if it exits with a non-zero code. Success (exit 0) leaves the Pod in Succeeded.
• Never: containers are never restarted. The Pod terminates when the last container exits. Used for Jobs that should run once to completion.

The restart applies per-container. In a multi-container Pod:

• Always: every container that exits is restarted (each on its own restart timer; the kubelet backs off failed restarts with exponential delay).
• OnFailure: each container is restarted independently on non-zero exit.
• Never: no container is restarted.

The termination sequence

When a Pod is deleted, the kubelet orchestrates a graceful shutdown:

1. API server marks the Pod for deletion. metadata.deletionTimestamp is set; the Pod is excluded from controllers’ selectors (with foreground propagation, the API server holds the request until dependents are gone).
2. Pre-stop hook runs (if defined). The kubelet executes the lifecycle.preStop handler. The hook is a chance for the application to do graceful work (deregister from service discovery, flush in-memory state, finish in-flight requests).
3. SIGTERM sent to containers. Each container’s PID 1 receives SIGTERM. The application should catch it and start shutdown.
4. Grace period countdown. The kubelet waits terminationGracePeriodSeconds (default 30s) for the containers to exit cleanly.
5. SIGKILL sent. If containers are still running after the grace period, the kubelet sends SIGKILL. The Pod is then marked as terminated.

sequenceDiagram
    participant API as API server
    participant Kubelet
    participant App as Application<br/>(PID 1)
    
    API->>Kubelet: DELETE Pod
    Kubelet->>App: run preStop hook (if defined)
    Kubelet->>App: SIGTERM
    Note over App: graceful shutdown
    alt exits within grace period
        App-->>Kubelet: exit 0
        Kubelet-->>API: Pod terminated
    else still running after grace
        Kubelet->>App: SIGKILL
        Kubelet-->>API: Pod terminated (forced)
    end

The full grace period default is 30 seconds. This includes the preStop hook runtime. If your preStop hook needs 10 seconds and the application needs 25 seconds to drain, the total grace period must be at least 35 seconds.

Termination grace period

spec:
  terminationGracePeriodSeconds: 60

The grace period is per-Pod and applies to all containers. Set it based on the slowest container’s shutdown needs:

• Web servers with connection draining: 30-60 seconds.
• Batch workers that flush state: 30-60 seconds.
• In-memory caches that dump on shutdown: 60-120 seconds.
• Stateless services with no cleanup: 5-10 seconds.

Production discipline: set the grace period explicitly. The 30-second default is fine for most stateless workloads; for stateful or slow-shutting-down workloads, raise it.

Pre-stop hooks

spec:
  containers:
  - name: nginx
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 5 && nginx -s quit"]
      preStop:
        httpGet:
          path: /admin/drain
          port: 8080

Three hook types:

• exec: run a command in the container.
• httpGet: send an HTTP request to the container.
• tcpSocket: open a TCP connection to a port.

The hook runs before SIGTERM. Use it for:

• Service deregistration: call the Service registry’s deregister endpoint before SIGTERM, so the Service stops routing traffic to the Pod.
• State flush: dump in-memory state to disk before SIGTERM.
• Sleep: a common pattern (sleep 5) to give the Service time to deregister the Pod before SIGTERM hits the application.

Lifecycle hooks summary

postStart runs after the container’s process starts but before the kubelet considers the container Started. It runs in parallel with the entrypoint — there is no ordering guarantee. Use it for warm-up tasks that the entrypoint does not handle.

preStop runs synchronously before SIGTERM. The kubelet waits for the hook to complete (or timeout) before sending SIGTERM.

The role of the controller

Directly-created Pods (kubectl run, kubectl apply -f pod.yaml) do not get recreated when they terminate. The controller (Deployment, StatefulSet, DaemonSet, Job) is responsible for recreating Pods to maintain the desired state.

flowchart LR
    User[Operator] -->|apply| Manifest[Manifest]
    Manifest --> Deployment[Deployment controller]
    Deployment -->|manages| RS["ReplicaSet"]
    RS -->|creates/manages| Pod1[Pod]
    RS -->|creates/manages| Pod2[Pod]
    Pod1 -.->|terminates| RS
    RS -.->|recreates| Pod3[Pod]

The controller pattern:

1. Operator applies a manifest (Deployment with replicas: 3).
2. Deployment controller creates a ReplicaSet.
3. ReplicaSet creates 3 Pods.
4. One Pod terminates (liveness failure, drain, etc.).
5. ReplicaSet sees one Pod is missing; creates a new one.
6. The cluster converges to 3 Pods again.

Production discipline: never deploy Pods directly for long-running workloads. Use a controller. Direct Pods are for one-off tasks (a Job with restartPolicy: Never) and debugging.

Cross-course references

• The Linux course part VI-Linux-Processes covers POSIX signals and process lifecycle; SIGTERM and SIGKILL are the same primitives the kubelet uses.
• The Docker course part XXX-Docker-Lifecycle covers container lifecycle and shutdown; Pod-level lifecycle is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service shutdown discipline; preStop hooks are the cluster-level equivalent.

Quiz

Production discipline

• Always implement SIGTERM handling in the application. The application should stop accepting new connections, drain in-flight requests, and exit. The orchestrator cannot know when the application is safe to kill.
• Set terminationGracePeriodSeconds based on the slowest container’s shutdown needs. The 30-second default is fine for stateless workloads; raise it for stateful or slow-shutting-down workloads.
• Never deploy long-running Pods directly. Use a controller (Deployment, StatefulSet, DaemonSet) that recreates Pods as needed.
• Configure the readiness probe to fail during shutdown. This signals the Service to stop routing traffic earlier and reduces the number of cut-off requests.
• Test graceful shutdown under load. Trigger a rolling update or node drain with traffic running; verify that no requests are cut off mid-response.