Graceful termination — SIGTERM and the grace period

Graceful termination is what separates a well-run cluster from one with dropped connections and lost work. This lesson walks through the kubelet’s termination sequence, explains why SIGTERM matters, and shows the discipline around sizing the grace period.

The termination sequence

When a Pod is deleted (by kubectl, by a controller, by a drain), the kubelet orchestrates this sequence:

sequenceDiagram
    participant API as API server
    participant Kubelet
    participant App as Application<br/>(PID 1)
    participant CRI as Container runtime
    
    API->>Kubelet: watch sees deletionTimestamp
    Kubelet->>App: run preStop hook (if defined)
    Kubelet->>App: SIGTERM
    Note over App: graceful shutdown begins
    par grace period
        App-->>App: drain connections, finish work
    and
        Kubelet->>Kubelet: countdown terminationGracePeriodSeconds
    end
    alt exits within grace
        App-->>Kubelet: exit 0
        Kubelet->>CRI: stop container
        Kubelet-->>API: Pod terminated
    else still running
        Kubelet->>App: SIGKILL
        Kubelet->>CRI: stop container
        Kubelet-->>API: Pod terminated (forced)
    end

The steps:

1. API server marks the Pod for deletion. The metadata.deletionTimestamp is set. The Pod enters a “being deleted” state — most controllers exclude it from selectors, but the object still exists in etcd until the finalizers are cleared.
2. kubelet sees the deletionTimestamp. The kubelet on the Pod’s node watches the API server for changes to its bound Pods. It sees the deletion timestamp and starts the termination process.
3. preStop hook runs (if defined). The kubelet executes the lifecycle.preStop handler. This is the application’s chance to do work before SIGTERM.
4. SIGTERM sent to each container’s PID 1. The kubelet uses the CRI to send SIGTERM. The application should catch the signal and start graceful shutdown.
5. Grace period countdown. The kubelet waits terminationGracePeriodSeconds (default 30s) for the containers to exit cleanly. The grace period INCLUDES the preStop hook time.
6. SIGKILL if still running. If a container is still running after the grace period, the kubelet sends SIGKILL (signal 9, uncatchable).
7. Container runtime cleans up. The CRI removes the container’s processes, network namespace, and storage.
8. API server removes the Pod. With no finalizers, the Pod is deleted from etcd.

Why SIGTERM matters

SIGTERM (signal 15) is the polite termination signal. Applications should:

• Stop accepting new connections.
• Drain in-flight requests.
• Flush in-memory state (caches, queues).
• Close database connections.
• Exit with code 0.

SIGKILL (signal 9) is the immediate termination signal. It cannot be caught or handled. The process is killed mid-execution; buffers are not flushed; in-flight requests are cut off; database connections are left open.

A Pod that doesn’t handle SIGTERM will be SIGKILL’d at the end of the grace period. The result: dropped connections, lost work, and the cluster sees exit code 137 (128 + 9).

Sizing the grace period

The grace period should be preStop duration + application shutdown duration + headroom. Examples:

Production discipline:

• Set terminationGracePeriodSeconds explicitly. The 30s default is a guess; tune it for your application.
• The grace period is a Pod-level field. All containers share the same deadline.
• A long grace period delays Pod replacement. If a Pod is unhealthy and must be replaced, the grace period adds to the replacement time. Balance graceful shutdown needs against availability.

spec:
  terminationGracePeriodSeconds: 60

Implementing SIGTERM handling

The application must catch SIGTERM and exit gracefully. In Python:

import signal
import sys

shutdown_requested = False

def handle_sigterm(signum, frame):
    global shutdown_requested
    shutdown_requested = True
    # Trigger graceful shutdown logic
    server.shutdown()

signal.signal(signal.SIGTERM, handle_sigterm)

while not shutdown_requested:
    # Main loop
    process_one_request()

In Go:

ctx, cancel := context.WithCancel(context.Background())

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM)

go func() {
    <-sigCh
    cancel()  // triggers graceful shutdown
}()

server.Run(ctx)

In shell scripts (entrypoint):

#!/bin/bash
trap "exit 0" SIGTERM
while true; do
    process_one_request
done

The trap directive in bash catches SIGTERM and runs the specified handler.

The readiness probe and SIGTERM

A common pattern is to coordinate the readiness probe with SIGTERM:

ready = True

def handle_sigterm(signum, frame):
    global ready
    ready = False
    # Sleep to allow Endpoints controller to update
    time.sleep(10)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

@app.get("/ready")
async def ready_endpoint():
    return ({"ready": ready}, 200 if ready else 503)

On SIGTERM:

1. The application sets ready = False.
2. The kubelet’s readiness probe runs (next period); it fails.
3. The Endpoints controller sees Ready=False; updates the Endpoints object.
4. kube-proxy on each node updates iptables/IPVS rules.
5. After ~10s, the application exits.

This pattern requires the application to keep running during the “drain” period. The grace period must accommodate the drain + readiness probe cycle + exit.

Production patterns

Slow-shutting application with dependency cleanup:

def handle_sigterm(signum, frame):
    # Stop accepting new work
    server.stop_accepting()
    # Drain in-flight requests (up to 30s)
    deadline = time.time() + 30
    while server.has_in_flight() and time.time() < deadline:
        time.sleep(0.1)
    # Flush in-memory state
    cache.flush_to_disk()
    # Close database connections
    db.close()
    # Exit
    sys.exit(0)

Application that doesn’t handle SIGTERM (legacy):

spec:
  terminationGracePeriodSeconds: 5
  containers:
  - name: legacy-app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "kill -TERM 1 && sleep 5"]

The preStop hook sends SIGTERM and waits 5s. The application must handle SIGTERM for this to work; otherwise the kubelet SIGKILLs it at the end of the grace period.

Application with long shutdown (database, batch worker):

spec:
  terminationGracePeriodSeconds: 600  # 10 minutes
  containers:
  - name: batch-worker
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "kill -TERM 1"]

The worker has 10 minutes to finish its current job and exit.

Cross-course references

• The Linux course part VI-Linux-Processes covers POSIX signal handling; SIGTERM and SIGKILL are the same primitives the kubelet uses.
• The Docker course part XXX-Docker-Lifecycle covers container shutdown; Pod-level graceful termination is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service shutdown discipline; the patterns are the same.

Quiz

Production discipline

• Always implement SIGTERM handling in the application. PID 1 does not handle SIGTERM by default in containers.
• Set terminationGracePeriodSeconds explicitly. The 30s default is a guess; tune it.
• The grace period includes preStop duration. Plan preStop + drain time together.
• Coordinate the readiness probe with shutdown. Set ready=False early; let the Endpoints controller update before exit.
• Test graceful shutdown under load. Trigger Pod deletions during traffic; verify zero connection resets.