terminationGracePeriodSeconds and SIGKILL — when the kubelet gives up

SIGKILL is the kubelet’s last resort. When the grace period expires and the container is still running, the kubelet sends SIGKILL — uncatchable, immediate, with no chance for the application to clean up. This lesson covers when SIGKILL happens, what its consequences are, and how to size the grace period to avoid it.

When SIGKILL fires

The kubelet sends SIGKILL when:

• The container is still running after terminationGracePeriodSeconds has elapsed.
• The container has not exited after receiving SIGTERM and the preStop hook (if any) has completed.
• A forced deletion is requested via kubectl delete pod <name> --grace-period=0 --force.

sequenceDiagram
    participant Kubelet
    participant App as Application

    Kubelet->>App: SIGTERM
    Note over App: graceful shutdown begins
    Kubelet->>Kubelet: countdown
    alt exits within grace
        App-->>Kubelet: exit 0
    else still running
        Kubelet->>App: SIGKILL (forced)
        App-->>App: process killed immediately
    end

The grace period countdown starts when SIGTERM is sent. The kubelet does not distinguish “still shutting down” from “hung”; the deadline is the deadline.

The consequences of SIGKILL

• Lost in-flight work: any request mid-flight is cut off. Clients see connection reset errors.
• Lost in-memory state: caches, queues, sessions — all gone.
• Database connection leaks: the application’s connection pool is not drained; the database sees connections dropped without proper close.
• File system inconsistency: if the application was mid-write, files may be partial.
• Exit code 137: the container exits with code 137 (128 + 9 = SIGKILL). The kubelet reports this in containerStatuses[*].state.terminated.

The operational signal: a Pod that frequently exits with code 137 is being SIGKILL’d. Either:

• The application doesn’t handle SIGTERM.
• The grace period is too short.
• The application is hung and cannot exit.

Diagnosing SIGKILL

kubectl get pod web-7c8 -o jsonpath='{.status.containerStatuses[*].state.terminated}' | jq

Output:

{
  "exitCode": 137,
  "reason": "Error",
  "finishedAt": "2026-08-16T12:00:30Z"
}

Or via events:

kubectl describe pod web-7c8
# Events:
#   ... reason: Killing   ... message: Stopping container web
#   ... reason: Failed    ... message: Error: container completed with exit code 137

The kubelet logs Killing with reason: Stopping container when it sends SIGTERM, and Failed with exit code 137 when the process is killed.

Sizing the grace period

The grace period must be greater than the application’s shutdown duration. The shutdown duration includes:

• preStop hook duration (counts against the grace period).
• Application drain time (close connections, finish requests).
• State flush time (write to disk, upload to remote store).
• Database disconnect time.
• Final exit time.

For a stateless HTTP service that handles 30s in-flight requests:

spec:
  terminationGracePeriodSeconds: 45

This gives 45s for the entire sequence: preStop (0s) + drain (30s) + exit (small) + headroom (15s).

For a stateful worker that flushes 5 minutes of work:

spec:
  terminationGracePeriodSeconds: 360

The grace period is 6 minutes. Most of that is flush time.

Production discipline:

• Set the grace period based on actual shutdown time. Measure during a test; do not guess.
• Add headroom. GC pauses, slow network, retries all add variability. 30-50% headroom is reasonable.
• The grace period is also replacement time. A long grace period delays Pod replacement. Balance graceful shutdown needs against availability.
• Per-Pod override is possible. StatefulSet’s spec.template.spec.terminationGracePeriodSeconds sets the field on each Pod.

Force deletion with --force

kubectl delete pod web-7c8 --grace-period=0 --force

This command:

• Sends a DELETE with --grace-period=0: the kubelet sends SIGKILL immediately, no SIGTERM.
• --force: removes finalizers from the API server before deletion. Useful for Pods stuck in Terminating due to finalizers.

sequenceDiagram
    participant User
    participant API as API server
    participant Kubelet

    User->>API: DELETE pod (grace=0, force)
    API->>API: remove finalizers
    API->>API: delete object from etcd
    API->>Kubelet: pod removed
    Kubelet->>Kubelet: SIGKILL container immediately

This is the harshest deletion path. Use it only when:

• The Pod is hung and not responding to SIGTERM.
• The Pod has finalizers that are blocking deletion and the finalizer’s controller is broken.
• You have documented the impact (lost work, partial state).

When SIGKILL is the right answer

SIGKILL is correct when:

• The application is hung and cannot exit cleanly. The grace period is the time the application has to exit; if it can’t, the kubelet should not wait forever.
• The Pod needs to be replaced quickly (e.g., during an incident).
• The application’s graceful shutdown is verified to not work (e.g., it doesn’t handle SIGTERM).

SIGKILL is the wrong answer when:

• The grace period is too short. Increase the grace period first.
• The application doesn’t handle SIGTERM. Fix the application.
• The application is shutting down but slowly. The kubelet should give it time.

Production discipline: SIGKILL should be rare. If SIGKILL is happening often, the underlying cause is a bug or misconfiguration that should be fixed.

Cross-course references

• The Linux course part VI-Linux-Processes covers POSIX signals; SIGKILL is the same primitive at the kernel level.
• The Docker course part XXX-Docker-Lifecycle covers container kill; SIGKILL is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service shutdown; SIGKILL is the last-resort equivalent.

Quiz

Production discipline

• Size terminationGracePeriodSeconds for actual shutdown duration. Measure; do not guess.
• Distinguish SIGKILL from OOMKill via exit reasons. Both produce exit code 137; the terminated.reason field tells them apart.
• Use --force --grace-period=0 only as a last resort. It bypasses cleanup and orphans external resources.
• Add a Prometheus alert on exit code 137. Sustained SIGKILL indicates a misconfigured grace period or an application bug.
• Test shutdown duration under load. Trigger Pod deletions during traffic; verify no SIGKILL occurs.