Troubleshooting termination — diagnosing stuck and slow shutdowns

When a Pod does not terminate cleanly, the symptom is the same — in-flight requests lost, partial state, or stuck Terminating — but the cause can be in any of several layers. This lesson is the systematic triage framework: where to look, what evidence to gather, and what the right fix is.

The five layers of termination

A Pod’s termination can fail at five layers:

flowchart LR
    API["API server<br/>deletionTimestamp"] --> Pre["preStop hook"]
    Pre --> App["Application SIGTERM"]
    App --> Grace["Grace period countdown"]
    Grace --> Kube["Kubelet cleanup"]
    Kube --> Finalizer["Finalizer clear"]
    Finalizer --> Done["Pod removed"]

Each layer has its own failure modes:

Triage framework

The systematic approach:

1. Identify the symptom: in-flight requests lost? Pod stuck Terminating? Exit code 137?
2. Find the layer: read events, containerStatuses, and node diagnostics.
3. Gather evidence at the layer: logs, cgroup stats, finalizer list.
4. Apply the fix: not force-delete.

flowchart TD
    S[Symptom] --> Q1{Stuck Terminating?}
    Q1 -- yes --> Finalizer{finalizer blocking?}
    Finalizer -- yes --> F[Identify finalizer; clear or wait]
    Finalizer -- no --> Node{Node reachable?}
    Node -- no --> K[kubelet unreachable; wait for eviction]
    Node -- yes --> App{App responding to SIGTERM?}
    Q1 -- no --> Q2{Exit code 137?}
    Q2 -- yes --> G{Grace period too short?}
    G -- yes --> GP[Increase grace period]
    G -- no --> H{App handles SIGTERM?}
    H -- no --> AH[Fix SIGTERM handling]
    H -- yes --> I{App hung?}

Reading the events

The first step is kubectl describe pod <name> and read the events:

Events:
  ... reason: Killing   message: Stopping container web
  ... reason: Failed    message: Error: container completed with exit code 137
  ... reason: Failed    message: Back-off restarting failed container

The events tell you:

• Killing: kubelet is terminating the container.
• Failed with exit code 137: SIGKILL (kubelet or cgroup).
• Back-off restarting: container is in CrashLoopBackOff.

The events are chronological; the latest event is the most recent state.

Reading containerStatuses

kubectl get pod web-7c8 -o jsonpath='{.status.containerStatuses}' | jq

Output:

[
  {
    "name": "nginx",
    "state": {
      "terminated": {
        "exitCode": 137,
        "reason": "Error",
        "finishedAt": "2026-08-16T12:00:30Z"
      }
    },
    "lastState": {
      "terminated": {
        "exitCode": 0,
        "reason": "Completed",
        "finishedAt": "2026-08-16T11:55:00Z"
      }
    },
    "ready": false,
    "restartCount": 3
  }
]

Three pieces of information:

• state: current state (running, waiting, terminated).
• lastState: previous instance’s state. If the current state is terminated with code 137, the previous state tells you what happened before the SIGKILL.
• restartCount: how many times the container has restarted. High counts indicate instability.

Node-level diagnostics

For termination issues, the node is often the source. SSH to the node and check:

# Kubelet logs
journalctl -u kubelet -n 200 --no-pager

# Cgroup state for the container
cat /sys/fs/cgroup/system.slice/kubepods-burstable.slice/.../memory.events

# OOMKill count
grep oom_kill /sys/fs/cgroup/.../memory.events

# Containerd logs (for container-level errors)
journalctl -u containerd -n 100 --no-pager

The cgroup memory.events shows oom_kill events; this tells you if the cgroup killed the container (separate from the kubelet’s SIGKILL).

Common scenarios and fixes

Scenario 1: Pod stuck in Terminating, no events

The Pod has been Terminating for hours with no events. This usually means:

• Finalizer blocking. Check metadata.finalizers.
• Kubelet unreachable. Check the node’s status.

Scenario 2: Exit code 137 with reason Error

The kubelet SIGKILL’d the container at the end of the grace period. This usually means:

• The application did not handle SIGTERM. The fix is application-level.
• The grace period was too short for the application’s shutdown duration. Increase the grace period.

Scenario 3: Exit code 137 with reason OOMKilled

The cgroup killed the container for exceeding its memory limit. This is unrelated to termination; the container was already failing. The fix is to increase the memory limit or fix the application’s memory usage.

Scenario 4: Stuck Terminating with finalizer

The Pod has a finalizer that is not being cleared. Steps:

1. Identify the finalizer: kubectl get pod -o yaml | grep finalizers.
2. Find the controller: search the codebase, or check the controller’s deployment status.
3. If the controller is broken: investigate. Do not force-delete unless the controller cannot be fixed.

Scenario 5: Slow drain on maintenance window

The drain is taking too long. Causes:

• Pods have long grace periods.
• PDB is blocking eviction.
• Application is not draining quickly.

Steps:

1. kubectl get pods -o custom-columns=NAME:.metadata.name,GRACE:.spec.terminationGracePeriodSeconds to see grace periods.
2. kubectl get pdb -A to see PDBs.
3. If PDBs are blocking: temporarily relax them.
4. If grace periods are too long: this is normal; the drain waits.

Production patterns

Capture termination state before debugging:

kubectl get pod web-7c8 -o yaml > /tmp/web-7c8.yaml
kubectl get events --field-selector involvedObject.name=web-7c8 > /tmp/web-7c8-events.txt

Save the Pod’s full spec and the events. These are the postmortem artefacts.

Test termination before debugging in production:

# Trigger a Pod deletion in staging
kubectl delete pod web-7c8 -n staging

# Watch the termination sequence
kubectl get pod web-7c8 -n staging -w

# Time the shutdown
time kubectl delete pod web-7c8 -n staging

The actual shutdown time is the right input for sizing terminationGracePeriodSeconds in production.

Avoid force-delete as the first step:

# WRONG: force-delete without diagnosis
kubectl delete pod web-7c8 --force --grace-period=0

# RIGHT: diagnose, then act
kubectl describe pod web-7c8
kubectl get pod web-7c8 -o jsonpath='{.metadata.finalizers}'
# ... find the cause ...
# THEN decide whether to force-delete

Force-delete orphans resources. Use it only when the alternative is worse.

Cross-course references

• The Linux course part XXII-Linux-NetTroubleshoot covers read-only triage; Pod termination triage is the cluster-level equivalent.
• The Ansible course part XLV-Ansible-Debugging covers systematic debugging; the same approach applies to termination.
• The Observability course part LXXXV-Kubernetes-Observability covers kube-state-metrics; many termination symptoms are exposed as metrics.

Quiz

Production discipline

• Use the systematic triage framework. Identify the layer first; gather evidence at that layer; apply the fix.
• Save the Pod’s full spec and events before debugging. Postmortem artefacts are the difference between a fix and a guess.
• Avoid force-delete without diagnosis. Force-delete orphans resources and bypasses cleanup.
• Test termination in staging. Time the shutdown under load; size the grace period from real data.
• Audit finalizers regularly. A stale finalizer is a future Terminating Pod waiting to happen.