KubernetesCXIX · Pod TroubleshootingPod troubleshooting
Stuck Terminating — the eviction and shutdown diagnostic
What you'll learn
- Apply the 11-step methodology to a Pod stuck in Terminating
- Identify the finalizer, the grace period, and the preStop hook as the cause
- Force-delete the Pod with --force --grace-period=0
- Identify the production failure modes of stuck Terminating
Prerequisites
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
A Pod stuck in Terminating is the kubelet waiting for the container to shut down. The grace period, the preStop hook, and the finalizer are the cause. The force-delete is the remediation. The discipline is to identify the cause before applying the force-delete.
The Terminating lifecycle
A Pod that is being deleted goes through the Terminating
state. The kubelet:
- Sends SIGTERM to the container.
- Waits for the grace period (default 30s).
- Sends SIGKILL if the container is still running.
- Removes the Pod from the API.
flowchart TD
A[Pod deleted] --> B[Send SIGTERM]
B --> C{Container exits?}
C -->|Yes| D[Remove from API]
C -->|No within grace period| E[Send SIGKILL]
E --> D
D --> F[Pod gone]
A Pod that is stuck in Terminating is one where the kubelet cannot complete the lifecycle. The cause is one of:
- The container is not responding to SIGTERM.
- The preStop hook is hanging.
- A finalizer is preventing the API from removing the Pod.
The diagnostic
A real kubectl describe pod for a stuck Terminating Pod:
Name: billing-7d8f-abcde
Namespace: prod
Status: Terminating
Termination Grace Period: 30s
Controlled By: Deployment/billing
Containers:
billing:
Image: registry.example.com/billing:1.2.3
State: Running
Started: Fri, 16 Aug 2026 04:23:01 +0000
Ready: True
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Killing 4m kubelet Stopping container billing
The events tell the operator:
- The kubelet sent SIGTERM 4 minutes ago.
- The container is still running.
- The grace period is 30s, but the Pod is still Terminating.
The diagnostic is the API object’s finalizers:
kubectl get pod billing-7d8f-abcde -n prod -o yaml | grep -A5 finalizers
The output may show:
metadata:
finalizers:
- billing.example.com/cleanup
The finalizer is preventing the API from removing the Pod.
The remediation
The remediation depends on the cause:
- Container not responding to SIGTERM. The application has a bug that ignores SIGTERM. Force-delete the Pod and fix the application.
- preStop hook hanging. The preStop hook is hanging (e.g., waiting for a network connection). Force-delete the Pod and fix the preStop hook.
- Finalizer blocking. The finalizer is preventing the API from removing the Pod. Force-delete the Pod and remove the finalizer.
kubectl delete pod billing-7d8f-abcde -n prod --force --grace-period=0
The --force flag bypasses the kubelet’s lifecycle. The
--grace-period=0 shortens the grace period to 0. The Pod is
removed from the API immediately.
Why Pods get stuck
The most common causes:
- Application bug. The application ignores SIGTERM. The force-delete is the only remediation.
- Long preStop hook. The preStop hook waits for a network connection that does not arrive. The force-delete is the only remediation.
- Finalizer. A controller adds a finalizer to the Pod (e.g., for cleanup). The controller is failing to remove the finalizer. The force-delete bypasses the controller.
- Kubelet is unhealthy. The kubelet cannot send the SIGTERM. The Pod is stuck at the kubelet level. The fix is the kubelet.
Production discipline
A Pod stuck in Terminating is a workload that will not shut down. The discipline is to identify the cause, fix the underlying issue, and only then force-delete. The force-delete is a fix, not a procedure.
- Identify the cause first. The events and the finalizers tell the operator.
- Investigate the controller. A finalizer is owned by a controller; the controller’s health is the diagnostic.
Quiz
Knowledge check · 4 questions
Q1. What is the difference between a finalizer and a preStop hook?
Q2. Force-deleting a Pod is a routine operation that should be the first response to a stuck Terminating Pod.
Q3. An operator runs `kubectl get pod billing-7d8f-abcde -n prod`. The Pod is in `Terminating` state for 8 minutes. The kubelet logs show `Stopping container billing`. The Pod's finalizers include `billing.example.com/cleanup`. What is the diagnostic and remediation?
The Pod is `billing-7d8f-abcde` in namespace `prod`. The workload is a 6-replica Deployment. The Pod has been Terminating for 8 minutes. The kubelet is the same version as the cluster. The Dep
Q4. Name three common causes of a Pod stuck in Terminating and the diagnostic command for each.
Passing score: 75%. Answers are checked in this browser.