Force deletion, stuck Pods, and the PodDisruptionBudget connection

A Pod stuck in Terminating for hours is one of the most common production symptoms. This lesson covers the diagnosis (finalizer or kubelet issue), the force-delete discipline, and how PodDisruptionBudgets prevent disruptions from cascading into outages.

Pods stuck in Terminating

kubectl get pod web-7c8
# NAME      STATUS        AGE
# web-7c8   Terminating   30m

The Pod is in Terminating state — the API server has marked it for deletion (deletionTimestamp is set), but it has not yet been removed from etcd. The kubelet is waiting for the finalizers to clear before completing the deletion.

The two common causes:

1. Finalizer blocking: a finalizer in metadata.finalizers is preventing deletion. The controller that owns the finalizer is supposed to do cleanup work and then remove the finalizer, but the controller is broken or slow.
2. Kubelet unreachable: the kubelet on the Pod’s node cannot report the Pod’s termination status. The API server keeps the Pod in Terminating until the kubelet responds (or until pod-eviction-timeout expires).

flowchart TD
    Del[DELETE pod] --> Mark[API marks deletionTimestamp]
    Mark --> Kube[kubelet sees deletion]
    Kube --> Term[terminate containers]
    Term --> Final{finalizers cleared?}
    Final -- yes --> Remove[API removes object]
    Final -- no --> Wait[wait for finalizer controller]
    Wait -.->|blocked| Stuck[stuck in Terminating]

Diagnosing finalizers

kubectl get pod web-7c8 -o yaml | grep -A 10 finalizers

Output:

finalizers:
- kubernetes.io/pv-protection
- example.com/blocking-finalizer

The two finalizers:

• kubernetes.io/pv-protection: blocks deletion while Pods reference a PVC. Common cause: a PVC is still attached. The Pod is Terminating; the PVC is also being deleted but not yet removed. Wait for the PVC to be removed, then this finalizer clears.
• example.com/blocking-finalizer: a custom finalizer added by a controller. The controller should remove the finalizer when its cleanup work is done.

To check which controller owns a custom finalizer:

kubectl get pod web-7c8 -o jsonpath='{.metadata.finalizers}' | jq
# find the controller: search the codebase for the finalizer name

The controller’s deployment status tells you if the controller is running. If the controller is broken, the finalizer will not be cleared.

Diagnosing kubelet issues

kubectl describe pod web-7c8 | grep -A 5 "Node:"

Output:

Node:         node-3/10.0.3.4

The Pod is bound to node-3. Check:

kubectl get node node-3 -o wide
# STATUS   ROLES    AGE   VERSION   INTERNAL-IP   ...
# Ready    <none>   30d   v1.34.1   10.0.3.4      ...

If node-3 is Ready, the kubelet is up. To look at the kubelet itself, SSH to the node: on a kubeadm cluster the kubelet is a systemd service on the host rather than a Pod, so no label selector will find it.

systemctl status kubelet
journalctl -u kubelet -n 100

If the kubelet is up but the Pod is still Terminating, the issue is likely the application not exiting (SIGTERM not handled, preStop hook hung).

If the kubelet is down, the API server waits for pod-eviction-timeout (5 minutes default) before the node controller evicts the Pod. During this window, the Pod stays in Terminating.

Force deletion

kubectl delete pod web-7c8 --grace-period=0 --force

This:

• Removes finalizers from the API server (--force).
• Sends SIGKILL to the container immediately (--grace-period=0).
• Deletes the API object.

The Pod disappears from kubectl get. The application’s shutdown logic does not run. The application’s external resources (DB connections, CSI volumes, cloud LBs) may be orphaned.

PodDisruptionBudgets

A PodDisruptionBudget (PDB) limits the number of Pods of a workload that can be voluntarily unavailable at any time. Voluntary disruptions are operations that cluster administrators initiate: node drains, cluster upgrades, Deployment rollouts. Involuntary disruptions (node failures, pod evictions) are not protected by PDBs.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: team-a-prod
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

This PDB says: at least 2 Pods of app=web must be available at all times during voluntary disruptions. If the Deployment has 3 replicas, at most 1 can be down at any moment.

Two fields:

• minAvailable: the minimum number (or percentage) of Pods that must be available. Cannot be 0.
• maxUnavailable: the maximum number (or percentage) of Pods that can be unavailable. Cannot exceed the replica count.

Choose one or the other:

# Either: minAvailable
spec:
  minAvailable: 2
  # OR

# maxUnavailable
spec:
  maxUnavailable: 1

PDBs and node drains

When you drain a node:

kubectl drain node-3 --ignore-daemonsets

The drain:

1. Evicts each Pod (with graceful termination).
2. Checks the PDB: would removing this Pod violate the budget?
3. If yes: the eviction is blocked; the drain pauses.

The drain waits indefinitely for the PDB to allow eviction. If the PDB blocks, the drain stops and you must either remove the PDB, increase the replica count, or use --disable-eviction (which uses force-delete instead).

PDB and application patterns

For a stateless HTTP service with 5 replicas:

spec:
  minAvailable: 3

At least 3 of 5 Pods must be available during disruptions. The cluster can lose up to 2 Pods (e.g., during a drain of a 2-replica node pool).

For a database with 3 replicas (one primary, two standbys):

spec:
  minAvailable: 2

At least 2 Pods must be available — the primary and one standby. This protects the database from voluntary disruption while still allowing one Pod to be evicted.

For a Job that runs once:

spec:
  minAvailable: 1

The Job’s single Pod must be available. This PDB prevents the Job from being disrupted while it’s running.

Production discipline

• Pair PDBs with replica counts. A PDB that protects more Pods than you have is a deadlock waiting to happen.
• Test drains against your PDBs. A maintenance window should never stall on a PDB; verify the sizing in staging.
• Investigate finalizer-based Terminating Pods before force-deleting. A finalizer is a contract; force-deleting orphans resources.
• Distinguish voluntary from involuntary disruptions. PDBs only protect against voluntary. Node failures and pod evictions are involuntary and bypass PDBs.
• Set pod-eviction-timeout for the cluster’s expectations. A 5-minute default is too long for high-traffic services; tune it.
• Audit finalizers regularly. A stale finalizer is a future Terminating Pod waiting to happen.

Cross-course references

• The Linux course part XVII-Linux-RAID covers RAID rebuild discipline; PDBs are the cluster-level equivalent of protecting against concurrent disk failures.
• The Ansible course part XXXV-Ansible-Scripting covers rolling restart discipline; PDBs are the cluster-level equivalent.
• The Terraform course part XXVIII-Terraform-Disaster-Recovery covers failover patterns; PDBs protect against rolling restarts breaking failover.

Quiz