Pending Pods — diagnosing a Pod that will not schedule

A Pending Pod is the scheduler’s signal that it cannot place the Pod on any node. The diagnostic is mechanical: read the Pod’s events, identify which filter eliminated all nodes, fix the constraint or the cluster, and verify the Pod schedules. This lesson walks the systematic approach and the most common production failure modes.

The diagnostic discipline

flowchart TB
    A[Pod Pending] --> B[kubectl describe pod]
    B --> C{Events present?}
    C -->|no| D[Wait, retry]
    C -->|yes| E[FailedScheduling?]
    E -->|no| F["Image pull or<br/>volume issue"]
    E -->|yes| G["Read filter<br/>messages"]
    G --> H[Identify root cause]
    H --> I[Fix]
    I --> J[Verify Pod schedules]

The first step is kubectl describe pod and reading the events. The events tell the operator which filter eliminated all nodes.

Reading a FailedScheduling event

kubectl describe pod web-7c8d9b1f8-abcd -n prod

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m     default-scheduler  0/6 nodes are available:
                                                       1 Node(s) didn't match Pod's node affinity,
                                                       2 Node(s) didn't match Pod's node selector,
                                                       3 Insufficient cpu.

The message has three parts: how many nodes total, how many failed each filter. The fix is in the filter names:

The most common production failures

Failure 1: NodeAffinity with no matching nodes

A Pod requires zone us-east-1c but no nodes are in that zone.

Events:

FailedScheduling: 0/6 nodes are available: 6 node(s) didn't match
  pod's node affinity.

Fix: correct the affinity, add nodes to the zone, or relax the constraint.

# Verify zone distribution
kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone
# NAME       ZONE
# node-01    us-east-1a
# node-02    us-east-1b
# (no us-east-1c)

Failure 2: Resource exhaustion

A Pod requests 32Gi memory but no node has 32Gi allocatable.

Events:

FailedScheduling: 0/6 nodes are available: 6 Insufficient memory.

Fix: reduce the request, add a node with more memory, or scale the cluster.

# Verify allocatable
kubectl describe nodes | grep -A 5 "Allocatable"
# Allocatable:
#   cpu:                7900m
#   ephemeral-storage:  469528482Ki
#   memory:             30185104Ki  # ~29Gi
#   pods:               110

The Pod’s 32Gi request exceeds 29Gi allocatable on every node. Reduce the request to ≤29Gi or add a node with more memory.

Failure 3: Taints without tolerations

A cluster has dedicated=ml:NoSchedule on every node; the Pod has no toleration.

Events:

FailedScheduling: 0/6 nodes are available: 6 node(s) had taints
  that the pod didn't tolerate.

Fix: add the toleration or untaint the nodes.

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ml
    effect: NoSchedule

Failure 4: PVC Pending

The Pod’s PVC is Pending; the Pod is waiting for the PVC.

Events:

Type     Reason            Age   From               Message
----     ------            ----  ----               -------
Warning  FailedScheduling  2m    default-scheduler  0/6 nodes are available: 6 node(s) didn't find available
                                                              persistent volumes to bind.

Fix: investigate the PVC.

kubectl get pvc -n prod
# NAME              STATUS    VOLUME
# data-web-0        Pending   <none>

kubectl describe pvc data-web-0 -n prod
# Events:
#   Warning  ProvisioningFailed  4m  persistentvolume-controller  no storage class matched

The StorageClass may be missing or the provisioner may be broken. Fix the StorageClass; the Pod’s PVC binds; the Pod schedules.

Failure 5: Image pull backoff

The kubelet cannot pull the image. The Pod is in ContainerCreating, not Pending. The scheduler has bound the Pod; the kubelet cannot start it.

Events:

Type     Reason            Age   From               Message
----     ------            ----  ----               -------
Warning  FailedScheduling  ...   default-scheduler  ...
Warning  Failed            2m    kubelet             Failed to pull image "web:v1.0.0": rpc error:
                                                                code = NotFound desc = ...

Fix: verify the image name, registry credentials, and network connectivity.

kubectl get pod web-7c8d9b1f8-abcd -o jsonpath='{.spec.nodeName}'
# node-01

# SSH to node-01 and check
ssh node-01
crictl pull web:v1.0.0
# FATA[0000] pull command failed: ...

Failure 6: PDB and voluntary disruption

A PodDisruptionBudget with maxUnavailable: 0 blocks any voluntary eviction. New Pods scheduled while the PDB is breached are not affected (PDB applies to evictions, not scheduling).

This is not the cause of Pending; PDBs do not block scheduling.

Failure 7: Resource quota

A Namespace’s ResourceQuota may be exhausted. The Pod’s request fits the nodes but exceeds the namespace’s quota.

Events:

Warning  FailedCreate  4m   replicaset-controller  Error creating: pods "web-abc" is forbidden:
                                          exceeded quota: prod-cpu, requested: cpu=2000m, used: cpu=30000m, limited: cpu=32000m

The Pod never gets created; the ReplicaSet reports the error. The fix: increase the quota or reduce the request.

The systematic approach

For each Pending Pod, the operator follows:

flowchart TB
    A[Pod Pending] --> B[describe pod]
    B --> C{FailedScheduling<br/>event?}
    C -->|no| D{kubelet event?}
    C -->|yes| E[Read filter messages]
    D -->|ImagePullBackOff| F[Verify image]
    D -->|Volume mount fail| G[Verify PVC]
    E --> H{Filter?}
    H -->|Affinity / Selector| I["Correct constraint<br/>or add nodes"]
    H -->|Resources| J["Reduce requests<br/>or scale cluster"]
    H -->|Taints| K[Add toleration]
    H -->|PVC| L[Fix PVC]
    F --> M[Verify Pod schedules]
    G --> M
    I --> M
    J --> M
    K --> M
    L --> M

Inspection tools

# All Pending pods in a namespace
kubectl get pods -n prod -o wide | grep Pending

# All FailedScheduling events cluster-wide
kubectl get events -A --field-selector reason=FailedScheduling \
  -o custom-columns=NAME:.involvedObject.name,NS:.involvedObject.namespace,REASON:.reason,MESSAGE:.message

# A Pod's node assignment (set by scheduler)
kubectl get pod web-7c8d9b1f8-abcd -n prod -o jsonpath='{.spec.nodeName}'
# (empty if not bound)

$ kubectl describe pod web-7c8d9b1f8-abcd -n prod | sed -n '/Events/,/Conditions/p'

Events:
Type     Reason            Age    From               Message
----     ------            ----   ----               -------
Warning  FailedScheduling  4m     default-scheduler  0/6 nodes are available: ...

Quiz

Production discipline

• Every Pending Pod has a reason. The events tell the operator what filter failed; the fix is mechanical.
• Validate constraints in CI. A Pod with nodeSelector: gpu: true should be checked against the cluster’s node labels before applying.
• Audit Pending Pods. A dashboard that surfaces Pending Pods and their events catches scheduling failures before they cascade.
• Distinguish scheduler failures from kubelet failures. Scheduler failures are FailedScheduling; kubelet failures are ImagePullBackOff, FailedMount, etc.
• Test the cluster’s capacity. A cluster running at 95% utilisation has 5% headroom; a burst of Pods can saturate it. The discipline is monitoring and headroom.

Pending Pods are the most common Kubernetes support ticket. The diagnostic is mechanical; the fix is in the filter name. Operators who follow the systematic approach have Pending Pods that resolve quickly.