The Pending Pod — diagsosing scheduling failures

A Pending Pod is a Pod whose spec.nodeName is empty after the scheduler has run. The scheduler tried to filter and score the cluster’s nodes and could not find a feasible node for the Pod. The Pod’s Status.Conditions and Events carry the diagnostic. This lesson walks the diagnostic workflow.

What “Pending” means

A Pod enters Pending the moment it is created. The Pod stays Pending until the scheduler binds it to a node (spec.nodeName is set) or the Pod is deleted. The Pod is not stuck at Pending in a pathological sense; the cluster is waiting for the scheduler to find a home for it.

# Substitute the Pending Pod's name before running:
POD=billing-1

kubectl get pod "$POD" -o wide

NAME        READY   STATUS    RESTARTS   AGE   NODE
billing-1   0/1     Pending   0          5m    <none>

The NODE column is empty. The Pod is unscheduled.

# Substitute the Pending Pod's name before running:
POD=billing-1

kubectl get pod "$POD" -o jsonpath='{.status.conditions}' | jq

[
  {
    "type": "PodScheduled",
    "status": "False",
    "reason": "Unschedulable",
    "message": "0/5 nodes are available: 1 node(s) had taint {dedicated=prod:NoSchedule}, 2 Insufficient memory, 2 node(s) didn't match Pod's node affinity."
  }
]

The PodScheduled=False condition with reason=Unschedulable is the primary signal. The message lists every filter that rejected every node.

The FailedScheduling event

The scheduler fires a FailedScheduling event every time it runs the filter cycle and finds no feasible node. The event is the canonical diagnostic.

# Substitute the Pending Pod's name before running:
POD=billing-1

kubectl describe pod "$POD" | grep -A 10 "Events:"

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  5m (x8 over 5m)   default-scheduler  0/5 nodes are available:
                                                              1 node(s) had taint {dedicated=prod:NoSchedule},
                                                              2 Insufficient memory,
                                                              2 node(s) didn't match Pod's node affinity.

The 0/5 nodes are available line is the cluster’s diagnosis. The number after 0/ is the count of nodes that the scheduler considered (that is, nodes that were Ready and not yet unschedulable). The number after that is the total count; the message lists the rejection reasons by count.

Filter rejection versus scoring avoidance

The event message distinguishes two failure modes:

• Filter rejection: a node did not pass a filter plugin. The node is listed in the message with a reason like Insufficient memory, taint {dedicated=prod:NoSchedule}, or didn't match Pod's node affinity.
• Scoring avoidance: the node was feasible but scored lower than the threshold. Scoring avoidance is not reported as a failure; the scheduler picks the highest-scoring feasible node, and an empty result means the filter rejected everything.

A Pod that is Pending is always a filter rejection. A Pod that is scheduled but lands on a “wrong” node (cordoned, or a node class the operator did not want) is a scoring decision, not a failure.

The top six common rejection reasons

The scheduler’s filter plugins cover a fixed set of reasons. The most common, in frequency order:

1. Insufficient CPU / memory / ephemeral storage

The Pod’s spec.containers[].resources.requests exceeds the node’s available capacity. The scheduler’s NodeResourcesFit plugin (formerly NodeResourcesMostAllocated, NodeResourcesLeastAllocated, NodeResourcesBalancedAllocation) computes the feasibility.

0/5 nodes are available: 2 Insufficient memory, 3 Insufficient cpu.

The fix: scale the cluster, reduce the Pod’s requests, or remove a Pod from the node to free capacity.

2. Taint not tolerated

A node carries a taint the Pod does not tolerate.

0/5 nodes are available: 3 node(s) had taint {dedicated=prod:NoSchedule}.

The fix: add the toleration, or remove the taint, or change the Pod’s deployment so it does not target the node class.

3. Node affinity unsatisfiable

The Pod’s spec.affinity.nodeAffinity requires a label the node does not have.

0/5 nodes are available: 4 node(s) didn't match Pod's node affinity.

The fix: relax the affinity, or label the node, or provision a node that matches.

4. Node unschedulable (cordoned)

The node is cordoned.

0/5 nodes are available: 1 node(s) had taint {node.kubernetes.io/unschedulable:NoSchedule}.

The fix: uncordon the node, or wait for the maintenance window to end.

5. PVC not bound, or bound to a different zone

The Pod’s PVC is not bound, or is bound to a PV in a different zone than the node.

0/5 nodes are available: 2 node(s) didn't match Pod's persistent volume zone.

The fix: bind the PVC first, or relax the Pod’s topologySpreadConstraints, or provision storage in the matching zone.

6. Pod affinity / anti-affinity unsatisfiable

The Pod’s spec.affinity.podAffinity requires a co-located Pod that does not exist, or podAntiAffinity requires the Pod to avoid a node that it would otherwise have to land on.

0/5 nodes are available: 3 node(s) didn't match Pod's anti-affinity rules.

The fix: ensure the co-located Pod is running, or relax the affinity, or reduce the cluster’s topology to satisfy the rule.

The diagnostic workflow

When a Pod is Pending, the workflow is:

flowchart TD
    A[Pod is Pending] --> B[describe pod,<br/>read FailedScheduling]
    B --> C{Filter rejections<br/>listed?}
    C -->|Yes| D[Identify filter: cpu, memory,<br/>taint, affinity, pvc, ...]
    D --> E[Apply fix:<br/>scale, request, toleration,<br/>label, provision]
    E --> F[Pod scheduled]
    C -->|No| G[Check pod-scheduling-readiness]
    G --> H[Check scheduler logs]
    H --> I[Check kube-scheduler metrics]

The steps:

1. Run kubectl describe pod. Read the FailedScheduling event. Identify the filter rejection.
2. Run kubectl get events --sort-by=.lastTimestamp. The cluster’s events, including the FailedScheduling series, are recorded at the API server.
3. Compare the Pod’s spec to the rejection. A taint rejection means the Pod lacks a toleration. An Insufficient memory rejection means the Pod’s request exceeds the node.
4. Apply the fix. Add the toleration, scale the cluster, label the node, fix the PVC binding.
5. Verify the next scheduling cycle. The scheduler retries every few seconds; the Pod should bind within one cycle of the fix.

Inspecting the scheduler’s view

The scheduler’s logs (when run with --v=4) record every filter cycle. The API server exposes the scheduler’s metrics on /metrics; the relevant signals include:

• scheduler_pending_pods — the queue length
• scheduler_schedule_attempts_total — the total attempts
• scheduler_e2e_scheduling_duration_seconds — the end-to-end latency
• scheduler_pod_scheduling_duration_seconds — the per-Pod scheduling latency

A queue that stays full and a high retry count confirm the cluster is failing to schedule.

When the event is missing

A Pod that is Pending but has no FailedScheduling event is not a scheduling failure. The Pod is Pending because:

• The scheduler has not yet run the filter cycle (a Pod that was just created).
• spec.schedulingGates is set; the Pod is gated.
• The Pod’s PreemptionPolicy is set to Never and the scheduler is waiting for capacity.

The fix in these cases is to remove the scheduling gate, wait for the cycle, or address the preemption policy.

Quiz

Production discipline

• A Pending Pod is a diagnostic artefact. The event message lists exactly which filters rejected which nodes. Read it; do not guess.
• Filter rejection is reported; scoring avoidance is not. A Pod that “lands on the wrong node” is a scoring decision, not a failure. The fix is to add a taint or affinity, not to read the FailedScheduling event.
• SchedulingGates is the most common silent cause of a Pending Pod. A Pod with spec.schedulingGates[].name set is held by the scheduler until every gate is removed. This is the mechanism used by kubectl wait and by controllers that want to defer scheduling.
• PreemptionPolicy: Never hides preemption. A Pod that would have been scheduled by preempting a lower-priority Pod is not scheduled if the policy is Never. The Pod stays Pending; the operator must decide whether to raise the priority or remove the policy.
• Audit FailedScheduling events. A cluster with sustained FailedScheduling events is a cluster that is losing capacity. The metrics should be reviewed at every release.