Pending pods — the scheduling diagnostic grid

A Pending Pod is one that the scheduler cannot place. The events tell the operator exactly why. The discipline is to read the events, identify the cause, and apply the documented remediation. A Pending Pod that has been Pending for more than a few minutes is a workload that is not running.

The Pending diagnostic grid

A Pending Pod is caused by one of four categories:

1. Resource exhaustion. The cluster has no node with enough CPU or memory.
2. Affinity / taint. The Pod’s nodeSelector, affinity, or anti-affinity rules do not match any node; or the nodes have taints that the Pod’s tolerations do not match.
3. PVC. The Pod’s PVC is Pending, so the Pod cannot be scheduled.
4. Scheduling gate. A SchedulingGates object is blocking the Pod.

flowchart TD
    A[Pending Pod] --> B{Resource?}
    B -->|No| C[Affinity / Taint]
    B -->|Yes| D[Resource exhaustion]
    C -->|No| E{PVC?}
    E -->|Yes| F[PVC Pending]
    E -->|No| G{SchedulingGate?}
    G -->|Yes| H[SchedulingGate blocking]
    G -->|No| I[Unknown]

The grid is the cause tree. The events tell the operator which branch is the answer.

The events

The events are the cluster’s hypothesis. The events are the output of the scheduler’s filter and score.

kubectl describe pod billing-7d8f-abcde -n prod

A real kubectl describe pod for a Pending Pod:

Name:         billing-7d8f-abcde
Namespace:    prod
Priority:     0
Node:         <none>
Labels:       app=billing
              version=1.2.3
Annotations:  <none>
Status:       Pending
IP:
IPs:          <none>
Controlled By:  Deployment/billing
Containers:
  billing:
    Image:      registry.example.com/billing:1.2.3
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  256Mi
    Environment:
      DB_HOST:  db.prod.svc.cluster.local
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xyz (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  kube-api-access-xyz:
    Type:                    Projected (a volume that contains injected data from more than one source)
    TokenExpirationSeconds:  3607
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12m   default-scheduler  0/12 nodes are available: 3 Insufficient memory, 9 Insufficient cpu.

The Events section is the answer. The FailedScheduling event tells the operator:

• 0/12 nodes are available: no node satisfies the Pod’s requirements.
• 3 Insufficient memory: three nodes have insufficient memory.
• 9 Insufficient cpu: nine nodes have insufficient CPU.

The remediation is to either add capacity to the cluster or to reduce the Pod’s requests.

What the events tell the operator

The events are the cluster’s hypothesis. The most common events:

The events are the input to the diagnostic.

The diagnostic command

The systematic diagnostic command:

# Step 1: symptom
kubectl get pod billing-7d8f-abcde -n prod -o wide

# Step 3: inspect
kubectl describe pod billing-7d8f-abcde -n prod

# Step 4: events
kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

# Step 5: check the cluster's resource state
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory

# Step 7: identify the component
# (Per the events, the component is the scheduler or the cluster's resource state)

The diagnostic is the same arc as the 11-step methodology. The events are the answer.

Production discipline

A Pending Pod is the scheduler’s hypothesis. The discipline is to read the events, identify the cause, and apply the remediation. A cluster that has Pending Pods for more than a few minutes is a cluster that has a resource, affinity, or storage problem.

• Identify the cause. The cause is one of: resource, affinity, taint, PVC, SchedulingGate.

Quiz