low riskservice affecting~25 min

Runbook: Investigate a Pending Pod

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the Pod and its events: kubectl get pod <name> -n <ns> and kubectl describe pod <name> -n <ns> | sed -n "/Events:/,$p"
· Capture the scheduler view of the cluster: kubectl get nodes -o wide and kubectl describe nodes | grep -E "Allocatable|Allocated resources"
· Confirm the Pod has been Pending long enough to be a real failure (>2 minutes is a useful threshold; >10 minutes is almost always a real failure)
· Confirm the namespace has no ResourceQuota that the Pod violates: kubectl describe resourcequota -n <ns>
· Capture the Pod spec the scheduler is evaluating: kubectl get pod <name> -n <ns> -o yaml | tee /tmp/pod.yaml
· Capture the controller that owns the Pod (Deployment, StatefulSet, Job, etc.) and any related HPA target

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the Pod events end-to-end: kubectl describe pod <name> -n <ns> | sed -n "/Events:/,$p" and identify the first Warning
2Classify the warning into one of: insufficient CPU/memory, no node matches affinity/selector, no node tolerates the taints, PVC unbound, scheduler error, runtime class missing
3For insufficient resources: read kubectl describe nodes | grep -A5 "Allocated resources" and compare to spec.containers[*].resources.requests
4For affinity or nodeSelector mismatch: kubectl get nodes --show-labels and compare to spec.affinity and spec.nodeSelector
5For taint mismatch: kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints and compare to spec.tolerations
6For PVC Pending: kubectl get pvc -n <ns> and kubectl describe pvc <name> -n <ns> (see also the kubernetes-rb-troubleshoot-pvc runbook)
7For scheduler error: read kubectl -n kube-system logs kube-scheduler-<node> and look for the binding attempt for this Pod UID
8For runtime class missing: kubectl get runtimeclass and confirm the Pod references an existing one
9Apply the smallest fix that resolves the cause: add a node, raise capacity, fix the affinity/selector, bind the PVC, add a toleration
10Re-check the Pod events; confirm the Pod is Scheduled (has a nodeName set)
11Validate it becomes Ready: kubectl wait --for=condition=Ready pod/<name> -n <ns> --timeout=120s

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get pod <name> -n <ns> reports Running and Ready 1/1 (or the appropriate count)
✓kubectl describe pod <name> -n <ns> shows Conditions: PodScheduled=True and no Warning events for the last 5 minutes
✓kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}' is non-empty
✓The owning controller reports the Pod as part of its set: kubectl get deploy,rs,sts -n <ns>
✓The Pod passes its readiness probe
✓kubectl describe node <node> does not show the new Pod contributing to pressure conditions

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix involved adding a toleration that should not have been added, remove it: kubectl edit pod/<name> is not allowed on a running Pod; revert the controller spec in Git and kubectl apply
↶If the fix involved raising a ResourceQuota, revert the quota change after the underlying capacity issue is fixed
↶If the fix was a one-off kubectl taint nodes --overwrite to remove a taint, restore the taint after the Pod schedules; the taint exists for a reason
↶If the Pod never schedules after multiple fixes, delete it: kubectl delete pod <name> -n <ns> --force --grace-period=0 and let the controller recreate it with a fresh UID
↶If the investigation is prolonged and a new Pod is needed, delete and recreate: kubectl delete pod/<name> -n <ns>; kubectl apply -f pod.yaml

6 · Escalation

When the runbook isn't enough, contact:

· No single node matches affinity and capacity is genuinely exhausted: escalate to capacity planning — do not relax affinity or add a toleration to 'make it work'
· The Pod is pending because of a scheduler bug or repeated bind errors: capture kubectl -n kube-system logs kube-scheduler-<node> and escalate to platform
· A PVC is Pending because the StorageClass has no provisioner or is misconfigured: see kubernetes-rb-troubleshoot-pvc and escalate to storage ownership
· Multiple namespaces have Pending Pods and the same root cause: a node or zone outage; escalate to platform and follow the kubernetes-rb-troubleshoot-node-notready flow if nodes are involved
· The Pod is repeatedly preempted: see kubernetes-cx-03-preemption lessons and consider the priority/preemption interaction before further changes

A Pending Pod has not been scheduled. The scheduler rejected every node, and the reason is in the Pod’s events. The runbook is structured around reading that reason without guessing.

1. Read the events

Read-only / SafeRead the events

kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30
kubectl get pod <name> -n <ns> -o jsonpath='{.metadata.uid}'

The event message is specific. Examples and what they mean:

Event message	Class	Real cause
`0/N nodes are available: N Insufficient cpu, M Insufficient memory.`	Resource	Requests exceed capacity on every node
`0/N nodes are available: N node(s) didn't match Pod's node affinity/selector.`	Affinity	Labels or expressions do not match any node
`0/N nodes are available: N node(s) had untolerated taint(s).`	Taint	Taints exist that the Pod does not tolerate
`PersistentVolumeClaim is not yet bound`	PVC	PVC is Pending or bound to a non-existent PV
`runtime class 'foo' not found`	RuntimeClass	Pod references a runtime class that does not exist
`skipping node ... because it has the X failure condition`	Node condition	Node is NotReady or under pressure

2. Resource exhaustion

Read-only / SafeResource exhaustion


kubectl describe pod <name> -n <ns> | grep -E 'Requests|Limits' -A5
kubectl describe nodes | grep -E 'Allocatable|Allocated resources' | head -40
kubectl top nodes

If requests are larger than allocatable on every node, three options:

Add a node (escalate to capacity ownership)
Reduce the request (must match actual usage; do not under-request for memory)
Increase node capacity (escalate; this is a node-pool change)

3. Affinity or selector mismatch

Read-only / SafeAffinity or selector mismatch


kubectl get nodes --show-labels | head
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeSelector}' | jq
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.affinity}' | jq

A mismatch usually points at one of:

A label was renamed on the node pool (no current node has the old label)
A typo in the manifest (the selector references a label key that does not exist)
A new node pool was added without the expected labels

Fix the manifest or the node labels; do not patch the Pod.

4. Taint mismatch

Read-only / SafeTaint mismatch


kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.taints}{"\n"}{end}'
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.tolerations}' | jq

If the Pod has no toleration matching the taint, options:

Add a toleration in the manifest (a code change, not a runtime fix)
Remove the taint if it is not actually needed (kubectl taint nodes <node> <key>-)
Schedule onto a different node that does not have the taint

A taint that exists for a reason (e.g. dedicated=db:NoSchedule) should not be removed to admit a generic workload.

5. PVC Pending

Read-only / SafePVC Pending


kubectl describe pvc <name> -n <ns>
kubectl get storageclass

See the kubernetes-rb-troubleshoot-pvc runbook for the full flow. PVC Pending is treated separately because it is the only failure mode here that may require a provisioner or storage back-end change.

6. Scheduler error

Read-only / SafeScheduler error

POD_UID=$(kubectl get pod <name> -n <ns> -o jsonpath='{.metadata.uid}')
kubectl -n kube-system logs kube-scheduler-<control-plane-node> | grep "$POD_UID" | tail -50

A scheduler error (predicates failed, plugin X rejected) usually correlates with a recent cluster change (admission webhook, scheduler configuration, CRD installation). Capture the logs and escalate to platform ownership.

7. Apply the fix

The fix is the smallest change that makes the Pod schedule. Most of the time this is:

A manifest correction in Git
A new node
A PVC bind
A toleration in the manifest

Read-only / SafeApply the fix


kubectl edit deploy/<name> -n <ns>     # DO NOT do this; fix in Git
git revert <bad-commit>                # DO this
git push
kubectl rollout restart deploy/<name> -n <ns
kubectl wait --for=condition=Ready pod -l app=<name> -n <ns> --timeout=5m

Common pitfalls

Symptom	Cause	Action
Pending Pod evicted by a node, returns to Pending	Node went NotReady, scheduler relocated, then the same issue returns	Fix the node (see `kubernetes-rb-troubleshoot-node-notready`)
Multiple Pods Pending with same first-warning class	Cluster-wide resource or quota issue, not per-Pod	Aggregate by cause; escalate to capacity ownership
Pod schedules then immediately evicted	Preemption by a higher-priority Pod	Check `priorityClassName` and PDB interaction
Pod Pending in `kube-system` namespace	System component needs more resources	Increase the node pool or reduce system DaemonSet footprint

A Pending Pod is a statement about a constraint, not a bug. The events say which constraint; the fix is whichever smallest change satisfies that constraint without weakening the system elsewhere.