KubernetesCXIX · Pod TroubleshootingPod troubleshooting
Pod field reference — the canonical `kubectl describe pod` excerpts
What you'll learn
- Use the canonical kubectl describe pod excerpts as a diagnosis cheat sheet
- Match the cluster-reported state to the failure mode
- Identify the field that is the source of truth for each failure mode
- Apply the diagnosis pattern to unfamiliar Pod failures
Prerequisites
- Pending pods — the scheduling diagnostic grid
- CrashLoopBackOff and ImagePullBackOff — the image and startup failures
- CreateContainerConfigError and OOMKilled — the misconfiguration and memory failures
- Probe failures — readiness, liveness, and startup
- Stuck Terminating — the eviction and shutdown diagnostic
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
The canonical kubectl describe pod excerpts are the diagnosis
cheat sheet. Each failure mode has a State, a Reason, and an
Events pattern. The discipline is to match the pattern and
apply the remediation. An operator who carries the cheat
sheet in their head is the operator who recovers in 5
minutes.
The cheat sheet
The cheat sheet is a table that maps the State.Reason field
to the failure mode:
| State.Reason | Failure mode | Look at |
|---|---|---|
Pending | Scheduler cannot place | Events for FailedScheduling |
Waiting / ImagePullBackOff | Image pull failed | Events for Failed with pull access denied |
Waiting / ErrImagePull | Image pull failed | Events for Failed |
Waiting / CreateContainerConfigError | ConfigMap/Secret missing | Events for not found |
Waiting / CrashLoopBackOff | Container crashing | Previous logs |
Running / Ready: False | Probe failure | Events for Unhealthy |
Terminating (long) | Finalizer or preStop | Pod’s finalizers, kubelet logs |
The cheat sheet is the operator’s first reference.
The Pending Pod excerpt
Status: Pending
Node: <none>
Conditions:
Type Status
PodScheduled False
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 12m default-scheduler 0/12 nodes are available: 3 Insufficient memory, 9 Insufficient cpu.
Diagnosis: Resource exhaustion. Remediation: Add capacity or reduce requests.
The ImagePullBackOff excerpt
Containers:
billing:
State: Waiting
Reason: ImagePullBackOff
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 8m kubelet Failed to pull image "registry.example.com/billing:1.2.3": rpc error: code = Unknown desc = pull access denied
Normal BackOff 5m kubelet Back-off pulling image
Diagnosis: Image pull credentials or wrong image name. Remediation: Fix imagePullSecrets or image name.
The CreateContainerConfigError excerpt
Containers:
billing:
State: Waiting
Reason: CreateContainerConfigError
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 4m kubelet Error: configmap "billing-config" not found
Diagnosis: Missing ConfigMap or Secret. Remediation: Create the resource.
The CrashLoopBackOff excerpt
Containers:
billing:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Ready: False
Restart Count: 12
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 12m kubelet Back-off restarting failed container
Diagnosis: Container crashing. Remediation: Read previous logs.
The OOMKilled excerpt
Containers:
billing:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Ready: False
Restart Count: 6
Limits:
memory: 512Mi
Diagnosis: Memory exhausted. Remediation: Increase memory limit or fix the workload.
The readiness probe failure excerpt
Containers:
billing:
State: Running
Started: Fri, 16 Aug 2026 04:23:01 +0000
Ready: False
Readiness: http-get http://:8080/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
Conditions:
Type Status
Ready False
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Diagnosis: Readiness probe failing. Remediation: Fix the application or the probe’s configuration.
The liveness probe failure excerpt
Containers:
billing:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Liveness: http-get http://:8080/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 1m kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning Killing 1m kubelet Killing container billing
Warning BackOff 30s kubelet Back-off restarting failed container
Diagnosis: Liveness probe failing. Remediation: Fix
the 500 from /healthz, or raise initialDelaySeconds and
failureThreshold so a slow recovery does not trigger the
restart loop.
The Terminating excerpt
Status: Terminating
Termination Grace Period: 30s
Containers:
billing:
State: Running
Started: Fri, 16 Aug 2026 04:23:01 +0000
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Killing 4m kubelet Stopping container billing
Diagnosis: Pod is stuck in Terminating. Remediation: Investigate the finalizer, preStop hook, application, or kubelet.
The diagnostic command
The canonical diagnostic command sequence:
# 1. Get the Pod's current state
kubectl get pod billing-7d8f-abcde -n prod -o wide
# 2. Describe the Pod (the cheat sheet)
kubectl describe pod billing-7d8f-abcde -n prod
# 3. Read the events
kubectl get events -n prod --sort-by=.lastTimestamp \
--field-selector involvedObject.name=billing-7d8f-abcde
# 4. Read the previous logs (for CrashLoopBackOff)
kubectl logs -n prod billing-7d8f-abcde -c billing --previous
# 5. Read the current logs
kubectl logs -n prod billing-7d8f-abcde -c billing --tail=200
# 6. Read the finalizers (for stuck Terminating)
kubectl get pod billing-7d8f-abcde -n prod -o yaml | grep -A5 finalizers
The command sequence is the 11-step methodology in muscle memory.
Production discipline
The cheat sheet is the operator’s first reference. The discipline is to match the State.Reason to the failure mode, read the events, apply the remediation. The cheat sheet is the muscle memory of the 11-step methodology.
- Read the State.Reason field first. The field is the canonical answer.
- Run the diagnostic command sequence. The sequence is the 11-step methodology in muscle memory.
Quiz
Knowledge check · 4 questions
Q1. Which field in `kubectl describe pod` is the canonical answer to the failure mode?
Q2. An exit code of 137 is the canonical signal of an OOMKilled container.
Q3. An operator runs `kubectl describe pod billing-7d8f-abcde -n prod`. The State is `Waiting`, the Reason is `CrashLoopBackOff`, the Last State Reason is `OOMKilled`, the Exit Code is 137. What is the failure mode and the remediation?
The Pod is `billing-7d8f-abcde` in namespace `prod`. The workload is a 6-replica Deployment. The Pod has restarted 12 times in 5 minutes. The memory limit is 512Mi.
Q4. Name three fields in `kubectl describe pod` that are the canonical answer to the failure mode and explain what each one tells the operator.
Passing score: 75%. Answers are checked in this browser.