Skip to main content
RunBook Academy

KubernetesCXIX · Pod TroubleshootingPod troubleshooting

Pod field reference — the canonical `kubectl describe pod` excerpts

Advanced⏱ ~16 minkubectl

What you'll learn

  • Use the canonical kubectl describe pod excerpts as a diagnosis cheat sheet
  • Match the cluster-reported state to the failure mode
  • Identify the field that is the source of truth for each failure mode
  • Apply the diagnosis pattern to unfamiliar Pod failures

Prerequisites

Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16

Not yet marked complete on this device.

The canonical kubectl describe pod excerpts are the diagnosis cheat sheet. Each failure mode has a State, a Reason, and an Events pattern. The discipline is to match the pattern and apply the remediation. An operator who carries the cheat sheet in their head is the operator who recovers in 5 minutes.

The cheat sheet

The cheat sheet is a table that maps the State.Reason field to the failure mode:

State.ReasonFailure modeLook at
PendingScheduler cannot placeEvents for FailedScheduling
Waiting / ImagePullBackOffImage pull failedEvents for Failed with pull access denied
Waiting / ErrImagePullImage pull failedEvents for Failed
Waiting / CreateContainerConfigErrorConfigMap/Secret missingEvents for not found
Waiting / CrashLoopBackOffContainer crashingPrevious logs
Running / Ready: FalseProbe failureEvents for Unhealthy
Terminating (long)Finalizer or preStopPod’s finalizers, kubelet logs

The cheat sheet is the operator’s first reference.

The Pending Pod excerpt

Status:       Pending
Node:         <none>
Conditions:
  Type           Status
  PodScheduled   False
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12m   default-scheduler  0/12 nodes are available: 3 Insufficient memory, 9 Insufficient cpu.

Diagnosis: Resource exhaustion. Remediation: Add capacity or reduce requests.

The ImagePullBackOff excerpt

Containers:
  billing:
    State:    Waiting
      Reason: ImagePullBackOff
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Failed          8m    kubelet            Failed to pull image "registry.example.com/billing:1.2.3": rpc error: code = Unknown desc = pull access denied
  Normal   BackOff         5m    kubelet            Back-off pulling image

Diagnosis: Image pull credentials or wrong image name. Remediation: Fix imagePullSecrets or image name.

The CreateContainerConfigError excerpt

Containers:
  billing:
    State:    Waiting
      Reason: CreateContainerConfigError
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Failed          4m    kubelet            Error: configmap "billing-config" not found

Diagnosis: Missing ConfigMap or Secret. Remediation: Create the resource.

The CrashLoopBackOff excerpt

Containers:
  billing:
    State:    Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason:   Error
      Exit Code:  1
    Ready:      False
    Restart Count:  12
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  BackOff         12m   kubelet            Back-off restarting failed container

Diagnosis: Container crashing. Remediation: Read previous logs.

The OOMKilled excerpt

Containers:
  billing:
    State:    Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason:   OOMKilled
      Exit Code:  137
    Ready:      False
    Restart Count:  6
    Limits:
      memory:  512Mi

Diagnosis: Memory exhausted. Remediation: Increase memory limit or fix the workload.

The readiness probe failure excerpt

Containers:
  billing:
    State:    Running
      Started:  Fri, 16 Aug 2026 04:23:01 +0000
    Ready:      False
    Readiness:  http-get http://:8080/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
Conditions:
  Type           Status
  Ready          False
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Unhealthy       2m    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

Diagnosis: Readiness probe failing. Remediation: Fix the application or the probe’s configuration.

The liveness probe failure excerpt

Containers:
  billing:
    State:    Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason:   Completed
      Exit Code:  0
    Liveness:  http-get http://:8080/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  Unhealthy       1m    kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Killing         1m    kubelet            Killing container billing
  Warning  BackOff         30s   kubelet            Back-off restarting failed container

Diagnosis: Liveness probe failing. Remediation: Fix the 500 from /healthz, or raise initialDelaySeconds and failureThreshold so a slow recovery does not trigger the restart loop.

The Terminating excerpt

Status:       Terminating
Termination Grace Period:  30s
Containers:
  billing:
    State:    Running
      Started:  Fri, 16 Aug 2026 04:23:01 +0000
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Killing         4m    kubelet            Stopping container billing

Diagnosis: Pod is stuck in Terminating. Remediation: Investigate the finalizer, preStop hook, application, or kubelet.

The diagnostic command

The canonical diagnostic command sequence:

# 1. Get the Pod's current state
kubectl get pod billing-7d8f-abcde -n prod -o wide

# 2. Describe the Pod (the cheat sheet)
kubectl describe pod billing-7d8f-abcde -n prod

# 3. Read the events
kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

# 4. Read the previous logs (for CrashLoopBackOff)
kubectl logs -n prod billing-7d8f-abcde -c billing --previous

# 5. Read the current logs
kubectl logs -n prod billing-7d8f-abcde -c billing --tail=200

# 6. Read the finalizers (for stuck Terminating)
kubectl get pod billing-7d8f-abcde -n prod -o yaml | grep -A5 finalizers

The command sequence is the 11-step methodology in muscle memory.

Production discipline

The cheat sheet is the operator’s first reference. The discipline is to match the State.Reason to the failure mode, read the events, apply the remediation. The cheat sheet is the muscle memory of the 11-step methodology.

  • Read the State.Reason field first. The field is the canonical answer.
  • Run the diagnostic command sequence. The sequence is the 11-step methodology in muscle memory.

Quiz

Knowledge check · 4 questions

  1. Q1. Which field in `kubectl describe pod` is the canonical answer to the failure mode?

  2. Q2. An exit code of 137 is the canonical signal of an OOMKilled container.

  3. Q3. An operator runs `kubectl describe pod billing-7d8f-abcde -n prod`. The State is `Waiting`, the Reason is `CrashLoopBackOff`, the Last State Reason is `OOMKilled`, the Exit Code is 137. What is the failure mode and the remediation?

    The Pod is `billing-7d8f-abcde` in namespace `prod`. The workload is a 6-replica Deployment. The Pod has restarted 12 times in 5 minutes. The memory limit is 512Mi.

  4. Q4. Name three fields in `kubectl describe pod` that are the canonical answer to the failure mode and explain what each one tells the operator.

Passing score: 75%. Answers are checked in this browser.