low riskservice affecting~25 min

Runbook: Investigate a CrashLoopBackOff Pod

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the current container state: kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses}' | jq
· Capture the recent events: kubectl describe pod <name> -n <ns> | sed -n '/Events:/,$p'
· Capture the previous container logs: kubectl logs <name> -n <ns> --previous --tail=200
· Capture the current container logs: kubectl logs <name> -n <ns> --tail=200
· Confirm the Pod is not Pending: kubectl get pod <name> -n <ns> shows Running and a non-zero restart count
· Capture the owning controller (Deployment, StatefulSet, Job) and its rollout history

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the previous-container logs (--previous) first; the current container may be too early in its lifecycle to have logs
2Classify the failure from the exit code and the log output: application panic, missing config, probe failure, OOMKill, dependency unavailable, runtime error
3For application panic: identify the exception/stack trace, capture it to the incident record, decide whether the bug is in the workload or the manifest
4For missing config: kubectl get cm,secret -n <ns> and compare the keys the application expects (kubectl logs typically prints the missing key)
5For probe failure: read kubectl describe pod <name> -n <ns> | sed -n "/Liveness|Readiness|Startup/,/Conditions:/p" and check the probe path/port against the application
6For OOMKill: read the container state reason (kubectl get pod -o jsonpath) and the previous lastState.terminated.reason (OOMKilled); see kubernetes-rb-investigate-oomkilled
7For dependency unavailable: capture the service endpoint state, CoreDNS resolution, and the network path (kubectl exec ... -- nc -vz <host> <port>)
8For runtime error: read the runtime logs (crictl logs <container-id> on the node) for the kubelet-side error
9Apply the smallest fix that resolves the cause
10Roll the Pod to pick up the fix: kubectl delete pod <name> -n <ns> for a Deployment-managed Pod, or kubectl rollout restart deploy/<name> -n <ns>
11Watch the new Pod become Ready: kubectl wait --for=condition=Ready pod/<name> -n <ns> --timeout=120s

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get pod <name> -n <ns> reports Running and a stable restart count
✓kubectl logs <name> -n <ns> --previous --tail=20 shows the previous failure mode (captured to the incident)
✓kubectl logs <name> -n <ns> --tail=50 shows a normal start sequence, no panic, no immediate exit
✓Readiness probe passes (Pod Ready becomes true and stays true)
✓Dashboard for the workload returns to baseline (error rate, latency)
✓No new Warning events in the last 5 minutes: kubectl get events -n <ns> --field-selector involvedObject.name=<name>

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix was a manifest change in Git, git revert <sha> and kubectl apply brings the bad version back; do this only if the fix made things worse
↶If the fix involved relaxing a probe, revert to the original probe settings once the underlying issue is fixed
↶If kubectl delete pod for a Deployment-managed Pod triggers the rollout to stall (rare), kubectl rollout undo deploy/<name> -n <ns> reverts to the previous ReplicaSet
↶If the workload cannot be made stable, scale the Deployment to zero to stop the crash loop while the fix is developed: kubectl scale deploy/<name> -n <ns> --replicas=0
↶Capture the failing logs to an off-cluster location before any rollback that would lose them

6 · Escalation

When the runbook isn't enough, contact:

· Container exits with code 137 (SIGKILL) but lastState.terminated.reason is not OOMKilled: a process outside the cgroup killed it; check the node journal for OOM events at the same time
· Repeated panics in the application with no config or probe cause: this is an application bug; hand off to application ownership with the stack trace and the manifest
· CrashLoopBackOff across every replica and every revision: the workload is broken at the image level; do not roll forward, escalate to image or build pipeline ownership
· Crash loop only on certain nodes: a node-level issue (kernel, runtime, network); see kubernetes-rb-troubleshoot-node-notready
· CrashLoopBackOff on a system component (kube-proxy, CNI, CoreDNS): cluster-level incident; escalate to platform ownership

A CrashLoopBackOff means the container exited and kubelet is throttling restarts (10s, 20s, 40s, … up to 5 minutes). The back-off is not the problem; the exit is.

1. Identify the exit

Read-only / SafeIdentify the exit

kubectl get pod <name> -n <ns> -o jsonpath='{.status}' | jq '.containerStatuses[] | {name, ready, restartCount, lastState: .lastState, state: .state}'
# Note: `lastState.terminated.reason` is the strongest signal
# Common reasons: Error, OOMKilled, Completed, ContainerStatusUnknown

Exit code / reason	Class	Where to look next
1 + `Error`	Application panic or uncaught exception	Application logs (`--previous`)
137 + `OOMKilled`	Memory limit hit	cgroup memory, see OOM runbook
139 + segfault	Native crash (linked library, JNI)	Node journal, `crictl logs`
143 + SIGTERM	External kill (probe, eviction, drain)	Events, kubelet logs
0 + `Completed`	Job-style workload finished	Owner controller, expected behaviour
`ContainerStatusUnknown`	Runtime lost contact	Node state, see `troubleshoot-node-notready`

2. Read the logs

Read-only / SafeRead the logs


kubectl logs <name> -n <ns> --tail=200

# Previous container - this is where the failure lives
kubectl logs <name> -n <ns> --previous --tail=200

# For multi-container Pods, target a specific container
kubectl logs <name> -c <container> -n <ns> --previous --tail=200

# If the logs are empty, the runtime lost them: check the kubelet on the node
kubectl get pod <name> -n <ns> -o jsonpath='{.status.containerStatuses[0].containerID}'
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}' | xargs -I{} ssh {} sudo crictl logs <container-id>

3. Apply the smallest fix

3a. Application panic

The stack trace is the evidence. Capture the first line of the exception (or the panic message) and forward it to the owning team. The fix is in the code, not in Kubernetes.

# Substitute your own values before running:
POD=payments-api-7d9f4c6b58-nkq2r
NS=production

# Capture the stack for the incident record
kubectl logs "$POD" -n "$NS" --previous --tail=500 > /tmp/crash.log
grep -m1 -E 'panic|Exception|Caused by|Traceback' /tmp/crash.log

3b. Missing config

Read-only / Safeb. Missing config


kubectl describe cm <name> -n <ns>
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[*].envFrom[*].configMapRef.name}{" "}{.spec.containers[*].envFrom[*].secretRef.name}{" "}{.spec.volumes[?(@.configMap)].configMap.name}{" "}{.spec.volumes[?(@.secret)].secret.secretName}'

The application log usually names the missing key. Compare against the manifest, fix in Git, and kubectl rollout restart.

3c. Probe failure

Read-only / Safec. Probe failure


kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[0].livenessProbe}' | jq
kubectl get pod <name> -n <ns> -o jsonpath='{.spec.containers[0].readinessProbe}' | jq

# Inside the container, run the probe manually
kubectl exec <name> -n <ns> -- curl -fsS http://localhost:8080/live || true
kubectl exec <name> -n <ns> -- curl -fsS http://localhost:8080/ready || true

A probe that fires before the application is ready will kill a healthy container. initialDelaySeconds, periodSeconds, and the probe path must all be correct for the application.

3d. OOMKill

See kubernetes-rb-investigate-oomkilled. Do not raise the limit without measuring actual usage first.

3e. Dependency unavailable

Read-only / Safee. Dependency unavailable


kubectl exec <name> -n <ns> -- nslookup <dependency-service>
kubectl exec <name> -n <ns> -- getent hosts <dependency-service>
kubectl exec <name> -n <ns> -- nc -vz <dependency-host> <dependency-port>

A DNS failure with a working CoreDNS is usually a NetworkPolicy. See kubernetes-rb-troubleshoot-networkpolicy.

4. Roll forward

Read-only / SafeRoll forward


kubectl rollout restart deploy/<name> -n <ns>

# For Pods without an owner (e.g. created via `kubectl run`), delete and recreate
kubectl delete pod <name> -n <ns>
kubectl apply -f pod.yaml

# Watch
kubectl wait --for=condition=Ready pod -l app=<name> -n <ns> --timeout=5m
kubectl logs -l app=<name> -n <ns> --tail=50

Common pitfalls

Symptom	Cause	Action
Logs empty, `OOMKilled` not in lastState	Runtime lost the logs	Read kubelet journal on the node
Application logs a panic, then container exits 0	Process forked a child that crashed	Check the child PID in node journal
Probe passes locally but fails in cluster	NetworkPolicy blocking the probe	See `kubernetes-rb-troubleshoot-networkpolicy`
Crash loop only on a single node	Node-level issue (runtime, kernel)	Cordon, drain, and follow the `kubernetes-rb-troubleshoot-node-notready` flow
Crash loop every 5 minutes (max back-off)	Back-off is working correctly; the application is consistently exiting	Treat as a workload bug, not a kubelet issue

A CrashLoopBackOff is a symptom, not a diagnosis. Read the exit code, read the previous logs, find the cause, fix the smallest thing.