As neededkubernetes-incident-triage

Kubernetes Incident Triage Checklist

13 items ·12 critical ·1 warn ·0 info

How to use this checklist

Work top to bottom during an incident. The ordering is deliberate: establish impact before cause, and collect evidence before changing anything, because the first change usually destroys the evidence.

Mark an item N/A when it genuinely does not apply, and write down why. An unexplained N/A is the most common way a checklist stops working.

Sign-off

Every critical item must pass. A failing critical item blocks the deployment or the maintenance window; it is not a note for later. Record the date, the reviewer, and the disposition of every item that did not pass.

Critical12 items

State the user-visible impact in one sentence: what is broken, for whom, since when. Everything else is secondary to this.
Establish scope: one Pod, one node, one namespace, or the whole cluster. This single question eliminates most hypotheses.
```
kubectl get pods -A --field-selector=status.phase!=Running
```
Check node health. A large fraction of workload symptoms are a node problem wearing a workload costume.
```
kubectl get nodes -o wide
```
Check control-plane and etcd health. If the API server is degraded, every other observation is unreliable.
```
kubectl get --raw /readyz?verbose && etcdctl endpoint health --cluster
```
Ask what changed: deploys, upgrades, config, certificates, external dependencies. Most incidents are a change, and the change is usually recent.
```
kubectl get events -A --sort-by=.lastTimestamp | tail -50
```
Capture evidence before remediating: describe, logs (including --previous), and events for the affected objects. A restart destroys the evidence.
```
kubectl describe pod <pod> -n <ns>; kubectl logs <pod> -n <ns> --previous
```

Check for resource pressure and eviction: node conditions, OOMKilled containers, disk and PID pressure.

kubectl get pods -A -o json | jq -r '.items[] | select(.status.containerStatuses[]?.lastState.terminated.reason=="OOMKilled") | .metadata.name'

State a hypothesis that the evidence supports and that predicts something testable. Do not act on the first plausible story.
Mitigate before fixing. Restoring service and finding the root cause are separate activities with different urgencies.
Change one thing at a time and record it. Parallel changes make the eventual post-mortem unresolvable.
Verify recovery through the user-visible symptom, not through the component you changed.
Write the post-mortem while the detail is fresh, and give it an owner and a date.

Warning1 item

Keep stakeholders updated on a fixed cadence, including when nothing has changed.