The 11-step methodology — the canonical troubleshooting workflow

The canonical 11-step Kubernetes troubleshooting methodology converts a panic into a procedure. The steps are: define symptom → determine impact → inspect object → events → logs → dependencies → identify component → hypothesis → test → restore → validate. The discipline is the same scale-free: every incident runs the same workflow; the depth scales with the complexity.

The 11 steps in order

The methodology is a sequence. Each step has a single output that feeds the next step. Skipping a step is what turns a recoverable incident into a multi-hour outage.

flowchart TD
    A[1. Define symptom] --> B[2. Determine impact]
    B --> C[3. Inspect object]
    C --> D[4. Events]
    D --> E[5. Logs]
    E --> F[6. Dependencies]
    F --> G[7. Identify component]
    G --> H[8. Hypothesis]
    H --> I[9. Test]
    I --> J[10. Restore]
    J --> K[11. Validate]
    K --> L[Post-incident review]

The output of each step is the input to the next. The discipline is to never skip a step.

Step 1: Define the symptom

The symptom is the observable fact. The symptom is not the cause; the symptom is the thing the user sees. A CrashLoopBackOff is a symptom; “the new image is broken” is a hypothesis. The discipline is to write the symptom down in one sentence:

• “The pod billing-7d8f has restarted 12 times in the last 5 minutes.”
• “The Service billing-api returns 503 from synthetic traffic.”
• “The node node-03 reports DiskPressure.”

The symptom is the input to step 2.

Step 2: Determine the impact

The impact is the business consequence. The impact is not the technical detail; the impact is the customer-visible consequence. The impact is what drives the urgency:

• “5% of billing requests are failing; the customer-facing status is degraded.”
• “The billing-api Service is unreachable from the checkout workload; checkout is failing.”
• “The cluster has 12 nodes; one is in DiskPressure and is rejecting new Pods.”

The impact is the input to step 3. The impact also decides whether the operator should stop and ship a mitigation, or continue investigating.

Step 3: Inspect the object

The object is the Kubernetes resource that is misbehaving. The inspection is kubectl get and kubectl describe:

kubectl get pod billing-7d8f -n prod -o yaml
kubectl describe pod billing-7d8f -n prod

The output is the object’s spec, status, conditions, and events. The output is the input to step 4.

Step 4: Events

The events are the cluster’s history of the object. Events are time-ordered, scoped to the namespace, and have a type (Normal, Warning) and a reason. The events are the narrative of how the object got to its current state.

kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f

The events are the input to step 5.

Step 5: Logs

The logs are the container’s history. The logs are ordered by line, scoped to the container, and contain the application’s output. The logs are the narrative of how the container got to its current state.

kubectl logs -n prod billing-7d8f -c billing --previous
kubectl logs -n prod billing-7d8f -c billing --tail=200

The --previous flag is the key: it shows the logs of the previous container instance, which is the one that crashed before the current restart. The logs are the input to step 6.

Step 6: Dependencies

The dependencies are the other objects the misbehaving object relies on. The dependencies are the cluster’s relationships: the Service that routes to the Pod, the EndpointSlice that lists the Pods, the ConfigMap that the Pod mounts, the Secret that the Pod reads, the PersistentVolume that the Pod claims.

kubectl get pod billing-7d8f -n prod -o json | \
  jq '.spec.volumes[]?, .spec.containers[].env[]?, .spec.containers[].envFrom[]?'

The dependencies are the input to step 7.

Step 7: Identify the component

The component is the piece of software that is failing. The component is not the cluster; the component is the Kubernetes control plane, the kubelet, the CNI, the CSI, the workload, or the user’s application. The identification is the moment when the operator decides which component is the source of the symptom.

flowchart TD
    A[Symptom] --> B{Where?}
    B -->|Pod| C[Workload, container]
    B -->|Node| D[Kubelet, runtime, kernel]
    B -->|Service| E[Service, EndpointSlice, kube-proxy]
    B -->|Control plane| F[API server, scheduler, controller]
    B -->|Storage| G[CSI, PV, PVC]
    B -->|Network| H[CNI, DNS, NetworkPolicy]

The identification is the input to step 8.

Step 8: Hypothesis

The hypothesis is the proposed cause. The hypothesis is a single sentence that explains the symptom and predicts the outcome of the next test. The hypothesis is the moment when the operator stops gathering evidence and proposes a candidate.

The hypothesis is testable. A hypothesis that is not testable is a guess. The discipline is to write the hypothesis down in the form “If X, then Y.”

• “If the container is crashing because the new image is missing a library, then kubectl logs --previous will show ImportError: No module named 'requests'.”
• “If the Service is returning 503 because the EndpointSlice is empty, then kubectl get endpointslices will show no addresses.”

Step 9: Test

The test is the experiment that confirms or refutes the hypothesis. The test is the only step that changes the cluster’s state. The test is the moment when the methodology produces evidence.

kubectl describe pod billing-7d8f -n prod
kubectl logs -n prod billing-7d8f -c billing --previous
kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing-api

The test is the input to step 10.

Step 10: Restore

The restore is the mitigation that brings the cluster back to a working state. The restore is not the fix; the restore is the rollback or the workaround. The fix is the follow-up that the post-incident review schedules.

The restore is mechanical. The restore is the documented procedure that the operator runs. The restore is the moment when the cluster’s state is restored.

Step 11: Validate

The validate is the evidence that the restore worked. The validate is the same mechanical checks that the post-change validation runs: KPIs, PDBs, synthetic traffic, workload Ready. The validate is the moment when the operator knows the incident is over.

Production discipline

The 11-step methodology is the cluster’s troubleshooting playbook. The discipline is to run the steps in order, never skip a step, and use the same template for every incident. The post-incident review is the artefact that pays down the lesson.

• Write the symptom down. A panic without a symptom is a panic without a playbook.
• Hypothesise before testing. A test without a hypothesis is a guess.
• Restore, then fix. The restore is the rollback; the fix is the follow-up.
• Validate, then close. The validation is the evidence that the incident is over.

Quiz